Which statistical method we can use for replacing missing values for categorical feature

Table of contents • • 1 Introduction • 2 Prerequisites • I. BASIC • 3 Descriptive Statistics • 4 Basic Statistical Inference • II. REGRESSION • 5 Linear Regression • 6 Non-linear Regression • 7 Generalized Linear Models • 8 Linear Mixed Models • 9 Nonlinear and Generalized Linear Mixed Models • III. RAMIFICATIONS • 10 Model Specification • 11 Imputation (Missing Data) • 12 Data • 13 Variable Transformation • 14 Hypothesis Testing • 15 Marginal Effects • 16 Prediction and Estimation • 17 Moderation • IV. CAUSAL INFERENCE • 18 Causal Inference • A. EXPERIMENTAL DESIGN • 19 Experimental Design • 20 Sampling • 21 Analysis of Variance (ANOVA) • 22 Multivariate Methods • B. QUASI-EXPERIMENTAL DESIGN • 23 Quasi-experimental • 24 Regression Discontinuity • 25 Difference-in-differences • 26 Synthetic Control • 27 Event Studies • 28 Matching Methods • 29 Interrupted Time Series • C. OTHER CONCERNS • 30 Endogeneity • 31 Controls • 32 Mediation • 33 Directed Acyclic Graph • V. MISCELLANEOUS • 34 Report • 35 Exploratory Data Analysis • 36 Sensitivity Analysis/ Robustness Check • APPENDIX • A Appendix • B Bookdown cheat sheet • 11 Imputation (Missing Data) Imputation is a statistical procedure where you replace missing data with some values • Unit imputation = single data point • Item imputation = single feature value Imputation is usually seen as the illegitimate child of statistical analysis. Several reasons that contribute to this negative views could be: • Peopled hardly do imputati...

I'm doing a project that involves replacing missing values in a set of data (first time doing this). This involves using two methods replacement by mean and replacement by median to fill in the missing values. There is not a lot of difference between the results of the minimum, median, maximum, mean and standard deviation of the data using both methods and I was wondering which method is better and how can I make a decision to which one is better using the results produced? $\begingroup$ If you replace missings with means, naturally the mean is preserved. Ditto medians. Nor will the extremes change. The SDs will typically be reduced slightly, but it would be reduced greatly if you do this a lot. These are predictable consequences of what you do and not ipso facto indications that the method is good. $\endgroup$ $\begingroup$ Analysts plugging missing values (MVs) with automatic "solutions" like this aren't thinking through the consequences. It's just an easily implemented approach. This "solution" introduces as many problems as it solves since an otherwise typically smooth pdf ends up with a large spike at the plugged value, as a function of the number of MVs, of course. Model-based imputations are demonstrably superior and less biasing than any automated approach. @NickCox can't be ignorant of this, despite what his suggestion implies. $\endgroup$ $\begingroup$ Replacement by mean or median --- or mode -- is in effect saying that you have no information on what a missing ...

Definition: Missing data imputation is a statistical method that replaces missing data points with substituted values. In the following step by step guide, I will show you how to: • Apply missing data imputation • Assess and report your imputed values • Find the best imputation method for your data But before we can dive into that, we have to answer the question… Why Do We Need Missing Value Imputation? The default method for handling missing data That approach is easy to understand and to Well, it’s as always: Because we can improve the quality of our data analysis! The impact of missing values on our data analysis depends on the response mechanism of our data ( Table 1 shows a comparison of listwise deletion (the default method in R) and missing data imputation. Table 1: Crosstabulation of bias, variance, and the three response mechanisms Missing Completely At Random (MCAR), Missing At Random (MAR), and Missing Not At Random (MNAR). Table 1 illustrates two major advantages of missing data imputation over listwise deletion: • The variance of analyses based on imputed data is usually lower, since missing data imputation does not reduce your sample size. • Depending on the response mechanism, missing data imputation outperforms listwise deletion in terms of bias. To make it short: Missing data imputation almost always improves the quality of our data! Therefore we should definitely replace missing values by imputation. But how does it work? That’s exactly what I’m going to ...

Sebastian Jäger*, Arndt Allhorn and Felix Bießmann • Beuth University of Applied Sciences, Berlin, Germany With the increasing importance and complexity of data pipelines, data quality became one of the key challenges in modern software applications. The importance of data quality has been recognized beyond the field of data engineering and database management systems (DBMSs). Also, for machine learning (ML) applications, high data quality standards are crucial to ensure robust predictive performance and responsible usage of automated decision making. One of the most frequent data quality problems is missing values. Incomplete datasets can break data pipelines and can have a devastating impact on downstream ML applications when not detected. While statisticians and, more recently, ML researchers have introduced a variety of approaches to impute missing values, comprehensive benchmarks comparing classical and modern imputation approaches under fair and realistic conditions are underrepresented. Here, we aim to fill this gap. We conduct a comprehensive suite of experiments on a large number of datasets with heterogeneous data and realistic missingness conditions, comparing both novel deep learning approaches and classical ML imputation methods when either only test or train and test data are affected by missing data. Each imputation method is evaluated regarding the imputation quality and the impact imputation has on a downstream ML task. Our results provide valuable insight...

$\begingroup$ Without further context an imputation model using a logistic regression model would deal fine with binary categorical variables, while a multinomial or ordinal regression could find replacement values for missing multilevel (>2 levels) or ordered multilevel variables respectively. If these models fit poorly or take a lot of computational time, permutative mean matching might be a quick option as well. Most importantly however, do not use single imputation strategies. These will incorrectly increase power, and might bias results (use multiple imputation or other techniques instead). $\endgroup$ Thanks for contributing an answer to Cross Validated! • Please be sure to answer the question. Provide details and share your research! But avoid … • Asking for help, clarification, or responding to other answers. • Making statements based on opinion; back them up with references or personal experience. Use MathJax to format equations. To learn more, see our

Let's suppose I have a column with categorical data "red" "green" "blue" and empty cells red green red blue NaN I'm sure that the NaN belongs to red green blue, should I replace the NaN by the average of the colors or is a too strong assumption? It will be col1 | col2 | col3 1 0 0 0 1 0 1 0 0 0 0 1 0.5 0.25 0.25 Or even scale the last row but keeping the ratio so these values have less influence? Usually what is the best practice? 0.25 0.125 0.125 The simplest strategy for handling missing data is to remove records that contain a missing value. The scikit-learn library provides the Imputer() pre-processing class that can be used to replace missing values. Since it is categorical data, using mean as replacement value is not recommended. You can use from sklearn.preprocessing import Imputer imp = Imputer(missing_values='NaN', strategy='most_frequent', axis=0) The Imputer class operates directly on the NumPy array instead of the DataFrame. Last but not least, not ALL ML algorithm cannot handle missing value. Different implementations of ML also different. It depends on what you want to do with the data. Is the average of these colors useful for your purpose? You are creating a new possible value doing that, that is probably not wanted. Especially since you are talking about categorical data, and you are handling it as if it was numeric data. In Machine Learning you would replace the missing values with the most common categorical value regarding a target attribute ( what you ...

6.4. Imputation of missing values For various reasons, many real world datasets contain missing values, often encoded as blanks, NaNs or other placeholders. Such datasets however are incompatible with scikit-learn estimators which assume that all values in an array are numerical, and that all have and hold meaning. A basic strategy to use incomplete datasets is to discard entire rows and/or columns containing missing values. However, this comes at the price of losing data which may be valuable (even though incomplete). A better strategy is to impute the missing values, i.e., to infer them from the known part of the data. See the glossary entry on imputation. 6.4.1. Univariate vs. Multivariate Imputation One type of imputation algorithm is univariate, which imputes values in the i-th feature dimension using only non-missing values in that feature dimension (e.g. impute.SimpleImputer). By contrast, multivariate imputation algorithms use the entire set of available feature dimensions to estimate the missing values (e.g. impute.IterativeImputer). 6.4.2. Univariate feature imputation The SimpleImputer class provides basic strategies for imputing missing values. Missing values can be imputed with a provided constant value, or using the statistics (mean, median or most frequent) of each column in which the missing values are located. This class also allows for different missing values encodings. The following snippet demonstrates how to replace missing values, encoded as np.nan, ...