Discussion
Missing variance measures are a prevalent problem in research synthesis. Yet, few ecological meta-analyses have adapted imputation algorithms to handle missing values (Fig. 1). Our study demonstrates how the omission of incompletely reported studies (complete-case analysis), generally increases the confidence intervals and how it results in deviating (potentially even biased) grand mean estimates if SDs/SSs are not missing completely at random. The R-code used to simulate and compare the effects of different meta-analysis datasets structures, patterns of missingness and options to handle missing data is freely available at github.com/StephanKambach/SimulateMissingDataInMeta-Analyses. Although our number of ten replicates is at the lower end of the desired replications in simulation studies, it was enough to show the general effects of treating missing SDs and SSs and meta-analysis data sets.
In accordance with previous publications, we found that unweighted analyses yielded grand mean estimates that were unbiased with regard to fully informed weighted analyses as long as effect sizes and their corresponding variance estimates were normally and independently distributed. The same holds for sample-size-approximated effect sizes variances. In case of a potential relationship between effect sizes and effect size precision (maybe due to different study designs) we advise to apply imputation methods to fill missing SDs and/or SSs.
If SDs and/or SSs are both MCAR and unrelated to effect sizes, the imputation of up to 90% of missing data yielded grand means similar to those obtained from fully informed weighted meta-analyses. Below a threshold of ca. 50-60% of missing SDs and/or SSs, imputation methods performed equally or outperformed complete-case, unweighted and sample-size weighted analyses. Yet, our results also demonstrated, that different imputation methods can accommodate different dataset structures regarding missingness and correlation patterns. Mean, median and random sample imputations are easy to implement but biased in case of a relationship between effect sizes and effect size precision. Methods applying predictive mean matching tend to suit such relationships but tend to yield a larger confidence intervals of the grand mean. Thus, for any meta-analysis, the method used to deal with missing SDs and/or SSs should be chosen under the following considerations: