The alchemy of confirmation bias transmutes expectations into bilingual advantages: A tale of two new meta-analyses

Abstract

Five recent meta-analyses of the bilingual advantage in executive functioning hypothesis have converged on the outcome that the mean effect size is very small and that the incidence of statistically significant bilingual advantages is very low (about 15% of all comparisons). Those analyses that used the PET-PEESE method to correct for publication bias show mean effect sizes that are not different from zero and sometimes negative. In contrast, van den Noort and colleagues provide a sixth review of 46 studies published before October 31, 2018, that appears to produce a very different outcome, namely that more than half the studies yield clear support for the bilingual advantage hypothesis. We show that the deviance is due in part to search terms that yielded far fewer relevant studies, but more importantly to a subjective method of evaluating the results of each study that enables confirmation biases on the part of study authors and meta-analysts to substantially distort the objective pattern of results. A seventh meta-analysis, by Armstrong and colleagues, reports significant bilingual advantages of g = 0.48 for 32 samples using Simon and Stroop colour–word interference tasks that tested older adults. However, all effects were entered into the funnel plots as positive even though many were negative (bilingual disadvantages). This and other striking anomalies are consistent with the view that confirmation bias can suspend critical judgement and promulgate errors. Meta-analyses that use preregistration and a many-labs collaboration can better control for both publication and experimenter biases.