Abstract
We begin this Editorial by making an easy claim: The world is multilevel. People live in houses and neighborhoods, learn in classrooms and schools, and work in teams and organizations—and researchers often collect data across these levels over time. At best, treating psychological data in a manner that ignores this nested structure will result in a loss of an empirical understanding of the nature of these levels and their interactions; and at worst, such ignorance may cause statistical and conceptual inefficiencies during the process of making recommendations from multilevel and meta-analytic data, resulting in potentially misleading or even harmful consequences. In this issue and the September 2019 issue, Advances in Methods and Practices in Psychological Science has been fortunate to feature a series of six methodological articles that embrace the multilevel structure of data, addressing critical issues in multilevel modeling and meta-analysis in a way that provides psychological researchers with practical guidance and embodies future-oriented thinking.
Hoffman (2019) has provided helpful educational and practical guidance for deciding among the many options that are currently available in multilevel-modeling software: For example, one may choose Bayesian or maximum likelihood modeling, observed or latent variables, and different types of centering and scaling. The availability of these choices might seem inherently beneficial, but they can lead to uncertainty and confusion; they also can increase researchers’ degrees of freedom. Likewise, equally legitimate decisions might lead to different substantive interpretations, so Hoffman’s guide for judiciously navigating through the sea of analytic choices in multilevel analysis is very timely for taking advantage of today’s advances in multilevel modeling in ways that will enable the appropriate analyses that will answer a psychological researcher’s multilevel questions of interest.
Lang, Bliese, and Adler (2019) have provided an example of these advanced multilevel-modeling options: a multilevel model that can be used to investigate how group perceptions and behaviors change over time. Individual and group change can be substantively interpreted in many ways in terms of their individual and joint influence on climate, consensus, and decision making, as Lang et al. demonstrated, for example, in their analysis of decisions made in mock-jury trials. Furthermore, researchers have found that multilevel models often serve to support the inference that many groups ultimately reach consensus, such that within-group variances tend to be worth accounting for via multilevel modeling, even after taking mean-level floor or ceiling effects into account (e.g., for additional multilevel modeling of between-groups differences in within-group, or residual, variances, see Lester, Cullen-Lester, & Walters, 2019).
McShane and Böckenholt’s (2020; this issue) contribution wrestled with the tension between the practical issue of typically having only summary data to enter into a meta-analysis and the wish to obtain richer information that would be available with multilevel modeling of the raw data (often at the individual level). The authors explored cases and research questions to illustrate when the meta-analysis approach is appropriate and relevant (e.g., understanding the overall effect of a single dependent measure in a single group, measured in multiple studies) and when richer multilevel modeling is needed more than meta-analysis (e.g., understanding effects found in multiple dependent measures, given multiple groups and covariates).
Measurement issues are also critical in multilevel analysis. Stapleton and Johnson (2019) emphasized, via simulation and an example with educational data, how the reliable variation in measurement found between clusters may be a function of sampling of individuals within them (an individual-driven effect) or may also be due to group-level influences above and beyond those individual effects. Both effects are important to model—and unconfound. Incorporating latent variables in multilevel modeling is an important step that is not taken often enough when the goal is to understand multilevel measurement and construct-level relationships. By contrast, meta-analysis has been much more focused than multilevel modeling on the consideration of, estimation of, and correction for measurement error variance, range restriction, and other psychometric artifacts. In meta-analysis, the purpose of artifact correction is to attempt to obtain better estimates of the meta-analytic mean and random-effects variance within a set of effect sizes of interest and to predict a portion of that variance with categorical or continuous moderators. Meta-analysis models that employ such statistical corrections (Schmidt & Hunter, 2014) can be useful, as they can give one a better idea about underlying latent relationships—as long as the increased standard errors that come with these corrections are manageable (Oswald, Ercan, McAbee, Ock, & Shaw, 2015).
The artifact of sampling error variance is almost always accounted for in meta-analytic modeling. However, as just noted, measurement error variance (the unreliability of measurement) is another key artifact, because measurement error variance can systematically distort both the absolute and the relative magnitudes of effect sizes of interest. The artifacts of direct and incidental range restriction cause similar systematic distortions, such as when a specific sample is known to be subject to nonrandom (e.g., selection) effects (e.g., when academic motivation and achievement of students are examined within elite schools, yet the goal is to generalize findings to a broader population of college applicants or to college students in general). Psychometric corrections in meta-analysis can be an important and useful tool for improving the interpretation of meta-analytic results, as cogently illustrated by Wiernik and Dahlke (2020; this issue); but just as clear is that no correction can perfectly address conceptual limitations, practical constraints, or other factors that influence the sampling, measures, and manipulations within any particular study. But roughly the opposite is true: Better samples, measures, and manipulations allow multilevel modeling and meta-analysis to become more powerful and informative tools.
The future of multilevel modeling and meta-analysis will be shaped by advances in technologies, analytic methods, and open-science practices that allow for interactive statistical tools, improved modeling and prediction, and rapid analysis (e.g., online interfaces that preserve subject-level anonymity while allowing for complex real-time modeling by external parties). A prime example of what the future might hold is metaBUS, an online interactive tool developed by Bosco, Field, Larsen, Chang, and Uggerslev (2020; this issue). First, across 27 journals and 39 years, the authors manually coded more than a million correlations, the largest database of its kind. Second, the correlations were categorized using a hierarchical taxonomy that was informed by extensive expert judgment. Third, the authors added a public-domain interactive tool that allows users to search correlations, conduct meta-analyses, and generate visualizations in real time. Psychological researchers who dream big can likely imagine a future in which (a) authors make use of a standardized taxonomy (metaBUS’s or another) for categorizing the variables and correlations being investigated, and, in tandem, (b) journals provide in-text and tabled quantitative information in formats that are at least more usable for multilevel modeling and meta-analysis, if not standardized. Psychological researchers should continue to dream big and work to make dreams come true—and in the meantime, we hope that these articles prove helpful in improving research, thinking, and application in the areas of multilevel modeling and meta-analysis.