Testing multilevel mediation using hierarchical linear modeling (HLM) has gained tremendous popularity in recent years. However, potential confounding in multilevel mediation effect estimates can arise in these models when within-group effects differ from between-group effects. This study summarizes three types of HLM-based multilevel mediation models, and then explains that in two types of these models confounding can be produced and erroneous conclusions may be derived when using popularly recommended procedures. A Monte Carlo simulation study illustrates that these procedures can underestimate or overestimate true mediation effects. Recommendations are provided for appropriately testing multilevel mediation and for differentiating within-group versus between-group effects in multilevel settings.
Baron, R.M., & Kenny, D.A. ( 1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations . Journal of Personality and Social Psychology, 51, 1173-1182.
2.
Bauer, D.J., Preacher, K.J., & Gil, K.M. ( 2006). Conceptualizing and testing random indirect effects and moderated mediation in multilevel models: New procedures and recommendations . Psychological Methods, 11, 142-163.
3.
Busing, F. ( 1993). Distribution characteristics of variance estimates in two-level models. Unpublished manuscript.
4.
Chen, G., & Bliese, P.D. ( 2002). The role of different levels of leadership in predicting self- and collective efficacy: Evidence for discontinuity. The Journal of Applied Psychology, 87, 549-556.
5.
Chen, G., Kirkman, B.L., Kanfer, R., Allen, D., & Rosen, B. ( 2007). A multilevel study of leadership, empowerment, and performance in teams. The Journal of Applied Psychology, 92, 331-346.
6.
Cheung, G.W., & Lau, R.S. ( 2008). Testing mediation and suppression effects of latent variables: Bootstrapping with structural equation models. Organizational Research Methods, 11, 296-325.
7.
Cohen, J. ( 1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8.
Dansereau, F., Alutto, J.A., & Yammarino, F.J. (1984). Theory testing in organizational behavior: The variant approach. Englewood Cliffs, NJ: Prentice Hall.
9.
Davis, J.A., Spaeth, J.L., & Huson, C. ( 1961). A technique for analyzing the effects of group composition . American Sociological Review, 26, 215-225.
10.
Edwards, J.R., & Lambert, L.S. ( 2007). Methods for integrating moderation and mediation: A general analytical framework using moderated path analysis. Psychological Methods, 12, 1-22.
11.
Freedman, L.S., & Schatzkin, A. ( 1992). Sample size for studying intermediate endpoints within intervention trials of observational studies. American Journal of Epidemiology, 136, 1148-1159.
12.
Hofmann, D.A., & Gavin, M.B. ( 1998). Centering decisions in hierarchical linear models: Implications for research in organizations. Journal of Management, 24, 623-641.
13.
Hofmann, D.A., Griffin, M.A., & Gavin, M.B. ( 2000). The application of hierarchical linear modeling to organizational research. In K. J. Klein & S. W. J. Kozlowski (eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 467-511). San Francisco, CA: Jossey-Bass.
14.
Kenny, D.A., Korchmaros, J.D., & Bolger, N. ( 2003). Lower level mediation in multilevel models. Psychological Methods, 8, 115-128.
15.
Klein, K.J., & Kozlowski, S.W.J. (2000). From micro to meso: Critical steps in conceptualizing and conducting multilevel research. Organizational Research Methods, 3, 211-236.
16.
Kreft, I., & de Leeuw, J. ( 1998). Introducing multilevel modeling. Thousand Oaks, CA: Sage.
17.
Krull, J.L., & MacKinnon, D.P. (1999). Multilevel mediation modeling in group based intervention studies. Evaluation Review, 33, 418-444.
18.
Krull, J.L., & MacKinnon, D.P. (2001). Multilevel modeling of individual and group level mediated effects. Multivariate Behavioral Research, 36, 249-277.
19.
MacKinnon, D.P. ( 2008). Introduction to statistical mediation analysis. Mahwah, NJ: Taylor & Francis.
20.
MacKinnon, D.P., Krull, J.L., & Lockwood, C.M. (2000). Equivalence of the mediation, confounding and suppression effect. Prevention Science, 1, 173-181.
21.
MacKinnon, D.P., Lockwood, C.M., Hoffman, J.M., West, S.G., & Sheets, V. ( 2002). A comparison of methods to test the significance of the mediated effect. Psychological Methods, 7, 83-104.
22.
MacKinnon, D.P., Lockwood, C.M., & Williams, J. ( 2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39, 99-128.
23.
MacKinnon, D.P., Warsi, G., & Dwyer, J.H. ( 1995). A simulation study of mediated effect measures. Multivariate Behavioral Research, 30, 41-62.
24.
Mathieu, J.E., DeShon, R.P., & Bergh, D.D. ( 2008). Mediational inferences in organizational research: Then, now and beyond. Organizational Research Methods, 11, 203-223.
25.
Mathieu, J.E., & Taylor, S.R. ( 2007). A framework for testing meso-mediational relationships in organizational behavior. Journal of Organizational Behavior , 28, 141-172.
26.
Muller, D., Yzerbyt, V.Y., & Judd, C.M. ( 2008). Adjusting for a mediator in models with two crossed treatment variables. Organizational Research Methods, 11, 224-240.
27.
Ostroff, C. ( 1993). Comparing correlations based on individual level and aggregate data. The Journal of Applied Psychology, 78, 569-582.
28.
Pituch, K.A., & Stapleton, L.M. (2008). The performance of methods to test upper-level mediation in the presence of nonnormal data. Multivariate Behavioral Research, 43, 237-267.
29.
Pituch, K.A., Tate, R.L., & Murphy, D.L. (in press). Three-level models for indirect effects in school- and class-randomized experiments in education. Journal of Experimental Education .
30.
Raudenbush, S.W., & Bryk, A.S. ( 2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
31.
Seibert, S.E., Silver, S.R., & Randolph, W.A. (2004). Taking empowerment to the next level: A multiple-level model of empowerment, performance, and satisfaction . Academy of Management Journal, 47, 332-349.
32.
Snijders, T.A., & Bosker, R.J. ( 1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage.
33.
Sobel, M.E. ( 1982). Asymptotic confidence intervals for indirect effects in structural models. In S. Leinhardt (ed.), Sociological Methodology 1982 (pp. 290-312). San Francisco, CA: Jossey-Bass.
34.
Stone-Romero, E.F., & Rosopa, P.J. ( 2008). The relative validity of inferences about mediation as a function of research design characteristics. Organizational Research Methods, 11, 326-352.
35.
Taylor, A.B., MacKinnon, D.P., & Tein, J.Y. ( 2008). Tests of the three-path mediated effect. Organizational Research Methods, 11, 241-269.
