The use of multilevel modeling to investigate organizational phenomena is rapidly increasing. Unfortunately, little advice is readily available for organizational researchers attempting to determine statistical power when using multilevel models or when determining sample sizes for each level that will maximize statistical power. This article presents an introduction to statistical power in multilevel models. The unique factors influencing power in multilevel models and calculations for estimating power for simple fixed effects, variance components, and cross-level interactions are presented. The results of simulation studies and the existing general rules of thumb are discussed, and the available power analysis software is reviewed.
Afshartous, D. (1995, April). Determination of sample size for multilevel model design. Paper presented at the annual meeting of the American Educational Research Association, San Francisco .
2.
Austin, J.T., Boyle, K.A., & Lualhati, J.C. (1998). Statistical conclusion validity for organizational science researchers: A review. Organizational Research Methods, 1, 164-208.
3.
Barcikowski, R.S. (1973). Optimum sample size and number of levels in a one-way random-effects analysis of variance. Journal of Experimental Education, 41, 10-16.
4.
Barcikowski, R.S. (1981). Statistical power with group mean as the unit of analysis. Journal of Educational Statistics, 6, 267-285.
5.
Bassiri, D. (1988). Large and small sample properties of maximum likelihood estimates for hierarchical linear models. Unpublished doctoral dissertation , Michigan State University, East Lansing.
6.
Bliese, P.D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349-381). San Francisco: Jossey-Bass.
7.
Bliese, P.D., & Halverson, R.R. (1998). Group size and measures of group-level properties: An examination of Eta-squared and ICC values. Journal of Management, 24, 157-172.
8.
Boyle, M.H., & Willms, J.D. (2001). Multilevel modeling of hierarchical data in developmental studies. Journal of Child Psychology and Psychiatry and Allied Disciplines, 42, 141-162.
9.
Brown, C.H., & Liao, J. (1999). Principles for designing randomized preventive trials in mental health: An emerging developmental epidemiology paradigm. American Journal of Community Psychology, 27, 673-710.
10.
Brown, W.J., & Daper, D. (2000). Implementation and performance issues in the Bayesian and likelihood fitting of multilevel models. Computational Statistics, 15, 391-420.
11.
Bryk, A.S., & Raudenbush, S.W. (1992). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
12.
Busing, F. (1993). Distribution characteristics of variance estimates in two-level models. Unpublished manuscript, Leiden University at Leiden, Netherlands.
13.
Cochran, W.G. (1977). Sampling techniques (3rd ed.). New York: John Wiley.
14.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
15.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.
16.
Cohen, M. (1998). Determining sample sizes for surveys with data analyzed by hierarchical linear models. Journal of Official Statistics, 14, 267-275.
17.
Davidson, M.L., Kwak, N., Seo, Y.S., & Choi, J. (2002). Using hierarchical linear models to examine moderator effects: Person-by-organization interaction. Organizational Research Methods, 5, 231-254.
18.
Hofmann, D.A. (1997). An overview of the logic and rationale of hierarchical linear models. Journal of Management, 23, 723-744.
19.
Hofmann, D.A., Griffin, M.A., & Gavin, M.B. (2000). The application of hierarchical linear modeling to organizational research. In K. Klein & S. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 467-511). San Francisco: Jossey-Bass.
20.
Hox, J.J. (1998). Multilevel modeling: When and why. In I. Balderjahn, R. Mathar , & M. Schader (Eds.), Classification, data analysis, and data highways (pp. 147-154). New York: Springer-Verlag.
21.
Hox, J.J. (2002). Multilevel analysis. Techniques and applications . Mahwah, NJ: Lawrence Erlbaum.
22.
James, L.R. (1982). Aggregation bias in estimates of perceptual agreement . Journal of Applied Psychology, 67, 219-229.
23.
Khuri, A.I. (2000). Designs for variance components estimation: Past and present. International Statistical Review, 68, 311-322.
24.
Kim, K.S. (1990). Multilevel data analysis: A comparison of analytical alternatives. Unpublished doctoral dissertation, University of California, Los Angeles.
25.
Kish, L. (1965). Survey sampling. New York: John Wiley.
26.
Kozlowski, S.W., & Klein, K.J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In K. Klein & S. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 3-90). San Francisco: Jossey-Bass.
27.
Kreft, I. (1996). Are multilevel techniques necessary? An overview including simulation studies. Unpublished manuscript, California State University, Los Angeles.
28.
Kreft, I., & de Leeuw, J. (1998). Introducing multilevel modeling. Thousand Oaks, CA: Sage.
29.
Liu, X. (2003). Statistical power and optimum sample allocation ratio for treatment and control having unequal costs per unit of randomization . Journal of Educational and Behavioral Statistics, 28, 231-248.
30.
Liu, X., & Raudenbush, S. (2001, April). Computation of power for 3-level hierarchical linear models. Paper presented at the annual meeting of the American Educational Research Association (AERA) , Seattle, WA.
31.
Longford, N.T. (1993). Random coefficients models. New York: Oxford University Press.
Maas, C.J.M., & Hox, J.J. (2005). Sufficient sample sizes for multilevel modeling . Methodology: European Journal of Research Methods for the Behavioral & Social Sciences, 1, 85-91.
34.
Mok, M. (1995). Sample size requirements for 2-level designs in educational research. Multilevel Modelling Newsletter, 7, 11-15.
35.
Mone, M.A., Mueller, G.C., & Mauland, W. (1996). The perceptions and usage of statistical power in applied psychology and management research. Personnel Psychology , 49, 103-120.
36.
Murphy, K.R. & Myors, B. (2003) Statistical power analysis: A simple and general model for traditional and modern hypothesis tests (2nd ed.). Mahwah, NJ: Lawrence Erlbaum .
37.
Raudenbush, S.W. (1988). Educational applications of hierarchical linear models: A review. Journal of Educational Statistics, 13, 85-116.
38.
Raudenbush, S.W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173-185.
39.
Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
40.
Raudenbush, S.W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5, 199-213.
41.
Raudenbush, S.W., Spybrook, J., Liu, X., & Congdon, R. (2005). Optimal design for longitudinal and multilevel research (Version 1.55) [Computer software]. Chicago: University of Chicago Press.
42.
Reise, S.P., & Duan, N. (2003). Design issues in multilevel studies. In S. P. Reise & N. Duan (Eds.), Multilevel modeling: Methodological advances, issues, and applications (pp. 285-298). Mahwah, NJ: Lawrence Erlbaum.
43.
Rose, R.A., & Bowen, G.L. (2005). Power for sample size in the evaluation of a whole-school educational change initiative. Unpublished manuscript. Retrieved from http://www.uncssp.org/rose_bowen_2005.pdf
44.
Scandura, T.A., & Williams, E.A. (2000). Research methodology in management: Current practices, trends, and implications for future research . Academy of Management Journal, 43, 1248-1264.
45.
Snijders, T.A. (2001). Sampling. In A. H. Leyand & H. Goldstein (Eds.), Multilevel modeling of health statistics (pp. 159-174). New York: John Wiley.
46.
Snijders, T.A. (2005). Power and sample size in multilevel modeling. In B. S. Everitt & D. C. Howell (Eds.), Encyclopedia of statistics in behavioral science (Vol. 3, pp. 1570-1573). Chichester, UK: Wiley.
47.
Snijders, T.A., & Bosker, R.J. (1993). Standard errors and sample sizes for two-level research. Journal of Educational Statistics, 18, 237-259.
48.
Snijders, T.A., & Bosker, R.J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand, Oaks, CA: Sage.
49.
Snijders, T.A., Bosker, R.J., & Guldemond, H. (1996). PINT user's manual (Version 1.6) [Computer program and manual]. Retrieved from http://stat.gamma.rug.nl/snijders/multilevel.htm
50.
Thomas, L., & Krebs, C. (1997). A review of statistical power analysis software . Bulletin of the Ecological Society of America, 78, 126-139.
51.
van der Leeden, R., & Busing, F. (1994). First iteration versus IGLS/RIGLS estimates in two-level models: A Monte Carlo study with ML3 (Preprint PRM 94-03). Leiden, the Netherlands: Leiden University.
52.
Zou, K.H., & Normand, S.L. (2001). On determination of sample size in hierarchical binomial models. Statistics in Medicine, 20, 2163-2182.
