In multilevel theory testing, estimation of group-level properties (i.e., consensus and diversity) is often complicated by missing data. Researchers are left to draw inferences about group constructs (e.g., organizational climate and climate strength) from the responses of only a subset of group members. This study analyzes the biasing impact of random and non-random missingness patterns on within-group agreement and reliability (standard deviation, coefficient of variation, rWG(J), r*WG(J), ADM, aWG , and intraclass correlation) across a range of response rates, numbers of items, and systematic missing data mechanisms. Results demonstrate biases up to 20% over- or underestimation for common response rates found in organizational research. Correction formulae are presented, which enable assessment of the sensitivity of multilevel results to survey nonresponse.
Austin, J.R. (2003). Transactive memory in organizational groups: The effects of content, consensus, specialization, and accuracy on group performance . Journal of Applied Psychology, 88, 866-878.
3.
Bartko, J.J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83, 762-765.
4.
Blalock, H.M., Jr. (1972). Social statistics (2nd ed.). New York: McGraw-Hill.
5.
Bliese, P.D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349-381). San Francisco, CA: Jossey-Bass.
6.
Bliese, P.D., & Halverson, R.R. (1998a). Group consensus and psychological well-being: A large field study. Journal of Applied Social Psychology, 28, 563-580.
7.
Bliese, P.D., & Halverson, R.R. (1998b). Group size and measures of group-level properties: An examination of eta-squared and ICC values. Journal of Management, 24, 157-172.
8.
Brown, R.D., & Hauenstein, N.M.A. (2005). Interrater agreement reconsidered: An alternative to the rWG indices. Organizational Research Methods, 8, 165-184.
9.
Burke, M.J., & Dunlap, W.P. (2002). Estimating interrater agreement with the average deviation index: A user's guide. Organizational Research Methods , 5, 159-172.
10.
Burke, M.J., Finkelstein, L.M., & Dusig, M.S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68.
11.
Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
12.
Cohen, A., Doveh, E., & Eick, U. (2001). Statistical properties of the rWG(J) index of agreement. Psychological Methods, 6, 297-310.
13.
Collins, L.M., Schafer, J.L., & Kam, C.M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330-351.
14.
Dillman, D.A. (2000). Mail and internet surveys: The tailored design method (2nd ed.). New York: Wiley.
15.
Enders, C.K. (2001). A primer on maximum likelihood algorithms for use with missing data. Structural Equation Modeling, 8(1), 128-141.
16.
Finn, R.H. (1970). A note on estimating the reliability of categorical data. Educational and Psychological Measurement, 10, 3-31.
17.
Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention, and behavior: An introduction to theory and research. Reading, MA: Addison-Wesley.
18.
Goodman, J.S., & Blum, T.C. (1996). Assessing the non-random sampling effects of subject attrition in longitudinal research. Journal of Management , 22, 627-652.
19.
Harrison, D.A. & Klein, K.J. (in press). What's the difference? Diversity constructs as separation, variety, or disparity in organizations. Academy of Management Review.
20.
Harrison, D.A., Price, K.H., & Bell, M.P. (1998). Beyond relational demography: Time and the effects of surface and deep-level diversity on work-group cohesion. Academy of Management Journal, 41, 55-67.
21.
House, R., Rousseau, D.M., & Thomas-Hunt, M. (1995). The meso paradigm: A framework for the integration of micro and macro organizational behavior. In L. L. Cummings & B. M. Staw (Eds.), Research in organizational behavior (vol. 16, pp. 71-114). Greenwich, CT: JAI Press.
22.
James, L.R. (1982). Aggregation bias in estimates of perceptual agreement . Journal of Applied Psychology, 67, 219-229.
23.
James, L.R., Demaree, R.G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology , 69, 85-98.
24.
James, L.R., Demaree, R.G., & Wolf, G. (1993). rWG: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306-309.
25.
James, L.R., Joyce, W., & Slocum, J. (1988). Comment: Organizations do not cognize. Academy of Management Review, 13, 129-132.
26.
Kendall, M.G., & Stuart, A. (1977). The advanced theory of statistics. London: Griffin & Co.
27.
Klein, K.J., Conn, A.B., Smith, B., & Sorra, J.S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3-16.
28.
Klein, J.K., & Kozlowski, S.W.J. (2000). From micro to meso: Critical steps in conceptualizing and conducting multilevel research. Organizational Research Methods, 3(3), 211-236.
29.
Koys, D.J. (2001). The effects of employee satisfaction, organizational citizenship behavior, and turnover on organizational effectiveness: A unit-level, longitudinal study. Personnel Psychology, 54(1), 101-114.
30.
Kozlowski, S.W.J., & Klein, K.J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 3-90). San Francisco: Jossey Bass.
Lindell, M.K., & Brandt, C.J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271-278.
33.
Lindell, M.K., & Brandt, C.J. (1999). Assessing interrater agreement on the job relevance of a test: A comparison of the CVI, T, rWG(J), and r*WG(J) indexes. Journal of Applied Psychology, 84, 640-647.
34.
Lindell, M.K., Brandt, C.J., & Whitney, D.J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement , 23, 127-135.
35.
Little, R.J.A., & Rubin, D.B. (1987). Statistical analysis with missing data. New York: Wiley.
36.
McGraw, K.O., & Wong, S.P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
37.
Newman, D.A. (2003). Longitudinal modeling with randomly and systematically missing data: A simulation of ad hoc, maximum likelihood, and multiple imputation techniques. Organizational Research Methods, 6, 328-362.
38.
Newman, D.A., Hanges, P.J., Duan, L., & Ramesh, A. (in press). A network model of organizational climate: Friendship clusters, subgroup agreement, and climate schemas. In D. B. Smith & H. W. Goldstein (Eds.), A festschrift for Benjamin Schneider. Mahwah, NJ: Lawrence Erlbaum Associates.
39.
Ostroff, C. (1993). Comparing correlations based on individual level and aggregate data. Journal of Applied Psychology, 78, 569-582.
40.
Rogelberg, S.G., Conway, J.M., Sederburg, M.E., Spitzmuller, C., Aziz, S., & Knight, W.E. (2003). Profiling active and passive nonrespondents to an organizational survey. Journal of Applied Psychology, 88, 1104-1114.
41.
Rogelberg, S.G., Luong, A., Sederburg, M.E., & Cristol, D.S. (2000). Employee attitude surveys: Examining the attitudes of noncompliant employees. Journal of Applied Psychology, 85, 284-293.
42.
Roth, P.L., & BeVier, C.A. (1998). Response rates in HRM/OB survey research: Norms and correlates, 1990-1994. Journal of Management, 24, 97-117.
43.
Roth, P.L., Switzer, F.S., & Switzer, D.M. (1999). Missing data in multiple item scales: A Monte Carlo analysis of missing data techniques. Organizational Research Methods, 2, 211-232.
Schafer, J.L., & Graham, J.W. (2002). Missing data: Our view of the state of the art . Psychological Methods, 7, 147-177.
47.
Schneider, B., Ehrhart, M.G., Mayer, D.M., Saltz, J.L., & Niles-Jolly, K. (2005). Understanding organization- customer links in service settings. Academy of Management Journal, 48, 1017-1032.
48.
Schneider, B., Salvaggio, A.N., & Subirats, M. (2002). Climate strength: A new direction for climate research. Journal of Applied Psychology, 87, 220-229.
