The measure of within-group agreement most frequently encountered in organizational psychology is the rWG index. The rWG index is determined by comparing the observed group variance among raters with an expected random variance. The most critical issue in calculating the rWG is the choice of an appropriate random distribution that would be expected to follow from raters making their ratings at random. A data-driven approach that uses random-group resampling (RGR) procedures to determine the expected random variance has been proposed. In the present study, the application of the RGR procedure will be illustrated with reference to students' ratings of their mathematics instruction and critically compared with a recently proposed simulation-based approach. It will be shown mathematically that the probability of obtaining statistically significant within-group agreement when applying the RGR procedure strongly depends on the intraclass correlation as well as on the group sizes. Finally, implications for applying the RGR procedure to assess within-group agreement in multilevel data will be discussed.
Baumert, J., Lehmann, R.H., Lehrke, M., Schmitz, B., Clausen, M., Hosenfeld, I., et al. (1997). TIMSS: Mathematisch-Naturwissenschaftlicher Unterricht im internationalen Vergleich. [TIMSS: Mathematics and science instruction in international comparison.]. Opladen, Germany: Leske & Budrich.
2.
Bliese, P.D. (2000). Within-group agreement, nonindependence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass.
3.
Bliese, P.D. (2006). Multilevel modeling in R (2.2): A brief introduction to R, the multilevel package and the NLME package. Washington, DC: Walter Reed Army Institute of Research.
4.
Bliese, P.D., & Halverson, R.R. (2002). Using random group resampling in multilevel research: An example of the buffering effects of leadership climate. Leadership Quarterly, 13, 53-68.
5.
Bliese, P.D., Halverson, R.R., & Rothberg, J.M. (1994). Within-group agreement scores: Using resampling procedures to estimate expected variance. In D. P. Moore (Ed.), Academy of Management Best Paper Proceedings (pp. 303-307). Dallas, TX : Academy of Management.
6.
Burke, M.J., & Dunlap, W.P. (2002). Estimating interrater agreement with the average deviation index: A user's guide. Organizational Research Methods , 5, 159-172.
7.
Burke, M.J., Finkelstein, L.M., & Dusig, M.S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68.
8.
Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
9.
Cohen, A., & Doveh, E. (2005). More on the comparison of intra-class correlation coefficients (ICCs) as measures of homogeneity. In F. Dansereau & F. J. Yammarino (Eds.), Research in multi-level issues (Vol. 4, pp. 455-459). Oxford, UK: Elsevier.
10.
Cohen, A., Doveh, E., & Eick, U. (2001). Statistical properties of the rWG(J) index of agreement. Psychological Methods, 6, 297-310.
11.
Cohen, A., Doveh, E., & Nahum-Shani, I. (in press). Testing agreement for multi-item scales with the indices rWG(J)and ADM(J). Organizational Research Methods.
12.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum .
13.
Dunlap, W.P., Burke, M.J., & Smith-Crowe, K. (2003). Accurate tests of statistical significance for rWG and average deviation interrater agreement indexes. Journal of Applied Psychology, 88, 356-362.
14.
Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
15.
Finn, R.H. (1970). A note on estimating the reliability of categorical data. Educational and Psychological Measurement, 30, 71-76.
George, J.M., & James, L.R. (1993). Personality, affect, and behavior in groups revisited: Comment on aggregation, levels of analysis, and a recent application of within and between analysis. Journal of Applied Psychology, 78, 798-804.
18.
Gonzalez-Roma, V., Peiro, J.M., & Tordera, N. (2002). An examination of the antecedents and moderator influences of climate strength. Journal of Applied Psychology , 87, 465-473.
19.
Good, P.I. (2005). Permutation, parametric, and bootstrap tests of hypotheses. Berlin, Germany: Springer .
20.
Hanges, P.J., & Lyon, J.S. (2005). Interpreting changes in ICCs: To agree or not to agree, that is the question. In F. Dansereau & F. J. Yammarino (Eds.), Research in multi-level issues (Vol. 4, pp. 421-431). Oxford, UK: Elsevier.
21.
Hoyt, W.T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods , 5, 64-86.
22.
James, L.R. (1982). Aggregation bias in estimates of perceptual agreement . Journal of Applied Psychology, 67, 219-229.
23.
James, L.R., Demaree, R.G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology , 69, 85-98.
24.
Klein, K.J., Conn, A.B., Smith, B.D., & Sorra, J.S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3-16.
25.
Lance, C.E., Butts, M.M., & Michels, L.C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods , 9, 202-220.
26.
LeBreton, J.M., Burgess, J.R.D., Kaiser, R.B., Atchley, E.K.P., & James, L.R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar ? Organizational Research Methods, 6, 80-128.
LeBreton, J.M., & Senter, J.L. (in press). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods.
29.
Lindell, M.K. (2001). Assessing and testing interrater agreement on a single target using multi-rating scales. Applied Psychological Measurement, 25, 89-99.
30.
Lindell, M.K., & Brandt, C.J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21, 271-278.
31.
Lindell, M.K., Brandt, C.J., & Whitney, D.J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement , 23, 127-135.
32.
Lüdtke, O., Köller, O., Marsh, H.W., & Trautwein, U. (2005). Teacher frame of reference and the big-fish-littlepond effect. Contemporary Educational Psychology, 30, 263-285.
33.
Lüdtke, O., Trautwein, U., Kunter, M., & Baumert, J. (2006). Reliability and agreement of student ratings of the classroom environment: A reanalysis of TIMSS data. Learning Environments Research, 9, 215-230.
34.
Manoukian, E.E. (1985). Modern concepts and theorems of mathematical statistics . New York: Springer.
35.
Masterson, S.S. (2001). A trickle-down model of organizational justice: Relating employees' and customers' perceptions of and reactions to fairness . Journal of Applied Psychology, 86, 594-604.
36.
McGraw, K.O., & Wong, S.P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
37.
Moritz, S., & Watson, C.B. (1998). Levels of analysis issues in group psychology: Using efficacy as an example of a multilevel model. Group Dynamics: Theory, Research, and Practice, 2, 285-298.
38.
R Development Core Team. (2006). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.
Ryan, A.M., Gheen, M.H., & Midgley, C. (1998). Why do some students avoid asking for help? An examination of the interplay among students' academic efficacy, teachers' social-emotional role, and the classroom goal structure. Journal of Educational Psychology, 90, 528-535.
41.
Schneider, B., White, S.S., & Paul, M.C. (1998). Linking service climate and customer perceptions of service quality: Test of a causal model. Journal of Applied Psychology, 83, 150-163.
42.
Snijders, T.A.B., & Bosker, R.J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
43.
Urdan, T., Midgley, C., & Anderman, E.M. (1998). The role of classroom goal structure in students' use of self-handicapping strategies. American Educational Research Journal, 35, 101-122.
44.
Wong, A.F., Young, D.J., & Fraser, B.J. (1997). A multilevel analysis of learning environments and student attitudes. Educational Psychology, 17, 449-468.
45.
Wooldridge, J.M. (2002). Econometric analysis of cross section and panel data. Cambridge: MIT Press.
