Testing Agreement for Multi-Item Scales With the Indices r WG(J) and AD M(J)

Abstract

The most popular index of agreement has been r_WG(J); more recently, the AD_M(J) index also has been used. This study addresses two problems: first, how to test the statistical significance of r_WG(J) and AD_M(J) and, second, how to infer from the indices that were evaluated for each group about the agreement of the ensemble of groups. The authors extend the inference based on either r_WG(J) or AD_M(J) by focusing on multiple-item scales and on the whole ensemble of groups. Their method is based on simulations, as was done by Dunlap, Burke, and Smith-Crowe (2003) and by Cohen, Doveh, and Eick (2001). The tests are illustrated on the data of Bliese, Halverson, and Schriesheim (2002) pertaining to a sample of 2,042 U.S Army soldiers in 49 U.S. Army companies. Software for our procedures is available both as a SAS code and in the Multilevel Modeling in R package (Bliese, 2006).

Keywords

indices of agreement RWG AD multi-item scale significance tests

Get full access to this article

View all access options for this article.

References

Bliese, P.D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analyses. In K. J. Klein & S. W.J. Kozlowski (Eds.), Multilevel theory, research, and methods in organization: Foundations, extensions, and new directions (pp 349-381) . San Francisco: Jossey-Bass.

Bliese, P.D. (2006). Multilevel modeling in R (2.2): A Brief Introduction to R, the Multilevel Package and the nlme Package. Washington, DC: Walter Reed Army Institute of Research.

Bliese, P.D. , & Halverson, R.R. (1996). Individual and nomothetic models of job stress: An examination of work hours, cohesion, and well-being . Journal of Applied Social Psychology, 26, 1171-1189.

Bliese, P.D. , & Halverson, R.R. (2002). Using random group resampling in multilevel research: An example of the buffering effects of leadership climate. Leadership Quarterly, 13, 53-68.

Bliese, P.D. , Halverson, R.R. , & Rothberg, J.M. (1994). Within-group agreement scores: Using resampling techniques to estimate expected variance. In D. P. Moore (Ed), Best paper proceedings (pp. 303-307). Dallas, TX: Academy of Management.

Bliese, P.D. , Halverson, R.R. , & Rothberg, J.M. (2000). Using random group resampling (RGR) to estimate within-group agreement with examples using the statistical language R. Unpublished working paper, Walter Reed Army Institute.

Bliese, P.D. , Halverson, R.R. , & Schriesheim, C.A. (2002). Benchmarking multilevel methods in leadership: The articles, the model, and the data set. Leadership Quarterly, 13, 3-14.

Brown, R.D. , & Hauenstein, N.M.A. (2005). Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 8, 165-184.

Burke, M.J. , & Dunlap, W.P. (2002). Estimating interrater agreement with the average deviation index: A user's guide. Organizational Research Methods , 5, 159-172.

10.

Burke, M.J. , Finkelstein, L.M. , & Dusig, M.S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68.

11.

Cario, M.C. , & Nelson, B.L. (1997). Modeling and generating random vectors with arbitrary marginal distributions and correlation matrix (Tech. Rep.). Evanston, IL: Northwestern University, Department of Industrial Engineering and Management Sciences.

12.

Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composite models . Journal of Applied Psychology, 83(2), 234-246.

13.

Charnes, J.M. , & Schriesheim, C.A. (1995). Estimation of quantiles for the sampling distribution of the rWG(J) within-groups agreement index . Educational and Psychological Measurement, 55, 435-437.

14.

Chen, G. , Mathieu, J.E. , & Bliese, P. (2004). A framework for conducting multilevel construct validation. In F. J. Dansereau & F. Yammarino (Eds.). Research in multi-level issues: Multi-level issues in organizational behavior and processes (Vol. 3, pp. 273-303). Elsevier: Oxford, UK

15.

Cohen, A. , Doveh, E. , & Eick, U. (2001). Statistical properties of the rWG(J) index of agreement. Psychological Methods, 6, 297-310.

16.

Derogatis, L.R. , & Spencer, P.M. (1982). The Brief Symptom Inventory: Administration, scoring and procedures manual - I. Baltimore: Johns Hopkins School of Medicine, Clinical Psychometrics Research Unit.

17.

Dunlap, W.P. , Burke, M.J. , & Smith-Crowe, K. (2003). Accurate tests of statistical significance for rWG and average deviation interrater agreement indexes. Journal of Applied Psychology, 88, 356-362.

18.

Ehrhart, M.G. (2004).Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Personnel Psychology, 57, 61-94.

19.

George, J.M. (1990). Personality, affect and behavior in groups. Journal of Applied Psychology, 75, 107-116.

20.

Ghosh, S. , & Henderson, S.G. (2002). Chessboard distributions and random vectors with specified marginals and covariance matrix. Operations Research, 50, 820-834.

21.

Gonzalez-Roma, V. , Peiro, J.M. , & Tordera, N. (2002). An examination of the antecedents and moderator influences of climate strength. Journal of Applied Psychology , 87, 465-473.

22.

Harvey, R.J. , & Hollander, E. (2004). Benchmarking rWG interrater agreement indices: Let's drop the .70 rule-of-thumb. Paper presented at the meeting of the Society for Industrial and Organizational Psychology , Chicago.

23.

James, L.R. , Demaree, R.G. , & Wolf, G. (1984). Estimating within groups interrater reliability with and without response bias. Journal of Applied Psychology , 69, 85-98.

24.

Kozlowski, S.W.J. , & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161-167.

25.

Lance, C.E. , Butts, M.M. , & Michels, L.C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods , 9, 202-220.

26.

LeBreton, J.M. , Burgess, J.R.D. , Kaiser, R.B. , Atchley, E.K. , & James, L.R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar ? Organizational Research Methods, 6, 80-128.

27.

LeBreton, J.M. , James, L.R. , & Lindell, M.K. (2005). Recent issues regarding rWG, r*WG, rWG(J), and r*WG(J). Organizational Research Methods, 8, 128-138.

28.

Li, S.T. , & Hammond, J.L. (1975). Generation of pseudorandom numbers with specified univariate distributions and correlation coefficients. IEEE Transactions on Systems, Man and Cybernetics, 5, 557-561.

29.

Lindell, M.K. , & Brandt, C.J. (1997). Assessing interrater agreement on the job relevance of a test: A comparison of the CVI, T, rWG(J), r*WG(J) and indexes. Journal of Applied Psychology, 84, 640-647.

30.

Lindell, M.K. , Brandt, C.J. , & Whitney, D.J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement , 23, 127-135.

31.

Schriesheim, C.A. , Cogliser, C.C. , & Neider, L. (1995). Is it ``trustworthy''? A multi-level of analysis reexamination of an Ohio State leadership study with implications for future research. Leadership Quarterly, 6, 111-145.

32.

Tinsley, H.E.A. , & Weiss, D.J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22, 358-376.