Quantifying similarity of team mental models: An introduction of the r rg index

Abstract

Previous research has emphasized the importance of shared cognitions such as team mental models (TMM) for high performance in teams. The most prominent approaches to measure TMMs are based on indices of within-group agreement (e.g., r_WG, r*_WG, and a_WG). However, a major shortcoming of these indices is the missing separation of team-specific agreement. Recent indices overestimate the existence of team-specific mental models if there is a common agreement beyond team borders. In this paper, we discuss team situations in which this team specific agreement is most important (e.g., competitive negotiation or action teams) and introduce the TMM index (r_RG) that is based on random group resampling. The advantage of this index is its focus on team specific agreement rather than on common agreement. We demonstrate the usefulness of the r_RG by comparing it to other agreement indices theoretically and by means of Monte Carlo simulation. Finally, limitations and fields of application are discussed.

Keywords

between-group variance team mental models within-group agreement

Get full access to this article

View all access options for this article.

References

Allen

N. J.

Stanley

D. J.

Williams

H. M.

Ross

S. J.

(2007). Assessing the impact of nonresponse on work group diversity effects. Organizational Research Methods, 10, 262–286. doi: 10.1177/1094428106/294731

Biemann

Rack

Ellwart

(2008). Shared mental models in e-sport teams: Antecedents and impact on team performance. Paper presented at the General Online Research Conference (GOR). March 2008, Hamburg.

Bliese

P. D.

(1998). Group size, ICC values, and group-level correlations: A simulation. Organizational Research Methods, 1, 355–373. doi: 10.1177/109442819814001

Bliese

P. D.

(2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In Klein

K. J.

Kozlowski

S. W.

(Eds.), Multilevel theory, research, and methods in organizations (pp. 349–381). San Francisco, CA: Jossey-Bass, Inc.

Bliese

P. D.

Halverson

R. R.

(1996). Individual and nomothetic models of job stress: An examination of work hours, cohesion, and well-being. Journal of Applied Social Psychology, 26, 1171–1189. doi: 10.1111/j.1559-1816.1996.tb02291.x

Bliese

P. D.

Halverson

R. R.

(2002). Using random group resampling in multilevel research: An example of the buffering effects of leadership. Leadership Quarterly, 13, 53–68. doi: 10.1016/S1048-9843(01)00104-7

Bliese

P. D.

Halverson

R. R.

Rothberg

J. M.

(1994). Within-group agreement scores: Using resampling techniques to estimate expected variance. Best Paper Proceedings of the Academy of Management Annual Meeting, pp. 303–307.

Brown

R. D.

Hauenstein

N. M. A.

(2005). Interrater agreement reconsidered: An alternative to the r_wg indices. Organizational Research Methods, 8, 165–184. doi:10.1177/1094428105275376

Burke

M. J.

Dunlap

W. P.

(2002). Estimating interrater agreement with the average deviation index: A user’s guide. Organizational Research Methods, 5, 159–172. doi:10.1177/1094428102005002002

10.

Burke

M. J.

Finkelstein

L. M.

Dusig

M. S.

(1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49–68. doi:10.1177/109442819921004

11.

Burtscher

M. J.

Manser

(2012). Team mental models and their potential to improve teamwork and safety: A review and implications for future research in healthcare. Team Safety Science, 50, 1344–1354. doi:10.1016/j.ssci/2011.12.033

12.

Cannon-Bowers

J. A.

Salas

Converse

(1993). Shared mental models in expert team decision making. In Jr. Castellan

J. N.

(Ed.), Individual and group decision making: Current issues (pp. 221–246). Hillsdale: Lawrence Erlbaum Associates, Inc.

13.

Castro

S. L.

(2002). Data analytic methods for the analysis of multilevel questions: A comparison of intraclass correlation coefficients, r_wg(j), hierarchical linear modelling, within- and between-analysis, and random group resampling. Leadership Quarterly, 13, 69–93. doi:10.1016/S1048-9843(01)00105-9

14.

Cohen

Doveh

Eick

(2001). Statistical properties of the r_WG(J) index of agreement. Psychological Methods, 6, 297–310. doi:10.1037/1082-989X.6.3.297

15.

Cohen

Doveh

Nahum-Shani

(2009). Testing agreement for multi-item scales with the indices r_WG(J) and AD_M(J). Organizational Research Methods, 12, 148–164. doi:10.1177/1094428107300365

16.

Cooke

N. J.

Salas

Cannon-Bowers

J. A.

Stout

R. J.

(2000). Measuring team knowledge. Human Factors, 42, 151–173. doi:10.1518/001872000779656561

17.

DeChurch

L. A.

Mesmer-Magnus

J. R.

(2010a). The cognitive underpinnings of effective teamwork: A meta-analysis. Journal of Applied Psychology, 95, 32–53. doi:10.1037/a0017328

18.

DeChurch

L. A.

Mesmer-Magnus

J. R.

(2010b). Measuring shared team mental models: A meta-analysis. Group Dynamics: Theory, Research, & Practice, 14, 1–14. doi:10.1037/a0017455

19.

Diamantopoulos

Winklhofer

H. M.

(2001). Index construction with formative indicators: An alternative to scale development. Journal of Marketing Research, 38, 269–277.

20.

Dunlap

W. P.

Burke

M. J.

Smith-Crowe

(2003). Accurate tests of statistical significance for r_(WG) and average deviation interrater agreement indexes. Journal of Applied Psychology, 88, 356–362. doi: 10.1037/0021-9010.88.2.356

21.

Eby

L. T.

Meade

A. W.

Parisi

A. G.

Douthitt

(1999). The development of an individual-level teamwork expectations measure and the application of a within-group agreement statistic to assess shared expectations for teamwork. Organizational Research Methods, 2, 366–394. doi: 10.1177/109442819924003

22.

Edwards

B. D.

Day

E. A.

Arthur

Bell

S. T.

(2006). Relationships among team ability composition, team mental models, and team performance. Journal of Applied Psychology, 91, 727–736. doi: 10.1037/0021-9010.91.3.727

23.

Efron

Tibshirani

(1993). An introduction to the bootstrap. Chapman & Hall.

24.

James

D. L.

Demaree

R. G.

Wolf

(1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.

25.

James

L. R.

Demaree

R. G.

Wolf

(1993). r_wg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306–309. doi: 10.1037/0021-9010.78.2.306

26.

Klimoski

R. J.

Mohammed

(1994). Team mental model: Construct or metaphor? Journal of Management, 20, 403–437. doi:10.1016/0149-2063(94)90021-3

27.

Langan-Fox

Code

Langfield-Smith

(2000). Team mental models: Techniques, methods and analytic approaches. Human Factors, 42, 242–271. doi:10.1518/001872000779656534

28.

LeBreton

J. M.

James

L. R.

Lindell

M. K.

(2005). Recent issues regarding r_WG, r*_WG, r_WG(J), and r*_WG(J). Organizational Research Methods, 8, 128–138. doi:10.1177/1094428104272181

29.

LeBreton

J. M.

Senter

J. L.

(2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852. doi:10.1177/1094428106296642

30.

Levesque

L. L.

Wilson

J. M.

Wholey

D. R

. (2001). Cognitive divergence and shared mental models in software development project teams. Journal of Organizational Behavior, 22, 135–144. doi:10.1002/job.87

31.

Lim

B.-C.

Klein

K. J.

(2006). Team mental models and team performance: A field study of the effects of team mental model similarity and accuracy. Journal of Organizational Behaviour, 27, 403–418. doi: 10.1002/job.387

32.

Lindell

M. K.

Brandt

C. J.

(1999). Assessing interrater agreement on the job relevance of a test: A comparison of the CVI, T, r_wg(j), and r*_wg(j) indexes. Journal of Applied Psychology, 84, 640–647. doi: 10.1037/0021-9010.84.4.640

33.

Lindell

M. K.

Brandt

C. J.

Whitney

D. J.

(1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127–135. doi: 10.1177/01466219922031257

34.

MacKenzie

S. B.

Podsakoff

P. M.

Jarvis

C. B.

(2005). The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology, 90, 710–730. doi: 10.1037/0021-9010.90.4.710

35.

Mathieu

J. E.

Heffner

T. S.

Goodwin

G. F.

Cannon-Bowers

J. A.

Salas

(2005). Scaling the quality of teammates’ mental models: Equifinality and normative comparisons. Journal of Organizational Behavior, 26, 37–56. doi:10.1002/job.296

36.

McGraw

K. O.

Wong

S. P.

(1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46. doi:10.1037/1082-989X.1.4.390 (corrected)

37.

Mohammed

Dumville

B. C.

(2001). Team mental models in a team knowledge framework: Expanding theory and measurement across disciplinary boundaries. Journal of Organizational Behavior, 22, 89–106. doi:10.1002/job.86

38.

Mohammed

Ferzandi

Hamilton

(2010). Metaphor no more: A 15-year review of the team mental model construct. Journal of Management, 36(4), 876–910. doi:10.1177/0149206309356804

39.

Mohammed

Klimoski

R. J.

Rentsch

J. R.

(2000). The measurement of team mental models: We have no shared schema. Organizational Research Methods, 3, 123–165. doi:10.1177/109442810032001

40.

Newman

D. A.

Sin

H.-P.

(2009). How do missing data bias estimates of within-group agreement? Sensitivity of SD_WG, CV_WG, r_WG(J), r_WG(J)*, and ICC to systematic nonresponse. Organizational Research Methods, 12, 113–147. doi: 10.1177/1094428106298969

41.

R Development Core Team (2010). R: A language and environment for statistical computing. Vienna, Austria, http://www.R-project.org (ISBN 3- 900051-07-0).

42.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods. Sage Publications.

43.

Rico

Sánchez-Manzanares

Gil

Gibson

(2008). Team implicit coordination processes: A team knowledge-based approach. Academy of Management Review, 33, 163–184. doi:10.5465/AMR.2008.27751276

44.

Rossiter

J. R.

(2002). The C-OAR-SE procedure for scale development in marketing. International Journal of Research in Marketing, 19(4), 305–335. doi: 10.1016/S0167-8116(02)00097-6

45.

Salas

Fiore

S.M.

(Eds.). (2004). Team cognition – Understanding the factors that drive process and performance. Washington, DC: American Psychological Association.

46.

Salas

Fiore

S. M.

Letsky

(2011). Theories of team cognition: Cross-disciplinary perspectives. New York & London: Routledge.

47.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. doi:10.1037/0033-2909.86.2.420

48.

Smith-Jentsch

K. A.

Mathieu

J. E.

Kraiger

(2005). Investigating linear and interactive effects of shared mental models on safety and efficiency in a field setting. Journal of Applied Psychology, 90, 532–535. doi:10.1037/0021-9010.90.3.523

49.

Stout

R. J.

Cannon-Bowers

J. A.

Salas

Milanovich

D. M.

(1999). Planning, shared mental models, and coordinated performance: An empirical link is established. Human Factors, 41, 61–71. doi: 10.1518/001872099779577273

50.

Webber

S. S.

Chen

Payne

S. C.

Marsh

S. M.

Zaccaro

S. J.

(2000). Enhancing team mental model measurement with performance appraisal practices. Organizational Research Methods, 3, 307–322. doi: 10.1177/109442810034001