This article illustrates the role of group-agreement indices in the process of multilevel construct validation. A two-step process is outlined that includes multitrait-multimethod confirmatory factor analysis (CFA) and the use of a dispersion model to examine correlates of consensus of multilevel constructs. The role of dispersion models for providing discriminant validity evidence for similar constructs, as well as providing insight as to why some teams share perceptions more strongly than others, is illustrated using three efficacy-related constructs.
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191–215.
2.
Bandura, A. (1982). Self-efficacy mechanism in human agency. American Psychologist, 37, 122–147.
3.
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall.
4.
Barrick, M. R., Stewart, G. L., Neubert, M. J., & Mount, M. K. (1998). Relating member ability and personality to work-team processes and team effectiveness. Journal of Applied Psychology, 83, 377–391.
5.
Bar-Tal, D. (1990). Group beliefs: A conception for analyzing group structure, processes, and behavior. New York: Springer-Verlag.
6.
Becker, T. E., & Cote, J. A. (1994). Additive and multiplicative method effects in applied psychological research: An empirical assessment of three models. Journal of Management, 20, 625–641.
7.
Bentler, P. M. (1990). Comparative fix indexes in structural models. Psychological Bulletin, 107, 238–246.
8.
Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606.
9.
Berger, P., & Luckmann, T. (1966). The social construction of reality. New York: Doubleday.
10.
Blau, P. M. (1977). Inequality and heterogeneity. New York: Free Press.
11.
Brown, K. G., & Kozlowski, S. W. J. (1999). Dispersion theory: Moving beyond a dichotomous conceptualization of emergent organizational phenomena. Paper presented at the 14th annual meeting of the Society of Industrial and Organizational Psychology, Atlanta, GA.
12.
Burke, M. J., & Dunlap, W. P. (2002). Estimating interrater agreement with the average deviation index: A user's guide. Organizational Research Methods, 5, 159–172.
13.
Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49–68.
14.
Campion, M. A., Medsker, G. J., & Higgs, A. C. (1993). Relations between work group characteristics and effectiveness: Implication for designing effective work groups. Personnel Psychology, 46, 823–850.
15.
Campion, M. A., Papper, E. M., & Medsker, G. J. (1996). Relations between work team characteristics and effectiveness: A replication and extension. Personnel Psychology, 49, 429–452.
16.
Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234–246.
17.
Conway, J. M. (1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139–162.
18.
Dobbins, G. H., & Zaccaro, S. J. (1986). The effects of group cohesion and leader behavior on subordinate satisfaction. Group and Organization Studies, 11, 203–219.
19.
Driskell, J. E., & Salas, E. (1992). Can you study real teams in contrived settings? The value of small group research to understanding teams. In R. W. Sweeny and E. Salas (Eds.), Teams: Their training and performance (pp. 101–126). Norwood, NJ: Ablex.
20.
Griffin, M. A. (1997). Interaction between individuals and situations: Using HLM procedures to estimate reciprocal relationships. Journal of Management, 23, 759–773.
21.
Gully, S. M., Incalcaterra, K. A., Joshi, A., & Beaubien, J. M. (2002). A meta-analysis of team-efficacy, potency, and performance: Interdependence and level of analysis as moderators of observed relationships. Journal of Applied Psychology, 87, 819–832.
22.
Guzzo, R. A., & Shea, G. P. (1992). Group performance and intergroup relations in organizations. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., pp. 269–313). Palo Alto, CA: Consulting Psychologists Press.
23.
Guzzo, R. A., Yost, P. R., Campbell, R. J., & Shea, G. P. (1993). Potency in groups: Articulating a construct. British Journal of Social Psychology, 32, 87–106.
24.
Harrison, D. A., Price, K. H., & Bell, M. P. (1998). Beyond relational demography: Time and the effects of surface and deep-level diversity on work group cohesion. Academy of Management Journal, 41, 96–107.
25.
Hittner, J. B., May, K., & Silver, N. C. (2003). A monte carlo evaluation of tests for comparing dependent correlations. The Journal of General Psychology, 130, 149–168.
26.
Hysong, S. J., & Quiñones, M. A. (1997). The relationship between self-efficacy and performance: A metaanalysis. Paper presented at the 12th annual conference of the Society for Industrial and Organizational Psychology, St. Louis, MO.
27.
Ilgen, D. R. (1999). Teams embedded in organizations: Some implications. American Psychologist, 54, 129–139.
28.
Jackson, S. E. (1992). Team composition in organizational settings: Issues in managing an increasingly diverse workforce. In S. Worchel, W. Wood, & J. A. Simpson (Eds.), Group process and productivity (pp. 138–173). Newbury Park, CA: Sage.
29.
Jackson, S. E., Brett, J. F., Sessa, V. I., Cooper, D. M., Julin, J. A., & Peyronnin, K. (1991). Some differences make a difference: Individual dissimilarity and group heterogeneity as correlates of recruitment, promotions, and turnover. Journal of Applied Psychology, 76, 675–689.
30.
James, L. R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67, 219–229.
31.
James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.
32.
Janis, I. L. (1972). Victims of groupthink: A psychological study of foreign-policy decisions and fiascos. Boston: Houghton Mifflin.
33.
Janis, I. L. (1982). Groupthink: Psychological studies of policy decisions and fiascoes. Boston: Houghton Mifflin.
34.
Janis, I. L., & Mann, L. (1977). Decision making: A psychological analysis of conflict, choice, and commitment. New York: Free Press.
35.
Jöreskog, K., Sörbom, D., du Toit, S., & du Toit, M. (1999). LISREL 8: New statistical features. Chicago: Scientific Software International.
36.
Joyce, W. F., & Slocum, J. W. (1984). Collective climate: Agreement as a basis for defining aggregate climates in organizations. Academy of Management Journal, 27, 721–742.
37.
Jung, D. I., & Sosik, J. J. (2003). Group potency and collective efficacy: Examining their predictive validity, level of analysis, and effects of performance feedback on future group performance. Group and Organization Management, 28, 366–391.
38.
Kenny, D. A., & Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 12, 165–172.
39.
Kidwell, R. E., Jr., Mossholder, K. W., & Bennett, N. (1997). Cohesiveness and organizational citizenship behavior: A multi-level analysis using work groups and individuals. Journal of Management, 23, 775–793.
40.
Klein, K. J., Conn, A. B., Smith, D. B., & Sorra, J. S. (2001). Is everyone in agreement? An exploration of withingroup agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3–16.
41.
Kozlowski, S. W. J., & Klein, K. J. (2000). A multi-level approach to theory and research in organizations: Contextual, temporal, and emergent processes. In K. J. Klein (Ed.), Multi-level theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 3–90). San Francisco: Jossey-Bass/Pfeiffer.
42.
Kozub, S. A., & McDonnell, J. F. (2000). Exploring the relationship between cohesion and collective efficacy in rugby teams. Journal of Sport Behavior, 23 (2), 120–129.
43.
Krackhardt, D., & Kilduff, M. (1990). Friendship patterns and culture: The control of organizational diversity. American Anthropologist, 92, 142–154.
44.
Lance, C. E., Noble, C. L., & Scullen, S. E. (2002). A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 7, 228–244.
45.
Lee, C., & Bobko, P. (1994). Self-efficacy beliefs: Comparison of five measures. Journal of Applied Psychology, 79, 364–369.
46.
Lindell, M. K., & Brandt, C. J. (1999). Assessing interrater agreement on the job relevance of a test: A comparison of CVI, T, r-sub(WG(J)), and r*-sub(WG(J)) indexes. Journal of Applied Psychology, 84, 640–647.
47.
Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127–135.
48.
Lindsley, D. H., Brass, D. J., & Thomas, J. B. (1995). Efficacy-performance spirals: A multi-level perspective. Academy of Management Review, 20, 645–678.
49.
Marsh, H. W. (1989). Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335–361.
50.
Marsh, H. W., & Bailey, M. (1991). Confirmatory factor analysis of multitrait-multimethod data: A comparison of alternative models. Applied Psychological Measurement, 15, 47–70.
51.
Marsh, H. W., Byrne, B. M., & Craven, R. (1992). Overcoming problems in confirmatory factor analyses of MTMM data: The correlated uniqueness model and factorial invariance. Multivariate Behavioral Research, 27, 489–507.
52.
McGrath, J. E. (1984). Groups: Interactions and performance. Englewood Cliffs, NJ: Prentice Hall.
53.
Medsker, G. J., Williams, L. J., & Holahan, P. J. (1994). A review of current practices of evaluating causal models in organizational behavior and human resources management research. Journal of Management, 20, 439–464.
54.
Mueller, R. O. (1996). Basic principles of structural equation modeling: An introduction to LISREL and EQS. New York: Springer.
55.
Mulvey, P. W., & Klein, H. J. (1998). The impact of perceived loafing and collective efficacy in group goal processes and group performance. Organizational Behavior and Human Decision Processes, 74, 62–87.
56.
O'Reilly, C. A., III, Caldwell, C., & Barnett, D. (1989). Work group demography, social integration, and turnover. Administrative Science Quarterly, 34, 21–37.
57.
Parker, L. E. (1994). Working together: Perceived self and collective-efficacy at the workplace. Journal of Applied Social Psychology, 24, 43–59.
58.
Pearce, C. L., Gallagher, C. A., & Ensley, M. D. (2002). Confidence at the group level of analysis: A longitudinal investigation of the relationship between potency and team effectiveness. Journal of Occupational and Organizational Psychology, 75, 115–119.
59.
Pfeffer, J. (1983). Organizational demography. In L. L. Cummings & B. M. Shaw (Eds.), Research in organizational behavior (Vol. 5, pp. 299–357). Greenwich, CT: JAI Press.
60.
Rentsch, J. R. (1990). Climate and culture: Interaction and qualitative differences in organizational meanings. Journal of Applied Psychology, 75, 668–681.
61.
Riggs, M. L., & Knight, P. A. (1994). The impact of perceived group success-failure on motivational beliefs and attitudes: A causal model. Journal of Applied Psychology, 79, 755–766.
62.
Schneider, B., & Reichers, A. E. (1983). One the etiology of climates. Personnel Psychology, 36, 19–39.
63.
Seers, A. (1989). Team-member exchange quality: A new construct for role-making research. Organizational Behavior and Human Decision Processes, 43, 118–135.
64.
Shea, G. P., & Guzzo, R. A. (1987). Groups as human resources. In K. M. Rowland & G. R. Ferris (Eds.), Research in personnel and human resources management (Vol. 5, pp. 323–356). Greenwich, CT: JAI Press.
65.
Sosik, J. J., Avolio, B. J., & Kahai, S. S. (1997). Effects of leadership style and anonymity on group potency and effectiveness in a group decision support system environment. Journal of Applied Psychology, 82, 89–103.
66.
Spiller, S., Seers, A., & Crown, D. (1996). Group efficacy: A conceptual model for the integration of differing perceptions. Paper presented at Academy of Management meetings, Cincinnati, OH.
67.
Spink, K. S. (1990). Collective efficacy in the sport setting. International Journal of Sport Psychology, 21, 380–395.
68.
Steiger, J. H., & Lind, J. C. (1980). Statistically based tests for the number of factors. Paper presented at the annual spring meeting of the Psychometric Society, Iowa City, IA.
69.
Terborg, J. R., Castore, C., & DeNinno, J. A. (1976). A longitudinal field investigation of the impact of group composition on group performance and cohesion. Journal of Personality and Social Psychology, 34, 782–790.
70.
Tesluk, P., Mathieu, J. E., Zaccaro, S. J., & Marks, M. (1997). Task and aggregation issues in the analysis and assessment of team performance. In M. T. Brannick, E. Salas, & C. Prince (Eds.), Team performance assessment and measurement: Theory, methods, and applications (pp. 197–224). Mahwah, NJ: Lawrence Erlbaum Associates.
71.
Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10.
72.
Vancouver, J. B. (1997). The application of HLM to the analysis of the dynamic interaction of environment, person and behavior. Journal of Management, 23, 795–818.
73.
Weick, K. E. (1979). The social psychology of organizing. Reading, MA: Addison-Wesley.
74.
Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1–26.
75.
Widmeyer, W. N., Brawley, L. R., & Carron, A. V. (1985). The measurement of cohesion in sports teams: The group environment questionnaire. London, ON: Sports Dynamics.
76.
Williams, E. J. (1959). The comparison of regression variables. Journal of the Royal Statistical Society, Series B, 21, 396–399.
77.
Williams, J. P. (1993). The effect of individual versus group efficacy on group task performance. Dissertation Abstracts International, 53 (7-A), 2306–2307.
78.
Williams, K. Y., & O'Reilly, C. A., III (1998). Demography and diversity in organizations: A review of 40 years of research. Research in Organizational Behavior, 20, 70–140.
79.
Wood, R. E., & Locke, E. A. (1987). The relation of self-efficacy and grade goals to academic performance. Educational and Psychological Measurement, 47, 1013–1024.
80.
Zaccaro, S. J., Blair, V., Peterson, C., & Zanis, M. (1995). Collective efficacy. In J. E. Maddux (Ed.), Self-efficacy, adaptation, and adjustment: Theory, research, and application (pp. 305–328). New York: Plenum.
