Archival data present opportunities to test important research questions. Unfortunately, archival data do not always include validated measures of the psychological constructs that may be of interest to researchers. In this article, the authors describe how the construct equivalence approach can be applied to organizational archival data to derive and empirically validate surrogate measures of psychological constructs. A step-by-step explanation of how to validate such measures, which includes the use of confirmatory factor analysis to examine the data from a multitrait-multimethod perspective, is described. This approach facilitates the use of organizational archives for research purposes.
Get full access to this article
View all access options for this article.
References
1.
Allen, N. J., & Meyer, J. P. (1990). The measurement and antecedents of affective, continuance, and normative commitment to the organization. Journal of Occupational Psychology, 63, 1-18.
2.
American Psychological Association. (1992). Ethical principles of psychologists and code of conduct. American Psychologist, 47, 1597-1611.
3.
American Psychological Association Committee on Professional Standards and Committee on Psychological Tests and Assessment. (1996). Guidelines for computer based tests and interpretations. Washington, DC: American Psychological Association.
Arthur, W., Jr., Fuentes, R., & Doverspike, D. (1990). Relationships among personnel tests, age, and job performance. Experimental Aging Research, 16(1), 11-16.
6.
Astin, A. W., Korn, W. S., Sax, L. J., & Mahoney, K. M. (1994). The American freshman: National norms for Fall 1994. Los Angeles: University of California, Los Angeles, Higher Education Research Institute.
7.
Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246.
8.
Bentler, P. M. (1995). EQS structural equations program manual. Encino, CA: Multivariate Software.
9.
Bentler, P. M., & Chou, C. P. (1988). Practical issues in structural modeling. In J. S. Long (Ed.), Common problems/proper solutions: Avoiding error in quantitative research (pp. 161-192). Newbury Park, CA: Sage.
10.
Binning, J. F., & Barrett, G. V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478-494.
11.
Bollen, K. A. (1989). A new incremental fit index for general structural equation models. Sociological Methods & Research, 17, 303-316.
12.
Bracken, B. A., & Barona, A. (1992). State of the art procedures for translating, validating, and using psychoeducational tests in cross-cultural assessment. School Psychology International, 12(1-2), 119-132.
13.
Browne, M. W. (1979). The maximum-likelihood solution in interbattery factor analysis. British Journal of Mathematical and Statistical Psychology, 32, 75-86.
14.
Browne, M. W. (1980). Factor analysis of multiple batteries by maximum likelihood. British Journal of Mathematical and Statistical Psychology, 33, 184-199.
15.
Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Newbury Park, CA: Sage.
16.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
17.
Costa, P. T., Jr., & McCrae, R. R. (1985). The NEO Personality Inventory manual. Odessa, FL: Psychological Assessment Resources.
18.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302.
19.
Cudeck, R. (1982). Methods for estimating between-battery factors. Multivariate Behavioral Research, 17, 47-68.
20.
Dooley, D. (2001). Social research methods (4th ed.). Englewood Cliffs, NJ: Prentice Hall.
21.
Eisenberger, R., Huntington, R., Hutchinson, S., & Sowa, D. (1986). Perceived organizational support. Journal of Applied Psychology, 71(3), 500-507.
22.
Evans, K. L. (1997). Estimating personality constructs from archival data (ARI Technical Report 1063). Alexandria, VA: US Army Research Institute for the Behavioral and Social Sciences. (AD A328816)
23.
Finch, J. F., Panter, A. T., & Caskie, G. I. (1999). Two approaches for identifying shared personality dimensions across methods. Journal of Personality, 67, 407-438.
24.
Finch, J. F., & West, S. G. (1997). The investigation of personality structure: Statistical models. Journal of Research in Personality, 31, 439-485.
25.
Gade, P. A., Tiggle, R. B., & Schumm, W. R. (2003). The measurement and consequences of military organizational commitment in soldiers and spouses. Military Psychology, 15, 191-207.
26.
Goodwin, C. J. (2002). Research in psychology: Methods and designs (3rd ed.). New York: John Wiley.
27.
Gowing, M. K., & Lancaster, A. R. (1996). Federal government surveys: Recent practices and future directions. In A. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 360-380). San Francisco: Jossey-Bass.
28.
Harris, B. C., Wochinger, K., Schwartz, J. P., & Parham, L. (1993). Longitudinal research on officer careers: Volume 1. Technical manual for 1988-1992 surveys (ARI Research Product 93-10). Alexandria, VA: US Army Research Institute for the Behavioral and Social Sciences. (AD 273 187)
29.
Hattrup, K., Schmitt, N., & Landis, R. S. (1992). Equivalence of constructs measured by jobspecific and commercially available aptitude tests. Journal of Applied Psychology, 77(3), 298-308.
30.
Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21(5), 967-988.
31.
Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104-121.
32.
Hinkin, T. R., & Tracey, J. B. (1999). An analysis of variance approach to content validation. Organizational Research Methods, 2(2), 175-186.
33.
Hough, L. M., Eaton, N. K., Dunnette, M. D., Kamp, J. D., & McCloy, R. A. (1990). Criterionrelated validities of personality constructs and the effect of response distortion on those validities [Monograph]. Journal of Applied Psychology, 75, 581-595.
34.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria analysis versus new alternatives. Structural Equation Modeling, 6(1), 1-55.
35.
Institutional Research and Analysis Branch. (1990). Class characteristics inventory for the class of 1994. West Pont, NY: U. S. Military Academy.
36.
Jöreskog, K. G. & Sörbom, D (1996). LISREL VIII: User’s reference guide. Chicago: Scientific Software International.
37.
Katzell, R. A. (1994). Contemporary meta-trends in industrial and organizational psychology. In H. Triandis, M. D. Dunnette, & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 4, pp. 1-89). Palo Alto, CA: Consulting Psychologists Press.
38.
Kenny, D. A., & Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.
39.
Leary, M. R. (2001). Introduction to behavioral research methods (3rd ed.). Boston: Allyn & Bacon.
40.
MacCallum, R. (1998). Commentary on quantitative methods in I-O research. Industrial-Organizational Psychologist, 35(4), 19-30.
41.
Marsh, H. W., & Grayson, D. (1995). Latent variable models of multitrait-multimethod data. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues and applications (pp. 177-198). Newbury Park, CA: Sage.
42.
Mathieu, J. E., & Zajac, D. M. (1990). A review and meta-analysis of the antecedents, correlates, and consequences of organizational commitment. Psychological Bulletin, 108(2), 171-194.
43.
McBurney, D. H. (2001). Research methods (5th ed.). Belmont, CA: Wadsworth/Thompson Learning.
44.
Mead, A. D., & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. Psychological Bulletin, 114, 449-458.
45.
Messick, S. (1980). Test validity and the ethics of assessment. American Psychologist, 35, 1012-1027.
46.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749.
47.
Meyer, J. P., & Allen, N. J. (1984). Testing the “Side-Bet Theory” of organizational commitment: Some methodological considerations. Journal of Applied Psychology, 69(3), 372-378.
48.
Meyer, J. P., Allen, N. J., & Gellatly, I. R. (1990). Affective and continuance commitment to the organization: Evaluations of measures and analysis of concurrent and time-lagged relations. Journal of Applied Psychology, 75(6), 710-720.
49.
Meyer, J. P., Stanley, D. J., Herscovitch, L., & Topolnytsky, L. (2002). Affective, continuance, normative commitment to the organization: A meta-analysis of antecedents, correlates, and consequences. Journal of Vocational Behavior, 60(3), 1-33.
50.
Mowday, R. T., Steers, R. M., & Porter, L. W. (1979). The measurement of organizational commitment. Journal of Vocational Behavior, 14, 224-247.
51.
Orvis, B. R., & Gahart, M. T. (1989). Quality-based analysis capability for national youth surveys. Report prepared for the Office of the Assistant Secretary of Defense for Force Management and Personnel (R-3675-FMP). Santa Monica, CA: RAND.
52.
Overton, R. C., Taylor, L. R., Zickar, M. J., & Harms, H. J. (1996). The pen-based computer as an alternative platform for test administration. Personnel Psychology, 49(2), 455-464.
53.
Panter, A. T., Tanaka, J. S., & Hoyle, R. H. (1994). Structural models for multimode designs in personality and temperament research. In C. F. Halverson, G. A. Kohnstamm, & R. P. Martin (Eds.), The developing structure of temperament and personality from infancy to adulthood (pp. 111-138). Hillsdale, NJ: Lawrence Erlbaum.
54.
Sackett, P. R., & Larson, J. R., Jr. (1990). Research strategies and tactics in industrial and organizational psychology. In M. D. Dunnette, & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 1, pp. 419-489). Palo Alto, CA: Consulting Psychologists Press.
55.
Sashkin, M., & Prien, E. P. (1996). Ethical concerns and organizational surveys. In A. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 381-403). San Francisco: Jossey-Bass.
56.
Saucier, G. (1994). Mini-markers: A brief version of Goldberg’s unipolar big-five markers. Journal of Personality Assessment, 63(3), 506-516.
57.
Schwab, D. P. (1980). Construct validity in organizational behavior. Research in Organizational Behavior (Vol. 2, pp. 3-43). Greenwich, CT: JAI.
58.
Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173-180.
59.
Tett, R. P., & Meyer, J. P. (1993). Job satisfaction, organizational commitment, turnover intention, and turnover: Path analyses based on meta-analytic findings. Personnel Psychology, 46(2), 259-293.
60.
Tremble, T. R., Jr., Payne, S. C., Finch, J. F., & Bullis, R. C. (2003). Opening organizational archives to research: Analog measures of organizational commitment. Military Psychology, 15, 167-190.
61.
Turban, D. B., Sanders, P. A., Francis, D. J., & Osburn, H. G. (1989). Construct equivalence as an approach to replacing validated cognitive ability selection tests. Journal of Applied Psychology, 74(1), 62-71.
62.
van Leest, P. F. (1998). Bias and equivalence research in the Netherlands. European Review of Applied Psychology, 47(4), 319-329.
63.
Wanous, J. P., & Hudy, M. J. (2001). Single-item reliability: A replication and extension. Organizational Research Methods, 4(4), 361-375.
64.
Wanous, J. P., Reichers, A. E., & Hudy, M. J. (1997). Overall job satisfaction: How good are single-item measures? Journal of Applied Psychology, 82(2), 247-252.
65.
Webb, E. J., Campbell, D. T., Schwartz, R. D., & Sechrest, L. (1966). Unobtrusive measures: Nonreactive research in the social sciences. Chicago: Rand McNally College.
66.
Wiley, J. W. (1996). Linking survey results to customer satisfaction and business performance. In A. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 330-359). San Francisco: Jossey-Bass.
