A Multitrait-Multimethod Validity Investigation of Scores from a Professional Licensure Examination

Abstract

The construct validity of scores from the Uniform CPA Examination was investigated through the construction of a multitrait-multimethod matrix. The four sections of the exam were treated as distinct traits, and within-section subscores were created according to item format. First, Campbell and Fiske’s four criteria were used to evaluate the matrix. After correcting the correlations for attenuation due to unreliability, evidence was moderate for the presence of four discrete constructs within the exam. The monotrait-heteromethod correlations were large and statistically significant and in a majority of cases were larger than other correlations found within the matrix. Structural equation models were fit to the data to explore whether excessive method variance was present. The results indicated that there was very little method variance present in the data. The implications for understanding the constructs measured by this exam, and for evaluating the current practice of reporting separate exam section scores, are discussed.

Get full access to this article

View all access options for this article.

References

Althauser, R. P. , & Heberlein, T. A. (1970). Validity and the multitrait-multimethod matrix. In E. F. Borgatta & G. W. Bohrnsted (Eds.), Sociological methodology, 1970 (pp. 151-169). San Francisco: Jossey-Bass.

American Educational Research Association , American Psychological Association , & National Council on Measurementin Education . (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

American Institute of Certified Public Accountants . (1995). Uniform CPA Examination preparation guide. New York: Author.

American Institute of Certified Public Accountants . (1998). Information for Uniform CPA Examination candidates. New York: Author.

American Institute of Certified Public Accountants Board of Examiners . (1996). 1996 annual report. New York: American Institute of Certified Public Accountants.

Browne, M. W. , & Cudek, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 445-455). Newbury Park, CA: Sage.

Byrne, B. M. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. Hillsdale, NJ: Lawrence Erlbaum.

Campbell, D. T. , & Fiske, D. W. (1959). Convergentand discriminantvalidation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

Egan, K. L. , Sireci, S. G. , & Swaminathan, H. (1998, April). Effect of item bundling on the assessment of test dimensionality. Paper presented at the annual meeting of the National Council on Measurementin Education, San Diego, CA.

10.

Jöreskog, K. G. (1974). Analyzing psychological data by structural analysis of covariance matrices. In R. C. Atkinson , D. H. Krantz , R. D. Luce , & P. Suppes (Eds.), Contemporary developments in mathematical psychology: Measurement, psycho-physics, and neural information processing (Vol. 2, pp. 1-56). San Francisco: Freeman.

11.

Jöreskog, K. G. , & Sörbom, D. (1999). LISREL 8.3 [computer program]. Chicago: Scientific Software.

12.

Lomax, R. G. , & Algina, J. (1979). Comparison of two procedures for analyzing multitrait multimethod matrices. Journal of Educational Measurement, 16, 177-186.

13.

Luzzo, D. A. (1993). A multi-trait, multi-method analysis of three career development measures. Career Development Quarterly, 41, 367-374.

14.

Marsh, H. W. (1989). Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361.

15.

Marsh, H. W. , & Grayson, D. (1995). Latent variable models of multitrait-multimethod data. In R. Hoyle (Ed.), Structural equation modeling: Concepts, issues and applications (pp. 177-198). Thousand Oaks, CA: Sage.

16.

Marsh, H. W. , & Hocevar, D. (1983). Confirmatory factor analysis of multitrait-multimethod matrices. Journal of Educational Measurement, 20, 231-248.

17.

Mulaik, S. A. , James, L. R. , Van Alstine, J. , Bennett, N. , Lind, S. , & Stilwell, C. D. (1989). Evaluation of goodness of fit indices for structural equation models. Psychological Bulletin, 105, 430-445.

18.

Schmitt, N. , Coyle, B. W. , & Saari, B. B. (1977). A review and critique of analyses of multitrait-multimethod matrices. Multivariate Behavioral Research, 12, 447-478.

19.

Sireci, S. G. (1998). The construct of content validity. Social Indicators Research, 45, 83-117.

20.

Sireci, S. G. , & Geisinger, K. F. (1995). Using subject matter experts to assess content representation: A MDS analysis. Applied Psychological Measurement, 19, 241-255.

21.

Sireci, S. G. , & Green, P. C. (2000). Legal and psychometric criteria for evaluating teacher certification tests. Educational Measurement: Issues and Practice, 19(1), 22-31, 34-34.

22.

Steigler, J. H. (1990). Structural equation model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173-180.

23.

Stoker, H. W. , & Impara, J. C. (1995). Basic psychometric issues in licensure testing. In J. C. Impara (Ed.), Licensure testing: Purposes, procedures, and practices (pp. 167-186). Lincoln, NE: Buros Institute of Mental Measurements.