Counseling psychology researchers devote little attention to theory-based measurement validation, as evidenced by cursory mention of validity issues in the method and discussion sections of published research reports. Especially, many researchers appear unaware of the limitations of correlations between pairs of self-report measures as evidence of construct validity. The authors provide an overview of the process of construct validation via user-friendly terminology and examples, with special attention to aspects often neglected in counseling psychology research, including specific recommendations for design and interpretation of multimethod validity investigations.
Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213-232.
2.
American Psychological Association. (1966). Standards for educational and psychological testing (2nd ed.). Washington, DC: Author.
3.
American Psychological Association. (1985). Standards for educational and psychological testing (3rd ed.). Washington, DC: Author.
4.
American Psychological Association. (1999). Standards for educational and psychological testing (4th ed.). Washington, DC: Author.
5.
American Psychological Association. (2003). Guidelines on multicultural education, training, research, practice, and organizational change for psychologists. American Psychologist, 58, 377-402.
6.
Bateson, G. (2000). Steps to an ecology of mind. Chicago: University of Chicago Press.
7.
Butcher, J. N., Graham, J. R., Williams, C. L., & Ben-Porath, Y. S. (1989). Development and use of the MMPI-2 content scales. Minneapolis: University of Minnesota Press.
8.
Campbell, D. T. (1960). Recommendations for APA test standards regarding construct, trait, or discriminant validity. American Psychologist, 15, 546-553.
9.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
10.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed). Hillsdale, NJ: Lawrence Erlbaum.
11.
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Hillsdale, NJ: Lawrence Erlbaum.
12.
Coleman, H. L. K. (1998). General and multicultural counseling competency: Apples and oranges?Journal of Multicultural Counseling & Development, 26, 147-156.
13.
Constantine, M. G. (2001). Predictors of observer ratings of multicultural counseling competence in Black, Latino, and White American trainees. Journal of Counseling Psychology, 48, 456-462.
14.
Constantine, M. G., Gloria, A. M., & Ladany, N. (2002). The factor structure underlying three self-report multicultural counseling competence scales. Cultural Diversity & Ethnic Minority Psychology, 8, 334-345.
15.
Constantine, M. G., & Ladany, N. (2000). Self-report multicultural counseling competence scales: Their relation to social desirability attitudes and multicultural case conceptualization ability. Journal of Counseling Psychology, 47, 155-164.
16.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Dallas, TX: Houghton Mifflin.
17.
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Fort Worth, TX: Holt, Rhinehart & Winston.
18.
Cronbach, L. J. (1947). Test “reliability”: Its meaning and determination. Psychometrika, 12, 1-16.
19.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
20.
Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 443-507). Washington, DC: American Council on Education.
21.
Cronbach, L. J. (1984). Essentials of psychological testing (4th ed.). New York: Harper & Row.
22.
Cronbach, L. J. (1995). Giving method variance its due. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods, and theory: A festschrift honoring Donald W. Fiske (pp. 145-157). Mahwah, NJ: Lawrence Erlbaum.
23.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.
24.
Crowne, D. P., & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349-354.
25.
D’Andrea, M., Daniels, J., & Heck, R. (1991). Evaluating the impact of multicultural counseling training. Journal of Counseling & Development, 70, 143-150.
26.
Dawis, R. V. (1987). Scale construction. Journal of Counseling Psychology, 34, 481-489.
27.
DeShon, R. P. (1998). A cautionary note on measurement error corrections in structural equation models. Psychological Methods, 3, 412-423.
28.
Eid, M., & Diener, E. (2005). Handbook of multimethod measurement in psychology. Washington, DC: American Psychological Association.
29.
Epstein, S. (1979). The stability of behavior: I. On predicting most of the people much of the time. Journal of Personality & Social Psychology, 37, 1097-1126.
30.
Epstein, S. (1983). Aggregation and beyond: Some basic issues on the prediction of behavior. Journal of Personality, 51, 360-392.
31.
Fiske, D. W., & Campbell, D. T. (1992). Citations do not solve problems. Psychological Bulletin, 112, 393-395.
32.
Gottfredson, L. S. (2005). What if the hereditarian hypothesis is true?Psychology, Public Policy, and Law, 11, 311-319.
33.
Guilford, J. P. (1946). New standards for test evaluation. Educational & Psychological Measurement, 6, 427-438.
34.
Helms, J. E., Henze, K. T., Sass, T. L., & Mifsud, V. A. (2006). Treating Cronbach’s alpha reliability coefficients as data in counseling research. The Counseling Psychologist, 34, 630-660.
35.
Henson, R. K. (2006). Effect-size measures and meta-analytic thinking in counseling psychology research. The Counseling Psychologist, 34, 601-629.
36.
Heppner, P. P., & Heppner, M. J. (2004). Writing and publishing your thesis, dissertation, and research. Belmont, CA: Brooks/Cole.
37.
Heppner, P. P., Kivlighan, D. M., Jr., & Wampold, B. E. (1999). Research design in counseling (2nd ed.). Belmont, CA: Brooks/Cole.
38.
Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it?Psychological Methods, 5, 64-86.
39.
Hoyt, W. T., & Kerns, M.-D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403-424.
40.
Hoyt, W. T., & McCullough, M. E. (2005). Issues in the multimodal measurement of forgiveness. In E. L. Worthington Jr. (Ed.), Handbook of forgiveness (pp. 109-123). New York: Brunner-Routledge.
41.
Hoyt, W. T., & Melby, J. N. (1999). Dependability of measurement in counseling psychology: An introduction to generalizability theory. The Counseling Psychologist, 27, 325-352.
42.
Kahn, J. H. (2006). Factor analysis in counseling psychology research, training, and practice: Principles, advances, and applications. The Counseling Psychologist, 34, 684-718.
43.
Kazdin, A. E. (1995). Preparing and evaluating research reports. Psychological Assessment, 7, 228-237.
44.
Kline, R. (1998). Principles and practice of structural equation modeling. New York: Guilford.
45.
LaFromboise, T. D., Coleman, H. L., & Hernandez, A. (1991). Development and factor structure of the Cross-Cultural Counseling Inventory-Revised. Professional Psychology: Research & Practice, 22, 380-388.
46.
Lakes, K. D., & Hoyt, W. T. (in press). Applications of generalizability theory to clinical child psychology research. Journal of Clinical Child and Adolescent Psychology.
47.
Leong, F. T. L., & Brown, M. T. (1995). Theoretical issues in cross-cultural career development: Cultural validity and cultural specificity. In W. B. Walsh & S. H. Osipow (Eds.), Handbook of vocational psychology: Theory, research, and practice (2nd ed., pp. 143-180). Mahwah, NJ: Lawrence Erlbaum.
48.
Loevinger, J. (1957). Objective tests as instruments of psychological theory. Psychological Reports, 3, 635-694.
49.
Martens, M. P., & Haase, R. F. (2006). Advanced applications of structural equation modeling in counseling psychology research. The Counseling Psychologist, 34, 878-911.
50.
McCrae, R. R., & Costa, P. T. (1983). Social desirability scales: More substance than style. Journal of Consulting & Clinical Psychology, 51, 882-888.
51.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: Macmillan.
52.
Mischel, W. (1968). Personality and assessment. New York: John Wiley.
53.
Nauta, M. M., & Epperson, D. L. (2003). A longitudinal examination of the social-cognitive model applied to high school girls’ choices of nontraditional college majors and aspirations. Journal of Counseling Psychology, 50, 448-457.
54.
Nauta, M. M., Kahn, J. H., Angell, J. W., & Cantarelli, E. A. (2002). Identifying the antecedent in the relation between career interests and self-efficacy: Is it one, the other, or both?Journal of Counseling Psychology, 49, 290-301.
55.
Ones, D. S., Viswesvaran, C., & Reiss, A. D. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81, 660-679.
56.
Ponterotto, J. G., Gretchen, D., Utsey, S. O., Rieger, B. P., & Austin, R. (2002). A revision of the Multicultural Counseling Awareness Scale. Journal of Multicultural Counseling & Development, 30, 153-180.
57.
Quintana, S. M., & Minami, T. (2006). Guidelines for meta-analyses of counseling psychology research. The Counseling Psychologist, 34, 839-877.
58.
Quintana, S. M., Troyano, N., & Taylor, G. (2001). Cultural validity and inherent challenges in quantitative methods for multicultural research. In J. G. Ponterotto, J. M. Casas, L. A. Suzuki, & C. M. Alexander (Eds.), Handbook of multicultural counseling (2nd ed., pp. 604-630). Thousand Oaks, CA: Sage.
59.
Rothstein, H. R. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75, 322-327.
60.
Sarnacki, R. E. (1979). An examination of test-wiseness in the cognitive test domain. Review of Educational Research, 49, 252-279.
61.
Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.
62.
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.
63.
Schmidt, F. L., & Hunter, J. E. (2003). History, development, evolution, and impact of validity generalization and meta-analysis methods, 1975-2001. In K. R. Murphy (Ed.), Validity generalization: A critical review (pp. 31-65). Mahwah, NJ: Lawrence Erlbaum.
64.
Schmidt, F. L., Le, H., & Ilies, R. (2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual-differences constructs. Psychological Methods, 8, 206-224.
65.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
66.
Sherry, A. (2006). Discriminant analysis in counseling psychology research. The Counseling Psychologist, 34, 661-683.
67.
Smith, H. M., & Betz, N. E. (2002). An examination of efficacy and esteem pathways to depression in young adulthood. Journal of Counseling Psychology, 49, 438-448.
68.
Sodowsky, G. R., Kuo-Jackson, P. Y., Richardson, M. F., & Corey, A. T. (1998). Correlates of self-reported multicultural competencies: Counselor multicultural social desirability, race, social inadequacy, locus of control racial ideology, and multicultural training. Journal of Counseling Psychology, 45, 256-264.
69.
Sodowsky, G. R., Taffe, R. C., Gutkin, T. B., & Wise, S. L. (1994). Development of the Multicultural Counseling Inventory: A self-report measure of multicultural competencies. Journal of Counseling Psychology, 41, 137-148.
70.
Sternberg, R. J. (1992). Psychological Bulletin’s top 10 “hit parade.” Psychological Bulletin, 112, 387-388.
71.
Sternberg, R. J. (2005). There are no public-policy implications: A reply to Rushton and Jensen (2005). Psychology, Public Policy, and Law, 11, 295-301.
72.
Sternberg, R. J., & Williams, W. M. (1997). Does the Graduate Record Examination predict meaningful success in the graduate training of psychology? A case study. American Psychologist, 52, 630-641.
73.
Sue, D. W., Arredondo, P., & McDavis, R. J. (1992). Multicultural counseling competencies and standards: A call to the profession. Journal of Multicultural Counseling & Development, 20, 64-88.
74.
Tracey, T. J., Glidden, C. E., & Kokotovic, A. M. (1988). Factor structure of the Counselor Rating Form-Short. Journal of Counseling Psychology, 35, 330-335.
75.
Vacha-Haase, T., Henson, R. K., & Caruso, J. C. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational & Psychological Measurement, 62, 562-569.
76.
Westen, D., & Rosenthal, R. (2003). Quantifying construct validity: Two simple measures. Journal of Personality & Social Psychology, 84, 608-618.
77.
Weston, R., & Gore, P. A., Jr. (2006). A brief guide to structural equation modeling. The Counseling Psychologist, 34, 719-751.
78.
Widiger, T. A., & Clark, L. A. (2000). Toward DSM-V and the classification of psychopathology. Psychological Bulletin, 126, 946-963.
79.
Worthington, R. L., Mobley, M., Franks, R. P., & Tan, J. A. (2000). Multicultural counseling competencies: Verbal content, counselor attributions, and social desirability. Journal of Counseling Psychology, 47, 460-468.
80.
Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34, 806-838.
