Sources of Validity Evidence for Educational and Psychological Tests: A Follow-Up Study

Abstract

This study followed up on previous work that examined the incidence of reporting evidence based on test consequences in Mental Measurements Yearbook. In the present study, additional possible outlets for what has been called ‘‘consequential validity’’ evidence were investigated, including all articles published in the past 10 years in several applied journals devoted to educational assessment and educational policy, and all presentations at recent annual meetings for the three organizations that sponsor the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education). Consistent with previous findings, consequences of testing as a source of validity evidence is essentially nonexistent in the professional literature and applied measurement and policy work. The article concludes with implications of these findings for extending and refining current validity theory and validation practice.

Keywords

validity consequences consequential validity

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. ( 1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association .

Borsboom, D. , Mellenbergh, G.J. , & van Heerden, J. (2004). The concept of validity . Psychological Review, 111, 1061-1071.

Brennan, R.L. ( 2006). Perspectives on the evolution and future of educational measurement. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 1-16). Westport, CT: Praeger.

Cizek, G.J. ( 2010, March). Error of measurement: Validity and the place of consequences. NCME Newsletter, 18(1), 4-5. Available at http://www.ncme.org/pubs/pdf/vol_18_num_1_v2.pdf

Cizek, G.J. , Rosenberg, S.L. , & Koons, H.H. ( 2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68, 397-412.

Cone, J.D. , & Foster, S.L. ( 1991). Training in measurement: Always the bridesmaid. American Psychologist, 46, 653-654.

Cronbach, L.J. ( 1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 443-507). Washington, DC: American Council on Education.

Ebel, R.L. ( 1961). Must all tests be valid? American Psychologist , 16, 640-647.

Geisinger, K.F. ( 1992). The metamorphosis of test validation. Educational Psychologist, 27, 197-222.

10.

International Reading Association & National Council of Teachers of English. (1994). Standards for the assessment of reading and writing. Newark, DE: International Reading Association.

11.

Kane, M.T. ( 2001). Current concerns in validity theory. Journal of Educational Measurement, 38, 319-342.

12.

Kane, M.T. ( 2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: Praeger .

13.

Lane, S. , & Stone, C.A. ( 2002). Strategies for examining the consequences of assessment and accountability programs. Educational Measurement: Issues and Practice, 21(1), 23-30.

14.

Linn, R. L. (Ed.). (1989). Educational measurement (3rd ed.). New York, NY: Macmillan.

15.

Messick, S. ( 1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York, NY: Macmillan.

16.

Shepard, L.A. ( 1997). The centrality of test use and consequences for test validity. Educational Measurement: Issues and Practice, 16(2), 5-8, 13, 24.

17.

Spies, R. A. , & Plake, B. S. (Eds.). (2005). The sixteenth mental measurements yearbook. Lincoln, NE: Buros Institute of Mental Measurements.

18.

Thompson, B. ( 2003). Guidelines for authors reporting score reliability estimates . In B. Thompson (Ed.), Reliability: Contemporary thinking on reliability issues (pp. 91-102). Thousand Oaks, CA: Sage.

19.

Wainer, H. , & Braun, H. I. (Eds.). (1988). Test validity. Hillsdale, NJ: Lawrence Erlbaum.