Education policy over the past 40 years has focused on the importance of accountability in school improvement. Although much of the scholarly discourse around testing and assessment is technical and statistical, understanding of validity by a non-specialist audience is essential as long as test results drive our educational system and accountability.
Purpose
To portray the roles and functions of testing in U.S. educational contexts and the challenges faced in upholding validity principles in practice.
Research Design
This historical and analytical piece explores the evolution of the concept of validity, its treatment in scholarly discourse, along with challenges faced in practice, and projects future possibilities for educational assessments and their validity.
Conclusions
The challenge is to imagine a fantasy (like a chimera) where technology, globalization, changes in demography, societal expectations, and individual preferences could impact new forms of tests without losing evidence of their technical quality, including validity. A brief argument is made to consider the learning of skills rather than an exclusive focus on content as the core business of assessment.
Get full access to this article
View all access options for this article.
References
1.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2.
American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (1966). Standards for educational and psychological tests and manuals. Washington, DC: American Psychological Association.
3.
BakerE. L. (in press). Learning and assessment: 21st century skills and cognitive readiness. In O'NeilH. F., PerezR. S., & BakerE. L. (Eds.), Teaching and measuring cognitive readiness. New York: Springer.
4.
BakerE. L. (1997, Autumn). Model-based performance assessment. Theory into practice, 36(4), 247–254.
5.
BakerE. L. (2005). Aligning curriculum, standards, and assessments: Fulfilling the promise of school reform. In DwyerC. A. (Ed.), Measurement and research in the accountability era (pp. 315–335). Mahwah, NJ: Erlbaum.
BakerE. L., ChungG. K. W. K., DelacruzG. C., & GriffinN. C. (2012). Engage validation plan (ENGAGE/N00014-12-C-0090/Year 1/Deliverable to DARPA). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
8.
BakerE. L., O'NeilH. F.Jr., & LinnR. L. (1993). Policy and validity prospects for performance-based assessment. American Psychologist, 48(12), 1210–1218.
9.
BennettR. E., & GitomerD. H. (2008). Transforming K-12 assessment: Integrating accountability testing, formative assessment, and professional support. Princeton, NJ: Educational Testing Service.
10.
BloomB. S. (Ed.). (with EngelhartM. D., FurstE. J., HillW. H., & KrathwohlD. R.). (1956). Taxonomy of education objectives: The classification of education goals. Handbook I: Cognitive domain. New York: David McKay.
11.
BrownJ. S. (2012). Cultivating the entrepreneurial learner in the 21st century. Keynote presentation at DML 2012 Conference in San Francisco, CA.
12.
CaiL. (in press). Potential applications of latent variable modeling for the psychometrics of medical simulation. Military Medicine.
13.
CaiL. (2010a). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35, 307–335.
14.
CaiL. (2010b). A two-tier full-information item factor analysis model with applications. Psychometrika, 75, 581–612.
ChungG. K. W. K., DelacruzG. C., DionneG. B., & BewleyW. L. (2003). Linking assessment and instruction using ontologies. Proceedings of the I/ITSEC, 25, 1811–1822.
17.
ConfreyJ. (2011). Presentation at the “Learning Panel,”National Academy of Education, Washington, DC.
18.
CronbachL. J. (1988). Five perspectives on validity argument. In WainerH. & BraunH. I. (Eds.), Test validity (pp. 3–17). Hillsdale, NJ: Erlbaum.
19.
CronbachL. J., & FurbyL. (1970). How we should measure “change”: Or should we?Psychological Bulletin, 70(1), 68–80.
20.
CronbachL. J., & SuppesP. (Eds.). (1969). Research for tomorrow's schools: Disciplined inquiry for education. Stanford, CA: National Academy of Education, Committee on Educational Research. New York: Macmillan.
GlaserR. (1963). Instructional technology and the measurement of learning outcomes: Some questions. American Psychologist, 18, 519–521.
23.
GoodwinL. D., & LeechN. L. (2003). The meaning of validity in the new Standards for Educational and Psychological Testing: Implication for measurement courses. Measurement and Evaluation in Counseling and Development, 36, 181–191.
24.
GordonE. W. (in process). The Gordon Commission on the Future of Assessment in Education.
25.
GordonE. W. (1963). Social status differences: Counseling and guidance for disadvantaged groups. In SchreiberD. (Ed.), Guidance and the school dropout. Washington, DC: National Education Association.
26.
GuilfordJ. P. (1946). New standards for test evaluation. Educational and Psychological Measurement, 6, 427–438.
27.
HarrisC. W. (1963). Problems in measuring change.New York: John Wiley.
28.
HermanJ. L., HeritageM., & GoldschmidtP. (2011). Developing and selecting assessments of student growth for use in teacher evaluation systems. Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
29.
HivelyW.II, PattersonH. L., & PageS. (1968). A “universe defined” system of arithmetic achievement tests. Journal of Educational Measurement, 5, 275–290.
30.
HollandJ. G., & SkinnerB. F. (1961). The analysis of behavior: A program for self-instruction. New York, NY: McGraw-Hill.
IseliM. (2011). Ontology development: Overview and example (Draft CRESST Whitepaper). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
33.
KhatibF., DiMaioF., Foldit Contenters Group, Foldit Void Crushers Group, CooperS., KazmierczykM., … BakerD. (2011). Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural & Molecular Biology. Retrieved from http://www.cs.washington.edu/homes/zoran/NSMBfoldit-2011.pdf
34.
LinnR. L., BakerE. L., & DunbarS. B. (1991). Complex, performance-based assessment: Expectations and validation criteria. Educational Researcher, 20(8), 15–21. (ERIC Document Reproduction Service No. EJ 436 999).
35.
LumsdaineA. A. (1965). Assessing the effectiveness of instructional programs. In GlaserR. (Ed.), Teaching machines and programmed learning. II: Data and directions (pp. 267–320). Washington, DC: National Education Association of the United States.
36.
MacArthur Foundation, Badges for Lifelong Learning Competition (2012). 4th Digital Media Competition.
37.
MessickS. (1989). Validity. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: American Council on Education/Macmillan.
MorrisW. (1981). The American heritage dictionary of the English language. Boston, MA: Houghton Mifflin.
40.
National Governors Association Center for Best Practices, Council of Chief State School Officers. (2010). Common Core State Standards. Washington, DC: Authors. http://www.corestandards.org/the-standards
41.
National Research Council. PellegrinoJ. W., & HiltonM. L. (Eds.). (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Committee on Defining Deeper Learning and 21st Century Skills, Center for Education, Division on Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
42.
No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 C.F.R. (2002).
43.
O'NeilH. F., LangJ. (Y.-C.), & PerezR. S. (in press). What is cognitive readiness. In O'NeilH. F., PerezR. S., & BakerE. L. (Eds.), Teaching and measuring cognitive readiness.New York: Springer.
44.
PentlandA. (2008). Honest signals: How they shape our world.Cambridge, MA: Massachusetts Institute of Technology.
45.
PirsigR. M. (1974). Zen and the art of motorcycle maintenance: An inquiry into values.New York: William Morrow.
46.
PophamW. J. (2001). The truth about testing: An educator's call to action. Alexandria, VA: Association for Supervision and Curriculum Development.
47.
ScrivenM. (1967). The methodology of evaluation. In GagneR. M., TylerR. W., & ScrivenM. (Eds.), Perspectives of curriculum evaluation (American Educational Research Association Monograph Series on Curriculum Evaluation, No. 1, pp. 39–83). Chicago, IL: Rand McNally.
48.
SmithE. R., & TylerR. W. (1942). Appraising and recording student progress. New York, NY: Harper & Row.
49.
ThorndikeE. L. (1904). An introduction to the theory of mental and social measurement.New York, NY: Science Press.
50.
WebbN. L. (2005). Alignment, depth of knowledge, & change. Presentation given at the Florida Educational Research Association 50th Annual Meeting, Miami, FL.
51.
WebbN. M., HermanJ. L., & WebbN. L. (2007). Alignment of mathematics state-level standards and assessments: The role of reviewer agreement. Educational Measurement: Issues and Practice, 26(2), 17–29.
