Researchers in the areas of psychology and education strive to understand the intersections among validity, educational measurement, and cognitive theory. Guided by a mixed model conceptual framework, this study investigates how respondents' opinions inform the validation argument. Validity evidence for a science assessment was collected through traditional paper-and-pencil tests, surveys, and think-aloud and exit interviews of fifth- and sixth-grade students. Item response theory analyses supplied technical descriptions of evidence investigating the internal structure. Surveys provided information regarding perceived item difficulty and fairness. Think-aloud and exit interviews provided context and response processes information to clarify and explain issues. This research demonstrates how quantitative and qualitative data can be used in concert to inform the validation process and highlights the use of think-aloud interviews as an explanatory tool.
American Psychological Association, American Educational Research Association, & National Council on Mathematics Education. ( 1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association .
2.
Angoff, W.H. (1988). Validity: An evolving concept. In H. Wainer & H. I. Braun (Eds.), Test validity. Hillsdale, NJ: Lawrence Erlbaum.
3.
Ayala, C.C., Yin, Y., Shavelson, R., & Vanides, J. (2002, April). Investigating the cognitive validity of science performance assessments with think-alouds: Technical aspects. Paper presented at the annual meeting of the American Educational Research Association, New Orleans.
4.
Boren, M.T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice . IEEE Transactions on Professional Communication, 43, 261-278.
5.
Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-106.
6.
Dellinger, A.B., & Leech, N.L. (2007). Toward a unified validation framework in mixed methods research. Journal of Mixed Methods Research, 1(4), 309-332.
7.
Embretson, S., & Gorin, J. (2001). Improving construct validity with cognitive psychology principles. Journal of Educational Measurement, 38, 343-368.
8.
Ericsson, K.A., & Simon, H.A. (1993). Protocol analysis: Verbal reports as data. London: A Bradford Book, MIT Press.
9.
Floyd, R.G., Phaneuf, R.L., & Wilczynski, S.M. (2005). Measurement properties of indirect assessment methods of functional behavioral assessment: A review of research. School Psychology Review, 34, 58-73.
10.
Full Option Science System. (2005). Environments posttest. Berkeley: Lawrence Hall of Science, Regents of the University of California.
11.
Greene, J.C., & Caracelli, V.J. (1997). Defining and describing the paradigm issue in mixed-method evaluation. In J. C. Greene & V. J. Caracelli (Eds.), Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms (pp. 5-18). San Francisco : Jossey-Bass.
12.
Greene, J.C., Caracelli, V.J., & Graham, W.F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis , 11, 255-274.
13.
Greene, J.C., & McClintock, C. (1985). Triangulation in evaluation: Design and analysis issues. Evaluation Review, 9, 523-545.
14.
Greeno, J.G. (1980). Some examples of cognitive task analysis with instructional implications. In R. E. Snow , P. A. Federico, & W. E. Montague (Eds.), Aptitude, learning and instruction: Vol. 2. Cognitive process analyses of learning and problem solving (pp. 1-21). Hillsdale, NJ: Lawrence Erlbaum.
Howe, K.R. (1988). Against the quantitative-qualitative incompatibility thesis, or, dogmas die hard. Educational Researcher, 17, 10-16.
19.
Johnson, R.B., & Onwuegbuzie, A.J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher , 33, 14-26.
20.
Lawrence Hall of Science. (2003). Assessing science knowledge (ASK): Project summary (Unpublished report). Berkeley: Regents of the University of California.
21.
Leighton, J.P. (2004). Avoiding misconception, misuse, and missed opportunities: The collection of verbal reports on educational achievement testing. Educational Measurement: Issues and Practice, 23, 6-15.
22.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: Macmillan .
23.
Messick, S. (1993). Foundations of validity: Meaning and consequences in psychological assessment. Prince-ton, NJ: Education Testing Service.
24.
Morell, L. (2002). Analysis of the Buck Institute for education's high school economics test. Unpublished manuscript, University of California, Berkeley.
25.
Morell, L. (2008). Contributions of middle grade students to the validation of a national science Assessment study. Middle Grades Research Journal, 3, 1-22.
26.
National Research Council. (2001). Knowing what students know: The science and design of educational assessment . Washington, DC: National Academies Press.
27.
Neilson, J. (1993). Usability engineering. Cambridge, MA: AP Professional.
28.
Onwuegbuzie, A.J., & Johnson, R.B. (2006). The validity issue in mixed research. Research in the Schools, 13, 48-63.
29.
Patton, M.Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA: Sage.
30.
Pellegrino, J.W. (1988). Mental models and mental tests. In H. Wainer & H. I. Brown (Eds.), Test validity (pp. 49-59). Hillsdale, NJ: Lawrence Erlbaum.
31.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danmarks Paedogogiske Institut.
32.
Snow, R.E., & Lohman, D.F. (1989). Implications of cognitive psychology for educational measurement. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 263-331). New York: Macmillan.
33.
Taskakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage.
34.
Taylor, K.L., & Dionne, J.P. (2000). Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing . Journal of Educational Psychology, 92, 413-425.
35.
Tzuriel, D. (2000). Dynamic assessment of young children: Educational and intervention perspectives. Educational Psychology Review , 12, 385-435.
36.
Wilson, M.R. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum .
