Onwuegbuzie and colleagues proposed the Instrument Development and Construct Validation (IDCV) process as a mixed methods framework for creating and validating measures. Examples applying IDCV are lacking. We provide an illustrative case integrating the Rasch model and cognitive interviews applied to the development of the Transformative Experience Questionnaire (TEQ). TEQ measures student engagement with the content they are learning. IDCV process yielded a high degree of utility for informing the psychometric properties of the TEQ. The quantitative data yielded supportive content, construct, and concurrent-related validity evidence using a sample of middle (n = 115) and high school students (n = 81). The qualitative data revealed item rewording was needed. Moderate to strong test–retest reliability was found for both forms.
AlwinD. F. (2001). Research on survey quality. Sociological Methods and Research, 20, 1-39. doi:10.1177/0049124191020001001
2.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC. American Educational Research Association.
3.
AndrichD. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 357-374.
4.
BeghettoR. A. (2010). Prospective teachers’ prior experiences with creativity suppression. International Journal of Creativity and Problem Solving, 20, 29-36. doi:10.1007/s11218-007-9014-2
5.
BondT. G.FoxC. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
6.
BooneW. J.ScantleburyK. (2006). The role of Rasch analysis when conducting science education research utilizing multiple-choice tests. Science Education, 90, 253-269. doi:10.1002/sce.20106
7.
BooneW. J.TownsendJ. S.StarverJ. (2011). Using Rasch theory to guide the practice of survey development and survey data analysis in science education and to inform science reform efforts: An exemplar utilizing STEBI self-efficacy data. Science Education, 95, 258-280. doi:10.1002/sce.20413
8.
CaracelliV. J.GreeneJ. C. (1993). Data analysis strategies for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 15, 195-207. doi:10.3102/01623737015002195
9.
CollinsK. M. T.OnwuegbuzieA. J.SuttonI. L. (2006). A model incorporating the rationale and purpose for conducting mixed methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4, 67-100.
10.
ConnellJ. P. (1990). Context, self, and action: A motivational analysis of self-system processes across the life-span. In CicchettiD.BeeghlyM. (Eds.), The self in transition: From infancy to childhood (pp. 61-97). Chicago, IL: University of Chicago Press.
11.
CreswellJ. W. (2007). Qualitative inquiry and research design: Choosing among the five approaches. Thousand Oaks, CA: Sage.
12.
CsikszentmihalyiM. (1991). Flow: The psychology of optimal experience. New York, NY: Harper Perennial.
13.
DeciE. L.RyanR. M. (1985). Intrinsic motivation and self-determination in human behavior. New York, NY: Plenum.
14.
DeMaioT. J.LandrethA. (2004). Do different cognitive interview techniques produce different results? In PresserS.RothgebJ. M.CouperM. P.LesslerJ. T.MartinE.MartinJ.SingerE. (Eds.), Methods for testing and evaluating survey questionnaires (Chapter 5). Hoboken, NJ: John Wiley. doi:10.1002/0471654728.ch5
15.
DeweyJ. (1958). Art as experience. New York, NY: Capricorn-Putnam. (Original work published 1934)
16.
DoleJ. A.SinatraG. M. (1998). Reconceptualizing change in the cognitive construction of knowledge. Educational Psychologist, 33, 109-128. doi:10.1080/00461520.1998.9653294
17.
DuncanT. G.McKeachieW. J. (2005). The making of the Motivated Strategies for Learning Questionnaire. Educational Psychologist, 40, 117-128.
18.
EggertS.BögenholzS. (2009). Students’ use of decision-making strategies with regard to socioscientific issues: An application of the Rasch partial credit model. Science Education, 94, 230-258. doi:10.1002/sce.20358
19.
FredricksJ. A.BlumenfeldP. C.ParisA. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59-109. doi:10.3102/00346543074001059
20.
GirodM.RauC.SchepigeA. (2003). Appreciating the beauty of science ideas: Teaching for aesthetic understanding. Science Education, 87, 574-587. doi:10.1002/sce.1054
21.
GirodM.TwymanT.WojcikiewiczS. (2010). Teaching and learning science for transformative, aesthetic experience. Journal of Science Teacher Education, 21, 801-824. doi:10.1007/s10972-009-9175-2
22.
GirodM.WongD. (2002). An aesthetic (Deweyan) perspective on science learning: Case studies of three fourth graders. Elementary School Journal, 102, 199-224. doi:10.1086/499700
23.
GreeneB. A. (2015). Measuring cognitive engagement with self-report scales: Reflections from over 20 years of research. Educational Psychologist, 50(1), 14-30. doi:10.1080/00461520.2014.989230
24.
GreeneJ. C.CaracelliV. J.GrahamW. F. (1989). Toward a conceptual framework for mixed-method evaluation design. Educational Evaluation and Policy Analysis, 11, 255-274. doi:10.1057/rlp.2010.6
25.
GrolnickW. S.RyanR. M. (1987). Autonomy in children’s learning: An experimental and individual difference investigation. Journal of Personality and Social Psychology, 52, 890-898. doi:10.1037/0022-3514.52.5.890
26.
GuestG.BunceA.JohnsonL. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18, 59-82. doi:10.1177/1525822X05279903
27.
HeddyB. C.SinatraG. M. (2013). Transforming misconceptions: Using transformative experience to promote positive affect and conceptual change in students learning about biological evolution. Science Education, 97, 723-744. doi:10.1002/sce.21072
28.
Hesse-BiberS. N.LeavyP. (2011). The practice of qualitative research (2nd ed.). Thousand Oaks, CA: Sage.
29.
HidiS.RenningerK. A. (2006). The four-phase model of interest development. Educational Psychologist, 41, 111-127. doi:10.1207/s15326985ep4102_4
30.
HughesK. A. (2004). Comparing pretesting methods: Cognitive interviews, respondent debriefing, and behavior coding (Research Report No. 2004-02). Retrieved from http://www.census.gov/srd/papers/pdf/rsm2004-02.pdf
31.
KarabenickS. A.WoolleyM. E.FriedelJ. M.BridgetV.BlazevskiJ.BonneyC. R.. . . KellyK. L. (2007). Cognitive processing of self-report items in educational research: Do they think what we mean?Educational Psychologist, 42(3), 37-41. doi:10.1080/00461520701416231
32.
LinacreM. J. (1999). Investigating rating scale category utility. Journal of Outcome Measurement, 3, 103-122.
33.
LinacreM. J. (2002). What do infit and outfit, mean-square, and standardized mean?Rasch Measurement Transactions, 16(2), 878.
LuytR. (2012). A framework for mixing methods in quantitative measurement development, validation, and revision: A case study. Journal of Mixed Methods Research, 6(4), 294-316. doi:10.1177/1558689811427912
36.
MasonL.GavaM.BoldrinA. (2008). On warm conceptual change: The interplay of text, epistemological beliefs, and topic interest. Journal of Educational Psychology, 100, 291-309. doi:10.1037/0022-0663.100.2.291
37.
MessickS. (1989). Validity. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 12-103). Old Tappan, NJ: Macmillan.
38.
MessickS. (1995). Validity of psychological assessment: Validation of inferences from persons responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
39.
OnwuegbuzieA. J.LeechN. L.WhitcomeJ. A. (2008). A framework for collecting and analyzing nonverbal communication data in interviews and focus groups in qualitative research. Quality & Quantity: International Journal of Methodology, 42, 75-87. doi:10.1007/s11135-006-9037-3
40.
OnwuegbuzieA. J.DanielL. G.CollinsK. M. T. (2009). A meta-validation model for assessing the score validity of student teacher evaluations. Quality & Quantity: International Journal of Methodology, 43, 197-209. doi:10.1007/s11135-007-9112-4
41.
OnwuegbuzieA. J.BustamanteR. M.NelsonJ. A. (2010). Mixed research as a tool for developing quantitative instruments. Journal of Mixed Methods Research, 4(1), 56-78. doi:10.1177/1558689809355805
42.
PintrichP. R.SmithD. A. F.GarciaT.McKeachieW. J. (1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and Psychological Measurement, 53, 801-813. doi:10.1177/0013164493053003024
43.
Plano ClarkV. L.GarrettA. L.Leslie-PeleckyD. L. (2010). Applying three strategies for integrating quantitative and qualitative databases in a mixed methods study of a nontraditional graduate education program. Field Methods, 22, 154-174. doi:10.1177/1525822X09357174
44.
PughK. J. (2002). Teaching for idea-based, transformative experiences in science: An investigation of the effectiveness of two instructional elements. Teachers College Record, 104, 1101-1137.
45.
PughK. J. (2004). Newton’s laws beyond the classroom walls. Science and Education, 88, 182-196. doi:10.1002/sce.10109
46.
PughK. J. (2011). Transformative experience: An integrative construct in the spirit of Deweyan pragmatism. Educational Psychologist, 46, 107-121. doi:10.1080/00461520.2011.558817
47.
PughK. J.BerginD. A. (2005). The effect of education on students’ out-of-school experience. Educational Researcher, 34, 15-23. doi:10.3102/0013189X034009015
48.
PughK. J.KleshinskiO.LinnenbrinkE. A.FoxC. M. (2004, April). Transformative experiences in science: Using Rasch to develop a quantitative measure. Paper presented at the meeting of the American Educational Research Association Annual Conference, San Diego, CA.
49.
PughK. J.Linnenbrink-GarciaE. A.KoskeyK. L. K.StewartV. C.ManzeyC. (2010a). Motivation, learning, and transformative experience: A study of deep engagement in science. Science Education, 94, 1-28. doi:10.1002/sce.20344
50.
PughK. J.Linnenbrink-GarciaE. A.KoskeyK. L. K.StewartV. C.ManzeyC. (2010b). Teaching for transformative experiences and conceptual change: A case study and evaluation of a high school biology teacher’s experience. Cognition & Instruction, 28, 273-316. doi:10.1080/07370008.2010.490496
51.
PughK.PhillipsM.BergstromC.SextonJ. M.RiggsE. J. (2014, October). Supporting transformative learning in the geosciences: Investigating the relation between the teacher-student dynamic and transformative experience. Paper presented at the annual meeting of the Geological Society of America, Vancouver, British Columbia.
52.
RaschG. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danmarks Paedagogiske Institut.
53.
RaschG. (1980). Probabilistic models for some intelligence and attainment tests (expanded ed.). Chicago, IL: University of Chicago Press.
54.
RyanR. M.DeciE. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54-67. doi:10.1006/ceps.1999.1020
55.
ShiffrinR. M.SchneiderW. (1977). Controlled and automatic human information processing: II. Perceptional learning, automatic attending and a general theory. Psychological Review, 84(2), 127-190.
56.
SondergeldT. A.JohnsonC. C. (2014). Using Rasch measurement for the development and use of affective assessments in science education research. Science Education, 98, 581-613. doi:10.1002/sce.21118
57.
SudmanS.BradburnN.SchwarzN. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass.
58.
TourangeauR.RipsL. J.RasinskiK. (2000). The psychology of survey response. Cambridge, England: Cambridge University Press.
59.
WillisG. B. (2004). Cognitive interviewing revisited: A useful technique, in theory? In PresserS.RothgebJ. M.CouperM. P.LesslerJ. T.MartinE.MartinJ.SingerE. (Eds.), Methods for testing and evaluating survey questionnaires (pp. 23-43). New York, NY: John Wiley & Sons.
60.
WongE. D.PughK. J., & the Dewey Ideas Group at Michigan State University. (2001). Learning science: A Deweyan perspective. Journal of Research on Science Teaching, 38, 317-336.
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
