Applying the Mixed Methods Instrument Development and Construct Validation Process: The Transformative Experience Questionnaire

Abstract

Onwuegbuzie and colleagues proposed the Instrument Development and Construct Validation (IDCV) process as a mixed methods framework for creating and validating measures. Examples applying IDCV are lacking. We provide an illustrative case integrating the Rasch model and cognitive interviews applied to the development of the Transformative Experience Questionnaire (TEQ). TEQ measures student engagement with the content they are learning. IDCV process yielded a high degree of utility for informing the psychometric properties of the TEQ. The quantitative data yielded supportive content, construct, and concurrent-related validity evidence using a sample of middle (n = 115) and high school students (n = 81). The qualitative data revealed item rewording was needed. Moderate to strong test–retest reliability was found for both forms.

Keywords

Instrument Development and Construct Validation IDCV process transformative experience engagement out-of-school learning motivation Rasch model cognitive interviews

Get full access to this article

View all access options for this article.

References

Alwin

D. F.

(2001). Research on survey quality. Sociological Methods and Research, 20, 1-39. doi:10.1177/0049124191020001001

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC. American Educational Research Association.

Andrich

(1978). A rating formulation for ordered response categories. Psychometrika, 43, 357-374.

Beghetto

R. A.

(2010). Prospective teachers’ prior experiences with creativity suppression. International Journal of Creativity and Problem Solving, 20, 29-36. doi:10.1007/s11218-007-9014-2

Bond

T. G.

Fox

C. M.

(2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.

Boone

W. J.

Scantlebury

(2006). The role of Rasch analysis when conducting science education research utilizing multiple-choice tests. Science Education, 90, 253-269. doi:10.1002/sce.20106

Boone

W. J.

Townsend

J. S.

Starver

(2011). Using Rasch theory to guide the practice of survey development and survey data analysis in science education and to inform science reform efforts: An exemplar utilizing STEBI self-efficacy data. Science Education, 95, 258-280. doi:10.1002/sce.20413

Caracelli

V. J.

Greene

J. C.

(1993). Data analysis strategies for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 15, 195-207. doi:10.3102/01623737015002195

Collins

K. M. T.

Onwuegbuzie

A. J.

Sutton

I. L.

(2006). A model incorporating the rationale and purpose for conducting mixed methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4, 67-100.

10.

Connell

J. P.

(1990). Context, self, and action: A motivational analysis of self-system processes across the life-span. In Cicchetti

Beeghly

(Eds.), The self in transition: From infancy to childhood (pp. 61-97). Chicago, IL: University of Chicago Press.

11.

Creswell

J. W.

(2007). Qualitative inquiry and research design: Choosing among the five approaches. Thousand Oaks, CA: Sage.

12.

Csikszentmihalyi

(1991). Flow: The psychology of optimal experience. New York, NY: Harper Perennial.

13.

Deci

E. L.

Ryan

R. M.

(1985). Intrinsic motivation and self-determination in human behavior. New York, NY: Plenum.

14.

DeMaio

T. J.

Landreth

(2004). Do different cognitive interview techniques produce different results? In Presser

Rothgeb

J. M.

Couper

M. P.

Lessler

J. T.

Martin

Singer

(Eds.), Methods for testing and evaluating survey questionnaires (Chapter 5). Hoboken, NJ: John Wiley. doi:10.1002/0471654728.ch5

15.

Dewey

(1958). Art as experience. New York, NY: Capricorn-Putnam. (Original work published 1934)

16.

Dole

J. A.

Sinatra

G. M.

(1998). Reconceptualizing change in the cognitive construction of knowledge. Educational Psychologist, 33, 109-128. doi:10.1080/00461520.1998.9653294

17.

Duncan

T. G.

McKeachie

W. J.

(2005). The making of the Motivated Strategies for Learning Questionnaire. Educational Psychologist, 40, 117-128.

18.

Eggert

Bögenholz

(2009). Students’ use of decision-making strategies with regard to socioscientific issues: An application of the Rasch partial credit model. Science Education, 94, 230-258. doi:10.1002/sce.20358

19.

Fredricks

J. A.

Blumenfeld

P. C.

Paris

A. H.

(2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59-109. doi:10.3102/00346543074001059

20.

Girod

Rau

Schepige

(2003). Appreciating the beauty of science ideas: Teaching for aesthetic understanding. Science Education, 87, 574-587. doi:10.1002/sce.1054

21.

Girod

Twyman

Wojcikiewicz

(2010). Teaching and learning science for transformative, aesthetic experience. Journal of Science Teacher Education, 21, 801-824. doi:10.1007/s10972-009-9175-2

22.

Girod

Wong

(2002). An aesthetic (Deweyan) perspective on science learning: Case studies of three fourth graders. Elementary School Journal, 102, 199-224. doi:10.1086/499700

23.

Greene

B. A.

(2015). Measuring cognitive engagement with self-report scales: Reflections from over 20 years of research. Educational Psychologist, 50(1), 14-30. doi:10.1080/00461520.2014.989230

24.

Greene

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for mixed-method evaluation design. Educational Evaluation and Policy Analysis, 11, 255-274. doi:10.1057/rlp.2010.6

25.

Grolnick

W. S.

Ryan

R. M.

(1987). Autonomy in children’s learning: An experimental and individual difference investigation. Journal of Personality and Social Psychology, 52, 890-898. doi:10.1037/0022-3514.52.5.890

26.

Guest

Bunce

Johnson

(2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18, 59-82. doi:10.1177/1525822X05279903

27.

Heddy

B. C.

Sinatra

G. M.

(2013). Transforming misconceptions: Using transformative experience to promote positive affect and conceptual change in students learning about biological evolution. Science Education, 97, 723-744. doi:10.1002/sce.21072

28.

Hesse-Biber

S. N.

Leavy

(2011). The practice of qualitative research (2nd ed.). Thousand Oaks, CA: Sage.

29.

Hidi

Renninger

K. A.

(2006). The four-phase model of interest development. Educational Psychologist, 41, 111-127. doi:10.1207/s15326985ep4102_4

30.

Hughes

K. A.

(2004). Comparing pretesting methods: Cognitive interviews, respondent debriefing, and behavior coding (Research Report No. 2004-02). Retrieved from http://www.census.gov/srd/papers/pdf/rsm2004-02.pdf

31.

Karabenick

S. A.

Woolley

M. E.

Friedel

J. M.

Bridget

Blazevski

Bonney

C. R.

. . . Kelly

K. L.

(2007). Cognitive processing of self-report items in educational research: Do they think what we mean? Educational Psychologist, 42(3), 37-41. doi:10.1080/00461520701416231

32.

Linacre

M. J.

(1999). Investigating rating scale category utility. Journal of Outcome Measurement, 3, 103-122.

33.

Linacre

M. J.

(2002). What do infit and outfit, mean-square, and standardized mean? Rasch Measurement Transactions, 16(2), 878.

34.

Linacre

M. J.

(2006). WINSTEPS Rasch measurement computer program. Chicago, IL: MESA Press.

35.

Luyt

(2012). A framework for mixing methods in quantitative measurement development, validation, and revision: A case study. Journal of Mixed Methods Research, 6(4), 294-316. doi:10.1177/1558689811427912

36.

Mason

Gava

Boldrin

(2008). On warm conceptual change: The interplay of text, epistemological beliefs, and topic interest. Journal of Educational Psychology, 100, 291-309. doi:10.1037/0022-0663.100.2.291

37.

Messick

(1989). Validity. In Linn

R. L.

(Ed.), Educational measurement (3rd ed., pp. 12-103). Old Tappan, NJ: Macmillan.

38.

Messick

(1995). Validity of psychological assessment: Validation of inferences from persons responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.

39.

Onwuegbuzie

A. J.

Leech

N. L.

Whitcome

J. A.

(2008). A framework for collecting and analyzing nonverbal communication data in interviews and focus groups in qualitative research. Quality & Quantity: International Journal of Methodology, 42, 75-87. doi:10.1007/s11135-006-9037-3

40.

Onwuegbuzie

A. J.

Daniel

L. G.

Collins

K. M. T.

(2009). A meta-validation model for assessing the score validity of student teacher evaluations. Quality & Quantity: International Journal of Methodology, 43, 197-209. doi:10.1007/s11135-007-9112-4

41.

Onwuegbuzie

A. J.

Bustamante

R. M.

Nelson

J. A.

(2010). Mixed research as a tool for developing quantitative instruments. Journal of Mixed Methods Research, 4(1), 56-78. doi:10.1177/1558689809355805

42.

Pintrich

P. R.

Smith

D. A. F.

Garcia

McKeachie

W. J.

(1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and Psychological Measurement, 53, 801-813. doi:10.1177/0013164493053003024

43.

Plano Clark

V. L.

Garrett

A. L.

Leslie-Pelecky

D. L.

(2010). Applying three strategies for integrating quantitative and qualitative databases in a mixed methods study of a nontraditional graduate education program. Field Methods, 22, 154-174. doi:10.1177/1525822X09357174

44.

Pugh

K. J.

(2002). Teaching for idea-based, transformative experiences in science: An investigation of the effectiveness of two instructional elements. Teachers College Record, 104, 1101-1137.

45.

Pugh

K. J.

(2004). Newton’s laws beyond the classroom walls. Science and Education, 88, 182-196. doi:10.1002/sce.10109

46.

Pugh

K. J.

(2011). Transformative experience: An integrative construct in the spirit of Deweyan pragmatism. Educational Psychologist, 46, 107-121. doi:10.1080/00461520.2011.558817

47.

Pugh

K. J.

Bergin

D. A.

(2005). The effect of education on students’ out-of-school experience. Educational Researcher, 34, 15-23. doi:10.3102/0013189X034009015

48.

Pugh

K. J.

Kleshinski

Linnenbrink

E. A.

Fox

C. M.

(2004, April). Transformative experiences in science: Using Rasch to develop a quantitative measure. Paper presented at the meeting of the American Educational Research Association Annual Conference, San Diego, CA.

49.

Pugh

K. J.

Linnenbrink-Garcia

E. A.

Koskey

K. L. K.

Stewart

V. C.

Manzey

(2010a). Motivation, learning, and transformative experience: A study of deep engagement in science. Science Education, 94, 1-28. doi:10.1002/sce.20344

50.

Pugh

K. J.

Linnenbrink-Garcia

E. A.

Koskey

K. L. K.

Stewart

V. C.

Manzey

(2010b). Teaching for transformative experiences and conceptual change: A case study and evaluation of a high school biology teacher’s experience. Cognition & Instruction, 28, 273-316. doi:10.1080/07370008.2010.490496

51.

Pugh

Phillips

Bergstrom

Sexton

J. M.

Riggs

E. J.

(2014, October). Supporting transformative learning in the geosciences: Investigating the relation between the teacher-student dynamic and transformative experience. Paper presented at the annual meeting of the Geological Society of America, Vancouver, British Columbia.

52.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danmarks Paedagogiske Institut.

53.

Rasch

(1980). Probabilistic models for some intelligence and attainment tests (expanded ed.). Chicago, IL: University of Chicago Press.

54.

Ryan

R. M.

Deci

E. L.

(2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54-67. doi:10.1006/ceps.1999.1020

55.

Shiffrin

R. M.

Schneider

(1977). Controlled and automatic human information processing: II. Perceptional learning, automatic attending and a general theory. Psychological Review, 84(2), 127-190.

56.

Sondergeld

T. A.

Johnson

C. C.

(2014). Using Rasch measurement for the development and use of affective assessments in science education research. Science Education, 98, 581-613. doi:10.1002/sce.21118

57.

Sudman

Bradburn

Schwarz

(1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass.

58.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge, England: Cambridge University Press.

59.

Willis

G. B.

(2004). Cognitive interviewing revisited: A useful technique, in theory? In Presser

Rothgeb

J. M.

Couper

M. P.

Lessler

J. T.

Martin

Singer

(Eds.), Methods for testing and evaluating survey questionnaires (pp. 23-43). New York, NY: John Wiley & Sons.

60.

Wong

E. D.

Pugh

K. J.

, & the Dewey Ideas Group at Michigan State University. (2001). Learning science: A Deweyan perspective. Journal of Research on Science Teaching, 38, 317-336.

61.

Wright

B. D.

Masters

G. N.

(1982). Rating scale analysis. Chicago, IL: MESA Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.13 MB