Mixing Interviews and Rasch Modeling: Demonstrating a Procedure Used to Develop an Instrument That Measures Trust

Abstract

Developing psychometrically sound instruments can be difficult, especially if little is known about the constructs of interest. When constructs of interest are unclear, a mixed methods approach can be useful. Qualitative inquiry can be used to explore a construct’s meaning in a way that informs item writing and allows the strengths of one analysis method to compensate for the weaknesses of the other. Mixing method applications can be complex, however, there are few examples within the literature pertaining to the mix of interviews, Rasch modeling, and classical test theory. This article demonstrates how to mix qualitative inquiry with Rasch modeling (and classical test theory) in order to develop an instrument that measures a complex construct: patient trust.

Keywords

mixed methods Rasch modeling instrument development

Get full access to this article

View all access options for this article.

References

Alexander

L. B.

Luborsky

(1986). The Penn Helping Alliance Scales. In Greenberg

L. S.

Pinsof

W. M.

(Eds.), The psychotherapeutic process: A research handbook (pp. 325-366). New York, NY: Guilford Press.

American Education Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: Author.

Anderson

Dedrick

(1990). Development of the trust in physician scale: A measure to assess interpersonal trust inpatient-physician relationships. Psychological Reports, 67, 1091-1100. doi:10.2466/pr0.1990.67.3f.1091

Baker

(2012). Individual differences in rater decision-making style: An exploratory mixed-methods study. Language Assessment Quarterly, 9, 225-248. doi:10.1080/15434303.2011.637262

Benítez

Padilla

J. L.

(2013). Analysis of nonequivalent assessments across different linguistic groups using a mixed methods approach: Understanding the causes of differential item functioning by cognitive interviewing. Journal of Mixed Methods Research, 8(1), 52-68. doi:10.1177/1558689813488245

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 395-479). Reading, MA: Addison-Wesley.

Brantlinger

Jimenez

Klingner

Pugach

Richardson

(2005). Qualitative studies in special education. Exceptional Children, 71, 195-207. doi:10.1177/001440290507100205

Brewer

Van Raalte

Petitpas

Skylar

Pohlman

Krushell

. . . Weinstock

(2000). Preliminary psychometric evaluation of a measure of adherence to clinic-based sport injury rehabilitation. Physical Therapy in Sport, 1, 68-74. doi:10.1054/ptsp.2000.0019

Cella

Riley

Stone

Rothrock

Reeve

Yount

. . . Hays

(2010). The patient-reported outcome measure information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. Journal of Clinical Epidemiology, 63, 1179-1194. doi:10.1016/j.jclinepi.2010.04.011

10.

Cella

Yount

Rothrock

Gershon

Cook

Reeve

. . . Rose

(2007). The patient reported outcome measure information system (PROMIS): Progress of an NIH roadmap cooperative group during its first two years. Medical Care, 45(5 Suppl. 1), S3-S11. doi:10.1097/01.mlr.0000258615.42478.55

11.

Chen

(2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks, CA: Sage.

12.

Clement

Granquist

M. D.

Arvinen-Barrow

M. M.

(2013). Psychosocial aspects of athletic injuries as perceived by athletic trainers. Journal of Athletic Training, 48, 512-521. doi:10.4085/1062-6050-48.3.21

13.

Costello

A. B.

Osborne

J. W.

(2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10(7), 1-9.

14.

Creswell

J. W.

Plano Clark

V. L.

(2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

15.

Cronbach

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

16.

David

S. L.

(2013). Development and validation of the patient-AT trust instrument (Doctoral dissertation). Ohio University, Athens, Ohio.

17.

Derschied

G. L.

Feiring

D. C.

(1987). A statistical analysis to characterize treatment adherence of the 18 most common diagnoses seen at a sports medicine clinic. Journal of Orthopedic Sports Physical Therapy, 9, 40-46.

18.

Dillman

D. A.

(1978). Mail and telephone surveys: The total design method. New York, NY: John Wiley.

19.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2009). Internet, mail, and mixed-mode surveys: The tailored design approach. Hoboken, NJ: John Wiley.

20.

Ettinger

W. H.

Burns

Messier

Applegate

Rejeski

Morgan

. . . Craven

(1997). A randomized trial comparing aerobic exercise and resistance exercise with a health education program in older adults with knee osteoarthritis: The fitness arthritis and seniors trial (FAST). Journal of the American Medical Association, 277, 25-31. doi:10.1001/jama.1997.03540250033028

21.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299.

22.

Fan

(1998). Item response and classical test theory: An empirical comparison of their item/person statistics. Educational and Psychological Measurement, 58, 357-381. doi:10.1177/0013164498058003001

23.

Glaser

B. G.

Strauss

A. L.

(1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine.

24.

Granquist

Gill

Appaneal

(2010). Development of a measure of rehabilitation adherence for athletic training. Journal of Sports Rehabilitation, 19, 249-267.

25.

Granquist

Podlog

Engel

J. R.

Newland

(2014). Certified athletic trainers’ perspectives on rehabilitation adherence in collegiate athletic training settings. Journal of Sports Rehabilitation, 23, 123-133. doi:10.1123/JSR.2013-0009

26.

Greenfield

Kaplan

S. H.

Ware

J. E.

Yano

E. M.

Frank

H. J.

(1988). Patients’ participation in medical care: Effects on blood sugar control and quality of life in diabetes. Journal of General Internal Medicine, 3, 448-457.

27.

Guest

Bunce

Johnson

(2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18, 59-82.

28.

Hambleton

R. K.

Swaminathan

Rogers

J. H.

(1991). Fundamentals of item response theory. Newbury Park, CA: Sage.

29.

Harrison

D. A.

(1986). Robustness of IRT parameter estimation to violations of the unidimensionality assumption. Journal of Educational Statistics, 11, 91-115. doi:10.1177/01466210122031975

30.

Hays

R. D.

Morales

L. S.

Reise

S. P.

(2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38(Suppl. 9), 1128-1142.

31.

Hitchcock

J. H.

Johanson

(2015). Applying a mixed methods framework to differential item function analyses. Research in the Schools, 22(1), 1-14.

32.

Hitchcock

J. H.

Nastasi

B. K.

Dai

Newman

Jayasena

Bernstein-Moore

. . . Varjas

(2005). Illustrating a mixed-method approach for validating culturally specific constructs. Journal of School Psychology, 43, 259-278.

33.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185.

34.

Johnson

R. B.

Christensen

L. B.

(2012). Educational research: Quantitative, qualitative, and mixed approaches (4th ed.). Thousand Oaks, CA: Sage.

35.

Johnson

R. B.

Onwuegbuzie

A. J.

(2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26. doi:10.3102/0013189X033007014

36.

Kramer

(2011). Using mixed methods to establish the social validity of a self-report assessment: An illustration using the Child Occupational Self-Assessment (COSA). Journal of Mixed Methods Research, 5(1), 52-76. doi:10.1177/1558689810386376

37.

LeCompte

(2000). Analyzing qualitative data. Theory Into Practice, 39, 146-154.

38.

Lee

(2004). Examining passage-related local item dependence (LID) and measurement construct using Q3 statistics in an EFL reading comprehension test. Language Testing, 21, 74-100. doi:10.1191/0265532204lt260oa

39.

Leech

N. L.

Onwuegbuzie

A. J.

(2008). Qualitative data analysis: A compendium of techniques and a framework for selection for school psychology research and beyond. School Psychology Quarterly, 23, 587-604. doi:10.1037/1045-3830.23.4.587

40.

Linacre

(1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328.

41.

Linacre

J. M.

(1999). Investigating rating scale category utility. Journal of Outcome Measurement, 3, 103-122.

42.

Linacre

(2001). Category, step and threshold: Definitions & disordering. Rasch Measurement Transactions, 15(1), 794.

43.

Linacre

J. M.

(2002a). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3, 85-106.

44.

Linacre

J. M.

(2002b). What do infit and outfit, mean-squared and standardized mean? Rasch Measurement Transactions, 16, 878.

45.

Linacre

J. M.

(2011). Winsteps^® (Version 3.70.0) [Computer software]. Beaverton, OR: Winsteps.com.

46.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

47.

Magno

(2009). Demonstrating the difference between classical test theory and item response theory using derived test data. International Journal of Educational and Psychological Assessment, 1(1), 1-11.

48.

Mahler

H. I.

Kulik

J. A.

(1990). Preferences for health care involvement, perceived control and surgical recovery: A prospective study. Social Science & Medicine, 31, 743-751.

49.

Millman

Greene

(1993). The specification and development of tests of achievement and ability. In Linn

R. L.

(Ed.), Educational measurement (3rd ed., pp. 335-366). Phoenix, AZ: Oryx.

50.

Nandakumar

Ackerman

(2004). Test modeling. In Kaplan

D. W.

(Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 93-106). Thousand Oaks, CA: Sage.

51.

Nastasi

B. K.

Hitchcock

Sarkar

Burkholder

Varjas

Jayasena

(2007). Mixed methods in intervention research: Theory to adaption. Journal of Mixed Methods Research, 1(2), 164-182. doi:10.1177/1558689806298181

52.

Nastasi

B. K.

Schensul

S. L.

(2005). Contributions of qualitative research to the validity of intervention research. Journal of School Psychology, 43, 177-195. doi:10.1016/j.jsp.2005.04.003

53.

Newman

Benz

C. R.

(1998). Qualitative-quantitative research methodology: Exploring the interactive continuum. Carbondale: Southern Illinois University Press.

54.

Newman

Lim

Pineda

(2013). Content validity using a mixed methods approach: Its applications and development through the use of a table of specifications methodology. Journal of Mixed Methods Research, 6(3), 243-260. doi:10.1177/1558689813476922

55.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.

56.

O’Connor

B. P.

(2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instrumentation, & Computers, 32, 396-402.

57.

Onwuegbuzie

A. J.

Bustamante

R. M.

Nelson

J. A.

(2010). Mixed research as a tool for developing quantitative instruments. Journal of Mixed Methods Research, 4(1), 56-78.

58.

Patient-Reported Outcome Measurement Information System. (2012). PROMIS^®: Instrument development and psychometric evaluation scientific standards. Retrieved from http://www.nihpromis.org/science/publications

59.

Patton

M. Q.

(2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

60.

Pitney

W. A.

Parker

(2009). Qualitative research in physical activity and the health professions. Champaign, IL: Human Kinetics.

61.

Radwin

L. E.

Washko

Suchy

K. A.

Tyman

(2005). Development and pilot testing of four desired health outcome scales. Oncology Nursing Forum, 32, 92-96.

62.

Stellefson

Hanik

B. W.

Chaney

B. H.

Chaney

J. D.

(2009). Factor retention in EFA: Strategies for health behavior researchers. American Journal of Health Behavior, 33, 587-599.

63.

Tashakkori

Newman

(2010). Mixed methods: Integrating quantitative and qualitative approaches to research. In McGaw

Baker

Peterson

P. P.

(Eds.), International encyclopedia of education (3rd ed., pp. 514-520). Oxford, UK: Elsevier.

64.

Thankachan

Price

Hitchcock

J. H.

(2012, November). Instrument piloting in a cross-cultural context: Lessons learned. Paper presented at the 2009 annual meeting of the Mid-Western Educational Research Association (MWERA), Evanston, IL.

65.

Wang

Wilson

(2005). Exploring local item dependence using a random-effects facet model. Applied Psychological Measurement, 29, 296-318. doi:10.1177/0146621605276281

66.

Warne

McKyer

Smith

(2012). An introduction to item response theory for health behavior researchers. American Journal of Health Behavior, 36, 31-43. doi:10.5993/AJHB.36.1.4

67.

Willis

G. B.

(2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.

68.

Popp

S. O.

DiGangi

Jannasch-Pennell

A. J.

(2007). Assessing unidimensionality: A comparsion of Rasch modeling, parallel analysis, and TETRAD. Practical Assessment, Research & Evaluation, 12(14), 1-18.