Equidistant Response Options on Likert-Type Instruments: Testing the Interval Scaling Assumption Using Mplus

Abstract

The purpose of the present study was to provide the means to evaluate the “interval-scaling” assumption that governs the use of parametric statistics and continuous data estimators in self-report instruments that utilize Likert-type scaling. Using simulated and real data, the methodology to test for this important assumption is evaluated using the popular software Mplus 8.8. Evidence on meeting the assumption is provided using the Wald test and the equidistant index. It is suggested that routine evaluations of self-report instruments engage the present methodology so that the most appropriate estimator will be implemented when testing the construct validity of self-report instruments.

Keywords

Likert-type interval scaling survey assumptions Mplus

Get full access to this article

View all access options for this article.

References

Akdogan

Turkum

A. S.

(2018). Insight scale for nonclinical university students: Validity and reliability analysis. Measurement and Evaluation in Counseling and Development, 51, 250–262.

Aldawsari

Laux

Dari

Gaballa

(2021). The revised Arabic Schwartz outcome scale-10 (SOS-10-AR). Measurement and Evaluation in Counseling and Development, 54, 120–129.

Andrich

(1978a). A rating formulation for ordered response categories. Psychometrika, 43(4), 561–573.

Andrich

(1978a). A rating formulation for ordered response categories. Psychometrika, 43(4), 561–573.

Andrich

(1978b). Scaling attitude items constructed and scored in the Likert tradition. Educational and Psychological Measurement, 38, 665–680. https://doi.org/10.1177/001316447803800308

Arnold

M. E.

Nott

B. D.

Meinhold

J. L.

(2012). The Positive Youth Development Inventory Full Version. © Oregon State University.

Autin

Blustein

Duffy

Gensmer

Douglass

Allan

(2019). The development and initial validation of need satisfaction scales within the psychology of working theory. Journal of Counseling Psychology, 66, 195–209.

Babakus

Ferguson

C. E. J.

Jöreskog

K. G.

(1987). The sensitivity of confirmatory maximum likelihood factor analysis to violations of measurement scale and distributional assumptions. Journal of Marketing Research, 24, 222–228.

Bardoshi

Erford

Jang

(2019). Psychometric synthesis of the counselor burnout inventory. Journal of Counseling & Development, 97, 195–208.

10.

Barnette

(2000). Effects of stem and Likert response option reversals on survey internal consistency: If you feel the need, there is a better alternative to using those negatively worded stems. Educational and Psychological Measurement, 60, 361–370.

11.

Bass

B. M.

Cascio

W. F.

O’Connor

E. J.

(1974). Magnitude estimations of expressions of frequency and amount. Journal of Applied Psychology, 59(3), 313–320.

12.

Beauducel

Herzberg

P. Y.

(2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13, 186–203.

13.

Blau

Dimino

(2019). Prepared for counseling: Introducing a short scale and correlates. Measurement and Evaluation in Counseling and Development, 52, 274–283.

14.

Bloom

Dillman-Taylor

(2020). The online dating intensity scale: Exploratory factor analysis in a sample of emerging adults. Measurement and Evaluation in Counseling and Development, 53, 1–16.

15.

Bollen

K. A.

Barb

K. H.

(1981). Pearson’s R and coarsely categorized measures. American Sociological Review, 46, 232–239.

16.

Bolt

D. M.

Johnson

T. R.

(2009). Addressing score bias and differential item functioning due to individual differences in response style. Applied Psychological Measurement, 33(5), 335–352. https://doi.org/10.1177/0146621608329891

17.

Borgers

Hox

Sikkel

(2003). Response quality in survey research with children and adolescents: The effect of labeled response options and vague quantifiers. International Journal of Public Opinion Research, 15(1), 83–94.

18.

Brislin

W. R.

(1970). Back-translation for cross-cultural research. Journal of Cross-Cultural Psychology, 1, 185–216.

19.

Cho

Choi

Kim

Hong

(2018). Factor structure and other psychometric properties of the social phobia inventory in Korean samples. Measurement and Evaluation in Counseling and Development, 51, 263–280.

20.

Cimsir

Akdogan

(2021). Childhood emotional incest scale (CEIS): Development, cross-validation and reliability. Journal of Counseling Psychology, 68, 98–111

21.

Coleman

(2013). Effects of negative keying and wording in attitude measures: A mixed-methods study [Unpublished doctoral dissertation, James Madison University].

22.

Corwyn

(2000). The factor structure of global self-esteem among adolescents and adults. Journal of Research in Personality, 34, 357–379.

23.

Dillman

Bratton

Henson

(2019). Confirming the constructs of Adlerian personality priority assessment. Measurement and Evaluation in Counseling and Development, 52, 191–206.

24.

DiStefano

(2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9, 327–346.

25.

Dolan

C. V.

(1994). Factor analysis of variables with 2, 3, 5, and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47, 309–326.

26.

Dorans

N. J.

Holland

P. W.

(1992, October). DIF detection and description: Mantel-Haenszel and standardization [Paper presentation]. Presented at the educational testing service/AFHRL conference, Princeton, NJ, United States.

27.

Erford

Sriken

Sherman

Hibbs

Smith

Kipper-Smith

Niarhos

(2021). Psychometric analysis, internal structure, and measurement invariance of the alcohol use disorders identification test (AUDIT) scores from a large university sample. Measurement and Evaluation in Counseling and Development, 54, 188–205.

28.

Forero

C. G.

Maydeu-Olivares

Gallardo-Pujol

(2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling, 16, 625–641.

29.

Zhang

(2019). Developing and validating the career personality styles inventory. Measurement and Evaluation in Counseling and Development, 52, 38–51.

30.

Gaito

(1980). Measurement scales and statistics: Resurgence of an old misconception. Psychological Bulletin, 87(3), 564–567.

31.

Ganho-Avila

Moura-Ramos

Goncalves

Almeida

(2019). Measuring vulnerability to anxiety: Factorial structure, reliability, validity, and discriminatory accuracy of the anxiety sensitivity index-3-pt. Measurement and Evaluation in Counseling and Development, 52, 223–238.

32.

Garifio

Perla

(2007). Ten common misunderstandings, misconceptions, persistent myths and urban legends about Likert scales and Likert response formats and their antidotes. Journal of Social Sciences, 33(3), 106–116.

33.

Ghabrial

Andersen

(2021). Development and initial validation of the queer people of color identity affirmation scale. Journal of Counseling Psychology, 68, 38–53.

34.

Ghosh

Niileksela

Parham

Janis

(2021). Investigating factorial invariance of the counseling center assessment of psychological symptoms-34 (CCAPS-34) with military and nonmilitary students. Measurement and Evaluation in Counseling and Development, 54, 42–55.

35.

Gonzalez

Sperandio

Mullen

Tuazon

(2021). Development and initial testing of the multidimensional cultural humility scale. Measurement and Evaluation in Counseling and Development, 54, 56–70.

36.

Green

S. B.

Akey

T. M.

Fleming

K. K.

Hershberger

S. L.

Marquis

J. G.

(1997). Effect of the number of scale points on chi-square fit indices in confirmatory factor analysis. Structural Equation Modeling, 4, 108–120.

37.

Greene

(2019). The multicultural school counseling behavior scale: Development, psychometrics and use. Professional School Counseling, 22(1), 1–10.

38.

Griffin

Worthington

Hook

Davis

Maguen

(2018). Development of the self-forgiveness dual-process scale. Journal of Counseling Psychology, 65, 715–726.

39.

Halamova

Kanovsky

Petrocchi

Moreira

Lopez

Bamett

Yang

Benda

Brahler

Zeng

Zenger

(2021). Factor structure of the self-compassion scale in 11 international samples. Measurement and Evaluation in Counseling and Development, 54, 1–23.

40.

Henkel

R. E.

(1975). Part-whole correlations and the treatment of ordinal and quasi interval data as interval data. The Pacific Sociological Review, 18(1), 3–26. https://doi.org/10.2307/1388217

41.

Hensler

Stipak

(1979). Estimating interval scale values for survey item response categories. American Journal of Political Science, 23(3), 627–649. https://doi.org/10.2307/2111033

42.

Hiles

Dandy

Martino

Howard

Cross

(2019). An evaluation of the psychometric properties of the attachment behavior checklist. Measurement and Evaluation in Counseling and Development, 52, 171–190.

43.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 129–145). Lawrence Erlbaum.

44.

Hutchinson

S. R.

Olmos

(1998). Behavior of descriptive fit indexes in confirmatory factor analysis using ordered categorical data. Structural Equation Modeling: A Multidisciplinary Journal, 5(4), 344–364. https://doi.org/10.1080/10705519809540111

45.

Jakobsson

(2004). Statistical presentation and analysis of ordinal data in nursing research. Scandinavian Journal of Caring Sciences, 18, 437–440.

46.

Jamieson

(2004). Likert scales: How to (ab)use them. Medical Education, 38, 1212–1218.

47.

Jiang

Wang

Weiss

D. J.

(2016) Sample Size Requirements for Estimation of Item Parameters in the Multidimensional Graded Response Model. Frontiers in Psychology, 7, 109. doi: 10.3389/fpsyg.2016.00109

48.

Johnson

D. R.

Creech

J. C.

(1983). Ordinal measures in multiple indicator models: A simulation study of categorization error. American Sociological Review, 48, 398–407.

49.

Johnson

Karcher

(2019). Validity evidence for a state-specific assessment of in-session counselor empathy: The state-empathic concern scale. Measurement and Evaluation in Counseling and Development, 52, 284–296.

50.

Johnson

Knight

McHugh

(2021). Score reliability and validity evidence for the state-interpersonal reactivity index: A multidimensional assessment of in-session counselor empathy. Measurement and Evaluation in Counseling and Development, 54, 24–41.

51.

Kim

Wang

(2021). Examination of the identify style inventory with racial/ethnic minority college students: Revision of factor structure and psychometrics. Measurement and Evaluation in Counseling and Development, 54, 173–187.

52.

Kivlighan

D. M.

Adams

Tao

Drinane

Owen

(2019). Construction and validation of the multicultural orientation inventory-group version. Journal of Counseling Psychology, 66, 45–55.

53.

Knapp

T. R.

(1990). Treating ordinal scales as interval scales: An attempt to resolve the controversy. Nursing Research, 39(2), 121–123. https://doi.org/10.1097/00006199-199003000-00019

54.

Krosnick

J. A.

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.

55.

Lau

Vallett

(2019). The counseling training environment scale: Initial development and validity of a self-report measure to assess the counseling training environment. Measurement and Evaluation in Counseling and Development, 52, 255–273.

56.

Lee

Gaskin-Wasson

Jones

Harrell

Banks

Kohn-Wood

Sellers

Neblett

(2019). The daily life experiences scale: Factor structure, reliability, validity, and measurement invariance for African American males and females. Measurement and Evaluation in Counseling and Development, 54, 206–218.

57.

Lee

Shin

(2019). Experience in close relationships scale-short version (ECR-S) validation with Korean college students. Measurement and Evaluation in Counseling and Development, 52, 119–127.

58.

Levant

Parent

(2019). The development and evaluation of a brief short form of the normative male alexithymia scale (NMAS-BF). Journal of Counseling Psychology, 66, 224–233.

59.

Levant

Parent

Mahalik

McDermott

Alshabani

Hammer

(2020). Development and evaluation of a new short form of the conformity to masculine norms inventory (CMNI-30). Journal of Counseling Psychology, 67, 622–636.

60.

Likert

(1932). The method of constructing an attitude scale. In Houston

S. R.

Schmid

Lynch

Duff

Jr. . (Eds.), Methods and techniques in business research (pp. 53–65). MSS Information Corporation.

61.

Lim

Kim

(2020). A validation of a multicultural competency measure among South Korean counselors. Journal of Multicultural Counseling & Development, 48, 15–29.

62.

Liu

Maffini

Iwamoto

Wong

Y. J.

Mitts

(2018). Gendered racism scales for Asian American men: Scale development and psychometric properties. Journal of Counseling Psychology, 65, 556–570.

63.

Lin

P. Y.

Lin

Y. C.

(2013). Examining student factors in sources of setting accommodation DIF. Educational and Psychological Measurement, 74(5), 759–794.

64.

Woo

Huffman

(2018). Spiritual competency scale: A confirmatory factor analysis. Measurement and Evaluation in Counseling and Development, 51, 219–234.

65.

Ludlow

Matz-Costa

Klein

(2019). Enhancement and validation of the productive engagement portfolio-scenario (PEP-S8) scales. Measurement and Evaluation in Counseling and Development, 52, 15–37.

66.

Luo

Watson

Lenz

(2021). Development and initial validation of a social media identity distress scale among emerging adults. Measurement and Evaluation in Counseling and Development, 54, 141–155.

67.

MacCallum

R. C.

Browne

M. W.

Sugawara

H. M.

(1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130–149. https://doi.org/10.1037/1082-989X.1.2.130

68.

Martens

M. P.

(2005). The use of structural equation modeling in counseling psychology research. The Counseling Psychologist, 33(3), 269–298.

69.

Martin

Zamboanga

Haase

Buckner

(2020). Measurement invariance of the protective behavioral strategies scale across racial groups. Measurement and Evaluation in Counseling and Development, 53, 17–33.

70.

Mazahreh

Stoltz

Wolff

(2019). Validation of the career adapt-abilities scale in the Hashemite kingdom of Jordan. Measurement and Evaluation in Counseling and Development, 52, 108–118.

71.

Moate

Gnilka

West

Rice

(2019). Doctoral student perfectionism and emotional well-being. Measurement and Evaluation in Counseling and Development, 52, 145–155.

72.

Muraki

(1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.

73.

Muthen

B. O.

Kaplan

(1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38, 171–189.

74.

Muthén

L. K.

Muthén

B. O.

(1998–2018). Mplus user’s guide.

75.

Nowlis

S. M.

Kahn

B. E.

Dhar

(2002). Coping with ambivalence: The effect of removing a neutral option on consumer attitude and preference judgments. Journal of Consumer Research, 29, 319–334.

76.

Nunnally

J. C.

(1978). Psychometric theory (2nd ed.). McGraw-Hill.

77.

O’Brien

R. M.

(1979). The use of Pearson’s r with ordinal data. American Sociological Review, 44(5), 851–857. https://doi.org/10.2307/2094532

78.

Butler-Shillingford

(2019). The client assessment of multicultural competent behavior (CAMCB): Development and validation. Measurement and Evaluation in Counseling and Development, 54, 71–89.

79.

Pederson

Gorman-Ezell

Mayer

Brookings

(2021). Development and preliminary validation of a tool for screening suicide risk in chronically III women. Measurement and Evaluation in Counseling and Development, 54, 130–140.

80.

Perez-Rojas

Lockard

Bartholomew

Gonzalez

(2019). Development and initial validation of the therapist cultural comfort scale. Journal of Counseling Psychology, 66, 534–549.

81.

Pohl

N. F.

(1981). Scale considerations in using vague quantifiers. The Journal of Experimental Education, 49(4), 235–240. https://doi.org/10.1080/00220973.1981.11011790

82.

Potthast

M. J.

(1993). Confirmatory factor analysis of ordered categorical variables with large models. British Journal of Mathematical and Statistical Psychology, 46, 273–286.

83.

Poynton

Ruiz

Lapan

(2019). Development and validation of the college admissions knowledge evaluation. Professional School Counseling, 22(1b), 1–6.

84.

Pozza

Barcaccia

Dettoer

(2019). Psychometric evaluation of the Italian obsessive compulsive intentory-child version: Factor structure and predictive validity at one-year follow-up in adolescents. Measurement and Evaluation in Counseling and Development, 52, 239–254.

85.

Rhemtulla

Brosseau-Liard

P. É.

Savalei

(2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354–373. https://doi.org/10.1037/a0029315

86.

Rigdon

E. E.

Fergusson

C. E.

(1991). The performance of the polychoric correlation coefficient and selected fitting functions in confirmatory factor analysis with ordinal data. Journal of Marketing Research, 28, 491–497.

87.

Rowan-Kenyon

McCready

Aleman

Barone

(2021). Measuring racist aggressions on social media and the effects on U.S. college students of color: An instrument validation. Measurement and Evaluation in Counseling and Development, 54, 156–172.

88.

Samejima

(1969). Estimation of latent ability a response pattern of graded scores (Psychometrika Monograph Supplement, No. 17). https://doi.org/10.1007/BF03372160

89.

Satorra

Bentler

P. M.

(1988). Scaling corrections for statistics in covariance structure analysis (UCLA Statistics Series #2). Los Angeles, CA: University of California.

90.

Satorra

Bentler

P. M.

(1994). Corrections to test statistics and standard errors in covariance structure analysis. In von Eye

Clogg

C. C.

(Eds.), Latent variable analysis: Applications to developmental research (pp. 399–419). SAGE.

91.

Shea

Nguyen

Wong

Y. J.

Gonzalez

(2019). College students’ barriers to seeking mental health counseling: Scale development and psychometric evaluation. Journal of Counseling Psychology, 66, 626–639.

92.

Shin

Welch

Sharma

Vernay

Yee

Smith

(2018). The development and validation of the contemporary consciousness measure II. Journal of Counseling Psychology, 65, 539–555.

93.

Simons

(2018). School counselor sexual minority advocacy competence scale (SCSMACS): Development, validity, and reliability. Professional School Counseling, 21(1), 1–14.

94.

Spector

P. E.

(1980). Ratings of equal ad unequal response choice intervals. The Journal of Social Psychology, 112(1), 115–119. https://doi.org/10.1080/00224545.1980.9924303

95.

Spratto

E. M.

(2018). In search of equality: Developing an equal interval Likert response scale [Doctoral dissertations]. https://commons.lib.jmu.edu/diss201019/172

96.

Stevens

S. S.

(1946). On the theory of scales of measurement. Science, 103, 677–680. https://www.jstor.org/stable/1671815

97.

Sullivan

Artino

A. R.

Jr. (2013). Analyzing and interpreting data from Likert-type scales. Journal of Graduate Medical Education, 5(4), 541–542.

98.

Swank

Limberg

Liu

(2020). Development of the altruism scale for children: An assessment of caring behaviors among children. Measurement and Evaluation in Counseling and Development, 53, 34–43.

99.

Tadlock-Marlo

Hill

(2019). Validation and psychometric properties of one school, many differences. Measurement and Evaluation in Counseling and Development, 52, 156–170.

100.

TaeHyuk-Keum

Brady

Sharma

Kim

Thai

(2018). Gendered racial microaggressions scale for Asian American women: Development and initial validation. Journal of Counseling Psychology, 65, 571–585.

101.

Toland

Kodet

Reese

(2021). Psychometric properties of the outcome rating scale: An item response theory analysis. Measurement and Evaluation in Counseling and Development, 54, 90–105.

102.

Tourangeau

Couper

M. P.

Steiger

D. M.

(2003). Humanizing self-administered surveys: Experiments on social presence in Web and IVR surveys. Computers in Human Behavior, 19, 1–24.

103.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge University Press.

104.

Tourangeau

Smith

T. W.

(1996). Asking sensitive questions: The impact of data collection mode, question format, and question context. Public Opinion Quarterly, 60(2), 275–304.

105.

Trub

Barbot

(2020). Great escape or path to self-expression?: Development and validation of a scale of motivations for text messaging. Measurement and Evaluation in Counseling and Development, 53, 44–61.

106.

Tsaousis

Sideridis

G. D.

Alahmadi

M. T. S.

(2021). Applying a multidimensional item response theory approach in validating the dimensionality of the positive youth development scale. Journal of Psychoeducational Assessment, 397(7), 861–873. https://doi.org/10.1177/07342829211002332

107.

Veronese

Pepe

(2019). Using the posttraumatic growth inventory-short form with Palestinian helpers living in conflict areas. Measurement and Evaluation in Counseling and Development, 52, 207–221.

108.

Vigderhous

(1977). The level of measurement and “permissible” statistical analysis in social research. The Pacific Sociological Review, 20(1), 61–72. https://doi.org/10.2307/1388904

109.

Waldrop

Reschly

Fraysier

Appleton

(2019). Measuring the engagement of college students: Administration format, structure, and validity of the student engagement instrument-college. Measurement and Evaluation in Counseling and Development, 52, 90–107.

110.

Wang

Koay

Wei

Lee

(2019). The development and validation of the emotional cultivation scale: An east Asian cultural perspective. Journal of Counseling Psychology, 66, 409–423.

111.

Watson

Allen

Flores

Serpe

Farrell

(2019). The development and psychometric evaluation of the trans discrimination scale: TDS-21. Journal of Counseling Psychology, 66, 14–29.

112.

Watson

Prosek

Giordano

(2020). Investigating psychometric properties of social media addiction measures among adolescents. Journal of Counseling & Development, 98, 458–466.

113.

Xavier

Cunha

Pinto-Gouveia

(2019). Validation of the risk-taking and self-harm inventory for adolescents in a Portuguese community sample. Measurement and Evaluation in Counseling and Development, 52, 1–14.

114.

Young

Bryan

(2018). The school counselor leadership survey: Confirmatory factor analysis and validation. Measurement and Evaluation in Counseling and Development, 51, 235–249.

115.

Yuan

K.-H.

Bentler

P. M.

(1998). Normal theory based test statistics in structural equation modeling. British Journal of Mathematical and Statistical Psychology, 51, 289–309.

116.

Yuan

K.-H.

Bentler

P. M.

(2011). Ridge structural equation modeling with correlation matrices for ordinal and continuous data. British Journal of Mathematical and Statistical Psychology, 64, 107–133.

117.

Zumbo

B. D.

Zimmerman

D. W.

(1993). Is the selection of statistical methods governed by level of measurement? Canadian Psychology, 34, 390–400.

118.

Zwick

Thayer

D. T.

Lewis

(1999). An empirical Bayes approach to Mantel-Haenszel DIF analysis. Journal of Educational Measurement, 36, 1–28.