Revisiting the Effect of Varying the Number of Response Alternatives in Clinical Assessment: Evidence From Measuring ADHD Symptoms

Abstract

This study illustrated the effect of varying the number of response alternatives in clinical assessment using a within-participant, repeated-measures approach. Participants reported the presence of current attention-deficit/hyperactivity disorder symptoms using both a binary and a polytomous (4-point) rating scale across two counterbalanced administrations of the Current Symptoms Scale (CSS). Psychometric properties of the CSS were examined using (a) self-reported binary, (b) self-reported 4-point ratings obtained from each administration of the CSS, and (c) artificially dichotomized responses derived from observed 4-point ratings. Under the same ordinal factor analysis model, results indicated that the number of response alternatives affected item parameter estimates, standard errors, goodness of fit indices, individuals’ test scores, and reliability of the test scores. With fewer response alternatives, the precision of the measurement decreased, and the power of using the goodness-of-fit indices to detect model misfit decreased. These findings add to recent research advocating for the inclusion of a large number of response alternatives in the development of clinical assessments and further suggest that researchers should be cautious about reducing the number of response categories in data analysis.

Keywords

Likert-type scale response alternatives ordinal factor analysis clinical assessment

Get full access to this article

View all access options for this article.

References

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., Rev. ed.). Author.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Author. https://doi.org/10.1176/appi.books.9780890425596

Asparouhov

Muthén

(2006). Robust chi square difference testing with mean and variance adjusted test statistics (Mplus Web Notes No. 10). Muthén & Muthén. http://www.statmodel.com/download/webnotes/webnote10.pdf

Asparouhov

Muthén

B. O.

(2010). Simple second order chi-square correction scaled chi-square statistics (Technical appendix). Muthén & Muthén.

Asparouhov

Muthén

(2018). SRMR in Mplus. http://www.statmodel.com/download/SRMR2.pdf

Barkley

R. A.

Murphy

K. R.

(1998). A clinical workbook: Attention-deficit hyperactivity disorder. Guilford Press. https://doi.org/10.1038/scientificamerican0998-66

Barkley

R. A.

Murphy

K. R.

(2006). Attention-deficit hyperactivity disorder: A clinical workbook. Guilford Press.

Bech

Rasmussen

N. A.

Olsen

L. R.

Noerholm

Abildgaard

(2001). The sensitivity and specificity of the Major Depression Inventory, using the Present State Examination as the index of diagnostic validity. Journal of Affective Disorders, 66(2-3), 159-164. https://doi.org/10.1016/S0165-0327(00)00309-8

Beck

A. T.

Ward

C. H.

Mendelson

Mock

Erbaugh

(1961). An inventory for measuring depression. Archives of General Psychiatry, 4(6), 561-571. https://doi.org/10.1001/archpsyc.1961.01710120031004

10.

Bernstein

I. H.

Teng

(1989). Factoring items and factoring scales are different: Spurious evidence for multidimensionality due to item categorization. Psychological Bulletin, 105(3), 467-477. https://doi.org/10.1037/0033-2909.105.3.467

11.

Broman-Fulks

J. J.

Deacon

B. J.

Olatunji

B. O.

Bondy

C. L.

Abramowitz

J. S.

Tolin

D. F.

(2010). Categorical or dimensional: A reanalysis of the anxiety sensitivity construct. Behavior Therapy, 41(2), 154-171. https://doi.org/10.1016/j.beth.2009.02.005

12.

Brown

Croudace

T. J.

(2015). Scoring and estimating score precision using multidimensional IRT models. In Reise

S. P.

Revicki

D. A.

(Eds.), Multivariate applications series. Handbook of item response theory modeling: Applications to typical performance assessment (pp. 307-333). Routledge/Taylor & Francis.

13.

Butcher

Dahlstrom

Graham

Tellegen

Kaemmer

(1989). Manual for administering and scoring the MMPI-2. University of Minnesota Press.

14.

Chang

(1994). A psychometric evaluation of 4-point and 6-point Likert-type scales in relation to reliability and validity. Applied Psychological Measurement, 18(3), 205-215. https://doi.org/10.1177/014662169401800302

15.

Chung

Jeon

M. J.

Park

Lee

Kim

C. O.

Park

J. Y.

(2018). Development and evaluation of a mobile-optimized daily self-rating depression screening app: A preliminary study. PLOS ONE, 13(6), Article e0199118. https://doi.org/10.1371/journal.pone.0199118

16.

Cicchetti

D. V.

Showalter

Tyrer

P. J.

(1985). The effect of number of rating scale categories on levels of interrater reliability: A Monte Carlo investigation. Applied Psychological Measurement, 9(1), 31-36. https://doi.org/10.1177/014662168500900103

17.

Cohen

(1983). The cost of dichotomization. Applied Psychological Measurement, 7(3), 249-253. https://doi.org/10.1177/014662168300700301

18.

Comrey

A. L.

Montag

(1982). Comparison of factor analytic results with two-choice and seven-choice personality item formats. Applied Psychological Measurement, 6(3), 285-289. https://doi.org/10.1177/014662168200600304

19.

Costa

P. T.

McCrae

R. R.

(1992). Normal personality assessment in clinical practice: The NEO Personality Inventory. Psychological Assessment, 4(1), 5-13. https://doi.org/10.1037/1040-3590.4.1.5

20.

Courrégé

S. C.

Weed

N. C.

(2019). The role of common method variance in MMPI-2-RF response option augmentation. Psychological Assessment, 31(1), 126-131. https://doi.org/10.1037/pas0000634

21.

Cox

Courrégé

S. C.

Feder

A. H.

Weed

N. C.

(2017). Effects of augmenting response options of the MMPI-2-RF: An extension of previous findings. Cogent Psychology, 4(1), Article 1323988. https://doi.org/10.1080/23311908.2017.1323988

22.

Cox

Pant

Gilson

A. N.

Rodriguez

J. L.

Young

K. R.

Kwon

Weed

N. C.

(2012). Effects of augmenting response options on MMPI–2 RC scale psychometrics. Journal of Personality Assessment, 94(6), 613-619. https://doi.org/10.1080/00223891.2012.700464

23.

Cox

E. P.

III . (1980). The optimal number of response alternatives for a scale: A review. Journal of marketing research, 17(4), 407-422. https://doi.org/10.1177/002224378001700401

24.

DeCoster

Iselin

A. M. R.

Gallucci

(2009). A conceptual and empirical examination of justifications for dichotomization. Psychological Methods, 14(4), 349-366. https://doi.org/10.1037/a0016956

25.

DiStefano

(2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9(3), 327-346. https://doi.org/10.1207/S15328007SEM0903_2

26.

DiStefano

Zhu

Mindrila

(2009). Understanding and using factor scores: Considerations for the applied researcher. Practical Assessment, Research & Evaluation, 14(20), 1-11.

27.

Dolan

C. V.

(1994). Factor analysis of variables with 2, 3, 5 and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47(2), 309-326. https://doi.org/10.1111/j.2044-8317.1994.tb01039.x

28.

Dolnicar

Grün

(2007). How constrained a response: A comparison of binary, ordinal and metric answer formats. Journal of Retailing and Consumer Services, 14(2), 108-122. https://doi.org/10.1016/j.jretconser.2006.09.006

29.

Dolnicar

Grün

(2009). Response style contamination of student evaluation data. Journal of Marketing Education, 31(2), 160-172. https://doi.org/10.1177/0273475309335267

30.

DuPaul

G. J.

Weyandt

L. L.

O’Dell

S. M.

Varejao

(2009). College students with ADHD: Current status and future directions. Journal of Attention Disorders, 13(3), 234-250. https://doi.org/10.1177/1087054709340650

31.

Finn

J. A.

Ben-Porath

Y. S.

Tellegen

(2015). Dichotomous versus polytomous response options in psychopathology assessment: Method or meaningful variance? Psychological Assessment, 27(1), 184-193. https://doi.org/10.1037/pas0000044

32.

Finn

R. H.

(1972). Effects of some variations in rating scale characteristics on the means and reliabilities of ratings. Educational and Psychological Measurement, 32(2), 255-265. https://doi.org/10.1177/001316447203200203

33.

Flory

Shi

Siceloff

E. R.

Roberts

Castellanos

Neger

E. N.

Taylor

Benson

(2020). The factor structure and gender invariance of ADHD in college students. Assessment. Advance online publication. https://doi.org/10.1177/1073191120918934

34.

Forero

C. G.

Maydeu-Olivares

Gallardo-Pujol

(2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling, 16(4), 625-641. https://doi.org/10.1080/10705510903203573

35.

Garner

D. M.

Olmsted

M. P.

Bohr

Garfinkel

P. E.

(1982). The eating attitudes test: Psychometric features and clinical correlates. Psychological Medicine, 12(4), 871-878. https://doi.org/10.1017/S0033291700049163

36.

Gray

S. A.

Fettes

Woltering

Mawjee

Tannock

(2016). Symptom manifestation and impairments in college students with ADHD. Journal of Learning Disabilities, 49(6), 616-630. https://doi.org/10.1177/0022219415576523

37.

Green

S. B.

Akey

T. M.

Fleming

K. K.

Hershberger

S. L.

Marquis

J. G.

(1997). Effect of the number of scale points on chi-square fit indices in confirmatory factor analysis. Structural Equation Modeling, 4(2), 108-120. https://doi.org/10.1080/10705519709540064

38.

Green

B. F.

Bock

R. D.

Humphreys

L. G.

Linn

R. L.

Reckase

M. D.

(1984). Technical guidelines for assessing computerized adaptive tests. Journal of Educational Measurement, 21(4), 347-360. https://doi.org/10.1111/j.1745-3984.1984.tb01039.x

39.

Grice

J. W.

(2001). Computing and evaluating factor scores. Psychological Methods, 6(4), 430-450. https://doi.org/10.1037/1082-989X.6.4.430

40.

Gudjonsson

G. H.

Sigurdsson

J. F.

Sigfusdottir

I. D.

Young

(2014). A national epidemiological study of offending and its relationship with ADHD symptoms and associated risk factors. Journal of Attention Disorders, 18(1), 3-13. https://doi.org/10.1177/1087054712437584

41.

Guldberg-Kjär

Johansson

(2015). ADHD symptoms across the lifespan: A comparison of symptoms captured by the Wender and Barkley Scales and DSM-IV criteria in a population-based Swedish sample aged 65 to 80. Journal of Attention Disorders, 19(5), 390-404. https://doi.org/10.1177/1087054713514853

42.

Hamilton

(1960). A rating scale for depression. Journal of Neurology, Neurosurgery and Psychiatry, 23, 56-62. https://doi.org/10.1136/jnnp.23.1.56

43.

Hamilton

(1967). Development of a rating scale for primary depressive illness. British Journal of Social and Clinical Psychology, 6, 278-296. https://doi.org/10.1111/j.2044-8260.1967.tb00530.x

44.

Harris

C. W.

(1967). On factors and factor scores. Psychometrika, 32(4), 363-379. https://doi.org/10.1007/BF02289652

45.

Jenkins

G. D.

Taber

T. D.

(1977). A Monte Carlo study of factors affecting three indices of composite scale reliability. Journal of Applied Psychology, 62(4), 392-398. https://doi.org/10.1037/0021-9010.62.4.392

46.

Lenhard

(2014). Hypothesis tests for comparing correlations. https://www.psychometrica.de/correlation.html

47.

Likert

(1932). A technique for the measurement of attitudes (Archives of Psychology). Columbia University Press.

48.

Lissitz

R. W.

Green

S. B.

(1975). Effect of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60(1), 10-13. https://doi.org/10.1037/h0076268

49.

Lozano

L. M.

García-Cueto

Muñiz

(2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology, 4(2), 73-79. https://doi.org/10.1027/1614-2241.4.2.73

50.

Kaisari

Dourish

C. Y.

Higgs

(2017). Attention Deficit Hyperactivity Disorder (ADHD) and disordered eating behaviour: A systematic review and a framework for future research. Clinical Psychology Review, 53(April), 109-121. https://doi.org/10.1016/j.cpr.2017.03.002

51.

Mackinnon

McCallum

Andrews

Anderson

(1998). The center for epidemiological studies depression scale in older community samples in Indonesia, North Korea, Myanmar, Sri Lanka, and Thailand. Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 53(6), 343-352. https://doi.org/10.1093/geronb/53B.6.P343

52.

Mannuzza

Fyer

A. J.

Klein

D. F.

Endicott

(1986). Schedule for Affective Disorders and Schizophrenia: Lifetime Version modified for the study of anxiety disorders (SADS-LA): rationale and conceptual development. Journal of Psychiatric Research, 20(4), Article 317N325. https://doi.org/10.1016/0022-3956(86)90034-8

53.

Matell

M. S.

Jacoby

(1971). Is there an optimal number of alternatives for Likert scale items? Study I: Reliability and validity. Educational and Psychological Measurement, 31(3), 657-674. https://doi.org/10.1177/001316447103100307

54.

Maydeu-Olivares

(2017). Assessing the size of model misfit in structural equation models. Psychometrika, 82(3), 533-558. https://doi.org/10.1007/s11336-016-9552-7

55.

Maydeu-Olivares

Fairchild

A. J.

Hall

A. G.

(2017). Goodness of fit in item factor analysis: Effect of the number of response alternatives. Structural Equation Modeling, 24(4), 495-505. https://doi.org/10.1080/10705511.2017.1289816

56.

Maydeu-Olivares

Kramp

García-Forero

Gallardo-Pujol

Coffman

(2009). The effect of varying the number of response alternatives in rating scales: Experimental evidence from intra-individual effects. Behavior Research Methods, 41(2), 295-308. https://doi.org/10.3758/BRM.41.2.295

57.

McCoach

D. B.

(2002). A validation study of the School Attitude Assessment Survey. Measurement and Evaluation in Counseling and Development, 35(2), 66-78. https://doi.org/10.1080/07481756.2002.12069050

58.

McCoach

D. B.

Siegle

(2003). The school attitude assessment survey-revised: A new instrument to identify academically able students who underachieve. Educational and Psychological Measurement, 63(3), 414-429. https://doi.org/10.1177/0013164403063003005

59.

McKelvie

S. J.

(1978). Graphic rating scales: How many categories? British Journal of Psychology, 69(2), 185-202. https://doi.org/10.1111/j.2044-8295.1978.tb01647.x

60.

Miller

K. A.

Piombo

S. E.

Cho

Higgins

Wysong

Sussman

Cockburn

M. G.

Leventhal

A. M.

(2018). Prevalence of tanning addiction and behavioral health conditions among ethnically and racially diverse adolescents. Journal of Investigative Dermatology, 138(7), 1511-1517. https://doi.org/10.1016/j.jid.2018.02.018

61.

Morey

L. C.

(1991). Personality Assessment Inventory professional manual. Psychological Assessment Resources.

62.

Morey

L. C.

(2007). Personality Assessment Inventory professional manual (2nd ed.). Psychological Assessment Resources.

63.

Muthén

B. O.

(1998-2004). Mplus technical appendices. Muthén & Muthén.

64.

Muthén

L. K.

Muthén

B. O.

(1998-2012). Mplus user’s guide (7th ed.). Muthén & Muthén.

65.

Oliver

M. L.

Nigg

J. T.

Cassavaugh

N. D.

Backs

R. W.

(2012). Behavioral and cardiovascular responses to frustration during simulated driving tasks in young adults with and without attention disorder symptoms. Journal of Attention Disorders, 16(6), 478-490. https://doi.org/10.1177/1087054710397132

66.

Oswald

W. T.

Velicer

W. F.

(1980). Item format and the structure of the Eysenck Personality Inventory: A replication. Journal of Personality Assessment, 44(3), 283-288. https://doi.org/10.1207/s15327752jpa4403_11

67.

Preston

C. C.

Colman

A. M.

(2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104(1), 1-15. https://doi.org/10.1016/S0001-6918(99)00050-5

68.

Purpura

D. J.

Wilson

S. B.

Lonigan

C. J.

(2010). Attention-deficit/hyperactivity disorder symptoms in preschool children: Examining psychometric properties using item response theory. Psychological Assessment, 22(3), 546-558. https://doi.org/10.1037/a0019581

69.

Radloff

L. S.

(1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1(3), 385-401. https://doi.org/10.1177/014662167700100306

70.

Ramsay

J. O.

(1973). The effect of number of categories in rating scales on precision of estimation of scale values. Psychometrika, 38(4), 513-532. https://doi.org/10.1007/BF02291492

71.

Richter

Strauss

Braehler

Altmann

Berger

(2016). Psychometric properties of a short version of the Eating Attitudes Test (EAT-8) in a German representative sample. Eating Behaviors, 21(April), 198-204. https://doi.org/10.1016/j.eatbeh.2016.03.006

72.

Savalei

Rhemtulla

(2013). The performance of robust test statistics with categorical data. British Journal of Mathematical and Statistical Psychology, 66(2), 201-223. https://doi.org/10.1111/j.2044-8317.2012.02049.x

73.

Simms

L. J.

Zelazny

Williams

T. F.

Bernstein

(2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 557-566. https://doi.org/10.1037/pas0000648

74.

Shi

DiStefano

McDaniel

H. L.

Jiang

(2018). Examining chi-square test statistics under conditions of large model size and ordinal data. Structural Equation Modeling, 25(6). 924-945. https://doi.org/10.1080/10705511.2018.1449653

75.

Shi

Maydeu-Olivares

DiStefano

(2018). The relationship between the standardized root mean square residual and model misspecification in factor analysis models. Multivariate Behavioral Research, 53(5), 676-694. https://doi.org/10.1080/00273171.2018.1476221

76.

Van Dam

N. T.

Earleywine

. (2011). Validation of the Center for Epidemiologic Studies Depression Scale—Revised (CESD-R): Pragmatic depression assessment in the general population. Psychiatry Research, 186(1), 128-132. https://doi.org/10.1016/j.psychres.2010.08.018

77.

Velicer

W. F.

DiClemente

C. C.

Corriveau

D. P.

(1984). Item format and the structure of the Personal Orientation Inventory. Applied Psychological Measurement, 8(4), 409-419. https://doi.org/10.1177/014662168400800405

78.

Watson

(2003). Investigating the construct validity of the dissociative taxon: Stability analyses of normal and pathological dissociation. Journal of Abnormal Psychology, 112(2), 298-305. https://doi.org/10.1037/0021-843X.112.2.298

79.

Weyandt

L. L.

DuPaul

(2006). ADHD in college students. Journal of Attention Disorders, 10(1), 9-19. https://doi.org/10.1177/1087054705286061

80.

Weng

L. J.

(2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64(6), 956-972. https://doi.org/10.1177/0013164404268674

81.

Wilens

T. E.

Faraone

S. V.

Biederman

(2004). Attention-deficit/hyperactivity disorder in adults. Journal of the American Medical Association, 292(5), 619-623. https://doi.org/10.1001/jama.292.5.619

82.

Yesavage

J. A.

Brink

T. L.

Rose

T. L.

Lum

Huang

Adey

Leirer

V. O.

(1982). Development and validation of a geriatric depression screening scale: A preliminary report. Journal of Psychiatric Research, 17(1), 37-49. https://doi.org/10.1016/0022-3956(82)90033-4

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB