How Scoring Approaches Impact Estimates of Growth in the Presence of Survey Item Ceiling Effects

Abstract

Survey scores are often the basis for understanding how individuals grow psychologically and socio-emotionally. A known problem with many surveys is that the items are all “easy”—that is, individuals tend to use only the top one or two response categories on the Likert scale. Such an issue could be especially problematic, and lead to ceiling effects, when the same survey is administered repeatedly over time. In this study, we conduct simulation and empirical studies to (a) quantify the impact of these ceiling effects on growth estimates when using typical scoring approaches like sum scores and unidimensional item response theory (IRT) models and (b) examine whether approaches to survey design and scoring, including employing various longitudinal multidimensional IRT (MIRT) models, can mitigate any bias in growth estimates. We show that bias is substantial when using typical scoring approaches and that, while lengthening the survey helps somewhat, using a longitudinal MIRT model with plausible values scoring all but alleviates the issue. Results have implications for scoring surveys in growth studies going forward, as well as understanding how Likert item ceiling effects may be contributing to replication failures.

Keywords

ceiling effects survey design item response theory multidimensional IRT models growth modeling

Get full access to this article

View all access options for this article.

References

Bauer

D. J.

Curran

P. J.

(2016). The discrepancy between measurement and modeling in longitudinal data analysis. In Harring

J. R.

Stapleton

L. M.

Beretvas

S. N.

(Eds.), Advances in multilevel modeling for educational research (pp. 3–38). Information Age Publishing.

Bock

R. D.

Mislevy

R. J.

(1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431–444. https://doi.org/10.1177/014662168200600405

Bolt

D. M.

Newton

J. R.

(2011). Multiscale measurement of extreme response style. Educational and Psychological Measurement, 71(5), 814–833. https://doi.org/10.1177/0013164410388411

Cai

(2017). FlexMIRT R version 3.51: Flexible multilevel multidimensional item analysis and test scoring [Computer software]. Vector Psychometric Group.

Caprara

G. V.

Fida

Vecchione

Del Bove

Vecchio

G. M.

Barbaranelli

Bandura

(2008). Longitudinal analysis of the role of perceived self-efficacy for self-regulated learning in academic continuance and achievement. Journal of Educational Psychology, 100(3), 525–534. https://doi.org/10.1037/0022-0663.100.3.525

Chassin

Sher

K. J.

Hussong

Curran

(2013). The developmental psychopathology of alcohol use and alcohol disorders: Research achievements and future directions. Development and Psychopathology, 25(4 Pt 2), 1567–1584. https://doi.org/10.1017/S0954579413000771

DeWalt

D. A.

Thissen

Stucky

B. D.

Langer

M. M.

Morgan DeWitt

Irwin

D. E.

Lai

J. S.

Yeatts

K. B.

Gross

H. E.

Taylor

Varni

J. W.

Varni

J. W.

(2013). PROMIS pediatric peer relationships scale: Development of a peer relationships item bank as part of social health measurement. Health Psychology: Official Journal of the Division of Health Psychology, American Psychological Association, 32(10), 1093–1103. https://doi.org/10.1037/a0032670

D’Urso

E. D.

De Roover

Vermunt

J. K.

Tijmstra

(2022). Scale length does matter: Recommendations for measurement invariance testing with categorical factor analysis and item response theory approaches. Behavior Research Methods, 54(5), 2114–2145. https://doi.org/10.3758/s13428-021-01690-7

Feng

Hancock

G. R.

Harring

J. R.

(2019). Latent growth models with floors, ceilings, and random knots. Multivariate Behavioral Research, 54(5), 751–770. https://doi.org/10.1080/00273171.2019.1580556

10.

Flake

J. K.

Pek

Hehman

(2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370–378. https://doi.org/10.1177/1948550617693063

11.

Fricke

Loeb

Meyer

R. H.

Rice

A. B.

Pier

Hough

(2021). Stability of school contributions to student social-emotional learning gains. American Journal of Education, 128(1), 95–145. https://doi.org/10.1086/716550

12.

Gehlbach

Hough

H. J.

(2018). Measuring social emotional learning through student surveys in the CORE districts: A pragmatic approach to validity and reliability: Policy Analysis for California Education.

13.

Gorter

Fox

J. P.

Riet

G. T.

Heymans

M. W.

Twisk

J. W. R.

(2020). Latent growth modeling of IRT versus CTT measured longitudinal latent variables. Statistical Methods in Medical Research, 29(4), 962–986. https://doi.org/10.1177/0962280219856375

14.

Houts

C. R.

Cai

(2020). FlexMIRT user’s manual version 3.6: Flexible multilevel multidimensional item analysis and test scoring. Vector Psychometric Group.

15.

Jackson

D. L.

Gillaspy

J. A.

Jr. Purc-Stephenson

(2009). Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods, 14(1), 6–23. https://doi.org/10.1037/a0014694

16.

Kuhfeld

Soland

(2022). Avoiding bias from sum scores in growth estimates: An examination of IRT-based approaches to scoring longitudinal survey responses. Psychological Methods, 27(2), 234–260. https://doi.org/10.1037/met0000367

17.

Masino

Lam

T. C.

(2014). Choice of rating scale labels: Implication for minimizing patient satisfaction response ceiling effect in telemedicine surveys. Telemedicine Journal and e-Health: The Official Journal of the American Telemedicine Association, 20(12), 1150–1155. https://doi.org/10.1089/tmj.2013.0350

18.

McArdle

J. J.

Epstein

(1987). Latent growth curves within developmental structural equation models. Child Development, 58(1), 110–133. https://doi.org/10.2307/1130295

19.

McNeish

(2022). Limitations of the sum-and-alpha approach to measurement in behavioral research. Policy Insights from the Behavioral and Brain Sciences, 9(2), 196–203. https://doi.org/10.1177/23727322221117144

20.

McNeish

Wolf

M. G.

(2020). Thinking twice about sum scores. Behavior Research Methods, 52(6), 2287–2305. https://doi.org/10.3758/s13428-020-01398-0

21.

Mislevy

R. J.

(1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56(2), 177–196. https://doi.org/10.1007/bf02294457

22.

Mislevy

R. J.

Sheehan

K. M.

(1987). Marginal estimation procedures. In Beaton

A. E.

(Ed.), The NAEP 1983/84 technical report (NAEP Report 15-TR-20 (pp. 293–360): Educational Testing Service.

23.

Sahin

Anil

(2017). The effects of test length and sample size on item parameters in item response theory. Educational Sciences: Theory and Practice, 17(1), 321–335.

24.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(S1), 100–114. https://doi.org/10.1007/bf03372160

25.

Soland

Kuhfeld

(2021). Do response styles affect estimates of growth on social-emotional constructs? Evidence from four years of longitudinal survey scores. Multivariate Behavioral Research, 56(6), 853–873. https://doi.org/10.1080/00273171.2020.1778440

26.

Soland

Kuhfeld

Edwards

(2022). How survey scoring decisions can influence your study’s results: A trip through the IRT looking glass. In Psychological methods. Online first publication. https://doi.org/10.1037/met0000506

27.

Thissen

Orlando

(2001). Item response theory for items scored in two categories. In Thissen

Wainer

(Eds.), Test scoring (pp. 85–152). Routledge.

28.

Thomas

(2002). The role of secondary covariates when estimating latent trait population distributions. Psychometrika, 67(1), 33–48. https://doi.org/10.1007/bf02294708

29.

Von Davier

Gonzalez

Mislevy

(2009). What are plausible values and why are they useful? In IERI monograph series. Issues and methodologies in large scale assessments Vol. 2.

30.

Von Davier

Sinharay

Oranje

Beaton

(2006). Statistical procedures used in the national assessment of educational progress (NAEP): Recent developments and future directions. In Rao

C. R.

Sinharay

(Eds.), Handbook of statistics psychometrics (Vol. 26, pp. 1039–1056). Elsevier.

31.

Wang

Zhang

McArdle

J. J.

Salthouse

T. A.

(2008). Investigating ceiling effects in longitudinal data analysis. Multivariate Behavioral Research, 43(3), 476–496. https://doi.org/10.1080/00273170802285941

32.

West

M. R.

Pier

Fricke

Hough

Loeb

Meyer

R. H.

Rice

A. B.

(2020). Trends in student social-emotional learning: Evidence from the first large-scale panel student survey. Educational Evaluation and Policy Analysis, 42(2), 279–303. https://doi.org/10.3102/0162373720912236

33.

Yeager

D. S.

Walton

G. M.

(2011). Social-psychological interventions in education: They’re not magic. Review of Educational Research, 81(2), 267–301. https://doi.org/10.3102/0034654311405999

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.53 MB