Sage Journals: Discover world-class research

Abstract

Test speededness refers to a situation in which examinee performance is inadvertently affected by the time limit of the test. Because speededness has the potential to severely bias both person and item parameter estimates, it is crucial that speeded examinees are detected. In this article, we develop a change-point analysis (CPA) procedure for detecting test speededness. Our procedure distinguishes itself from existing CPA procedures by using information from both item scores and distractors. Using detailed simulations, we show that under most conditions, the new CPA procedure improves the detection of speeded examinees and produces more accurate change-point estimates. It therefore seems there is a considerable amount of information to be gained from the item distractors, which, quite notably are available in all multiple-choice data. A real data example is also provided.

Keywords

test speededness change-point analysis distractor analysis nested logit model

Get full access to this article

View all access options for this article.

References

Andrews

D. W. K.

(1993). Tests for parameter instability and structural change with unknown change point. Econometrica, 61(4), 821–856. https://doi.org/10.2307/2951764

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29–51. https://doi.org/10.1007/BF02291411

Bolt

D. M.

Cohen

A. S.

Wollack

J. A.

(2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39(4), 331–348. https://doi.org/10.1111/j.1745-3984.2002.tb01146.x

Bolt

D. M.

Wollack

J. A.

Suh

(2012). Application of a multidimensional nested logit model to multiple-choice test items. Psychometrika, 77(2), 339–357. https://doi.org/10.1007/s11336-012-9257-5

Chalmers

R. P.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://www.doi.org/10.18637/jss.v048.i06

Cheng

Shao

(2022). Application of change point analysis of response time data to detect test speededness. Educational and Psychological Measurement, 82(5), 1031–1062. https://doi.org/10.1177/00131644211046392

Cintron

D. W.

(2021). Methods for measuring speededness: Chronology, classification, and ensuing research and development (Research Report No. 21–22). Educational Testing Service. https://doi.org/10.1002/ets2.12337

Estrella

(2003). Critical values and p values of Bessel process distributions: Computation and application to structural break tests. Econometric Theory, 19(6), 1128–1143. https://doi.org/10.1017/S0266466603196107

Goegebeur

De Boeck

Wollack

J. A.

Cohen

A. S.

(2008). A speeded item response model with gradual process change. Psychometrika, 73(1), 65–87. https://doi.org/10.1007/s11336-007-9031-2

10.

Gorney

Wollack

J. A.

(2023). Using item scores and distractors in person-fit assessment. Journal of Educational Measurement, 60(1), 3–27. https://doi.org/10.1111/jedm.12345

11.

Gorney

Wollack

J. A.

Sinharay

Eckerly

(2023). Using item scores and distractors to detect item compromise and preknowledge. Journal of Educational and Behavioral Statistics. Advance online publication. https://doi.org/10.3102/10769986231159923

12.

Jin

K.-Y.

Wang

W.-C.

(2014). Item response theory models for performance decline during testing. Journal of Educational Measurement, 51(2), 178–200. https://doi.org/10.1111/jedm.12041

13.

Oshima

T. C.

(1994). The effect of speededness on parameter estimation in item response theory. Journal of Educational Measurement, 31(3), 200–219. https://doi.org/10.1111/j.1745-3984.1994.tb00443.x

14.

R Core Team . (2022). R: A language and environment for statistical computing (Version 4.2.0). [Computer software]. https://www.R-project.org/

15.

Rebouças

Cheng

(2019, July). Bivariate change-point analysis with response time and item responses [Paper presentation]. International Meeting of the Psychometric Society, Santiago, Chile.

16.

Shao

(2016). Aberrant response detection using change-point analysis (Publication No. 10308190) [Doctoral dissertation, University of Notre Dame]. ProQuest Dissertations & Theses Global.

17.

Shao

Cheng

(2016). Detection of test speededness using change-point analysis. Psychometrika, 81(4), 1118–1141. https://doi.org/10.1007/s11336-015-9476-7

18.

Sinharay

(2016). Person fit analysis in computerized adaptive testing using tests for a change point. Journal of Educational and Behavioral Statistics, 41(5), 521–549. https://doi.org/10.3102/1076998616658331

19.

Cai

(2023). A new perspective on detecting performance decline: A change-point analysis based on Jensen-Shannon divergence. Behavior Research Methods, 55(3), 963–980. https://doi.org/10.3758/s13428-021-01779-z

20.

Yamamoto

Everson

(1997). Modeling the effects of test length and test time on parameter estimation using the HYBRID model. In Rost

Langeheine

(Eds.), Applications of latent trait and latent class models in the social sciences (pp. 89–98). Waxmann.

21.

Cheng

(2022). A comprehensive review and comparison of CUSUM and change-point-analysis methods to detect test speededness. Multivariate Behavioral Research, 57(1), 112–133. https://doi.org/10.1080/00273171.2020.1809981

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.73 MB

Using Item Scores and Distractors to Detect Test Speededness

Abstract

Keywords

Get full access to this article

References

Supplementary Material