Administrative problems such as computer malfunction and power outage occasionally lead to missing item scores and hence to incomplete data on mastery tests such as the AP and U.S. Medical Licensing examinations. Investigators are often interested in estimating the probabilities of passing of the examinees with incomplete data on mastery tests. However, there is a lack of research on this estimation problem. The goal of this article is to suggest two new approaches—one each based on classical test theory and item response theory—for estimating the probabilities of passing of the examinees with incomplete data on mastery tests. The two approaches are demonstrated to have high accuracy and negligible misclassification rates.
American Educational Research Association, American Psychological Association, & National Council for Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
2.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397-479). Addison-Wesley.
3.
ByrneM. R. (2017, October23). Decisions, concerns, and questions pertaining to two 2017 statewide assessment events involving Algebra I and English II EOCs of the Missouri assessment program [Report submitted to Governor Eric Grietens]. https://www.moagainstcommoncore.com/2017EOCAssessmentIssues-10-23-17.pdf
4.
Cetin-BerberD. D.SariH. I.Huggins-ManleyA. C. (2019). Imputation methods to deal with missing responses in computerized adaptive multistage testing. Educational and Psychological Measurement, 79(3), 495-511. https://doi.org/10.1177/0013164418805532
5.
ChalmersR. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. https://doi.org/10.18637/jss.v048.i06
6.
ChangH. H.StoutW. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58(1), 37-52. https://doi.org/10.1007/BF02294469
7.
De AyalaR. J. (2009). The theory and practice of item response theory. Guilford Press.
8.
De AyalaR. J.PlakeB. S.ImparaJ. C. (2001). The impact of omitted responses on the accuracy of ability estimation in item response theory. Journal of Educational Measurement, 38(3), 213-234. https://doi.org/10.1111/j.1745-3984.2001.tb01124.x
9.
Educational Testing Service. (2020). The Praxis test information bulletin 2020-21.
10.
EdwardsJ. M.FinchW. H. (2018). Recursive partitioning methods for data imputation in the context of item response theory: A Monte Carlo simulation. Psicológica Journal, 39(1), 88-117. https://doi.org/10.2478/psicolj-2018-0005
11.
EndersC. K. (2010). Applied missing data analysis. Guilford Press.
12.
FeinbergR. (2020). Estimating classification decisions for incomplete tests. Educational Measurement: Issues and Practice. Advance online publication. https://doi.org/10.1111/emip.12412
13.
FergusonT. S. (1967). Mathematical statistics: A decision theoretic approach. Academic Press.
14.
FinchH. (2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225-245. https://doi.org/10.1111/j.1745-3984.2008.00062.x
15.
GinerG.SmythG. K. (2016). statmod: Probability calculations for the inverse Gaussian distribution. R Journal, 8(1), 339-351. https://doi.org/10.32614/RJ-2016-024
16.
GlasC. A. W.PimentelJ. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68(6), 907-922. https://doi.org/10.1177/0013164408315262
HansonB. A. (1994). Extension of Lord-Wingersky algorithm to computing test scores for polytomous items [Unpublished manuscript].
20.
HastieT.TibshiraniR.FriedmanJ. H. (2009). The elements of statistical learning: Data mining, inference, and prediction. Springer.
21.
HolmanR.GlasC. A. W. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58(1), 1-17. https://doi.org/10.1111/j.2044-8317.2005.tb00312.x
22.
HuismanM.MolenaarI. W. (2001). Imputation of missing scale data with item response models. In BoomsmaA.van DuijnM. A. J.SnijdersT. A. B. (Eds.), Essays on item response theory (pp. 221-244). Springer. https://doi.org/10.1007/978-1-4613-0169-1_13
23.
KangT.ChenT. T. (2008). Performance of the generalized S—χ2 item-fit index for polytomous IRT models. Journal of Educational Measurement, 45(4), 391-406. https://doi.org/10.1111/j.1745-3984.2008.00071.x
24.
KangY.LeeW.-C. (2011). Comparison of IRT linking and equating methods with mixed-format tests. In KolenM. J.LeeW.-C. (Eds.), Mixed-format tests: Psychometric properties with a primary focus on equating (pp. 47-76). University of Iowa.
25.
KohlerC.PohlS.CarstensenC. H. (2017). Dealing with item nonresponse in large-scale cognitive assessments: The impact of missing data methods on estimated explanatory relationships. Journal of Educational Measurement, 54(4), 397-419. https://doi.org/10.1111/jedm.12154
26.
KolenM. J.LeeW.-C. (2011). Psychometric properties of scores on mixed-format tests. Educational Measurement: Issues and Practice, 30(2), 15-24. https://doi.org/10.1111/j.1745-3992.2011.00201.x
27.
LeeW.-C. (2007). Multinomial and compound multinomial error model for tests with complex item scoring. Applied Psychological Measurement, 31(4), 255-274. https://doi.org/10.1177/0146621606294206
28.
LeeW.-C.WangT.KimS.BrennanR. L. (2006). A strong true-score model for test scores based on polytomous items (CASMA Research Report No. 16). University of Iowa.
29.
LimH. (2020). irtplay: Unidimensional item response theory modeling. R package Version 1.6.2.
MichelR. S. (2020). Remotely proctored K-12 high stakes standardized testing during COVID-19: Will it last?Educational Measurement: Issues and Practice, 39(3), 28-30. https://doi.org/10.1111/emip.12364
34.
MurakiE. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16(2), 159-176. https://doi.org/10.1177/014662169201600206
35.
NaylorJ. C.SmithA. F. M. (1982). Applications of a method for the efficient computation of posterior distributions. Applied Statistics, 31(3), 214-225. https://doi.org/10.2307/2347995
36.
PattersonB. F.EwingM. (2013). Validating the use of AP exam scores for college course placement [College Board Research Report No. 2013-2]. College Board.
37.
PohlS.GraefeL.RoseN. (2014). Dealing with omitted and not-reached items in competence tests. Educational and Psychological Measurement, 74(3), 423-452. https://doi.org/10.1177/0013164413504926
38.
R Core Team. (2020). R: A language and environment for statistical computing. R Soundation for Statistical Computing.
39.
RaghunathanT.LepkowskiJ. M.van HoewykJ.SolenbergerP. W. (2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27, 85-96.
40.
RobitzschA. (2019). sirt: Supplementary item response theory models. R package version 3.6-21.
41.
RoseN.von DavierM.NagengastB. (2017). Modeling omitted and not-reached items in IRT models. Psychometrika, 82(3), 795-819. https://doi.org/10.1007/s11336-016-9544-7
ShahB. K. (1994). On the distribution of the sum of independent integer valued random variables. American Statistician, 27, 123-124.
45.
ShinS.-H. (2009). How to treat omitted responses in Rasch model-based equating. Practical Assessment, Research, and Evaluation, 14, Article 1.
46.
SijtsmaK.van der ArkL. A. (2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research, 38(4), 505-528. https://doi.org/10.1207/s15327906mbr3804_4
47.
SinharayS. (2016). An NCME instructional module on data mining methods for classification and regression. Educational Measurement: Issues and Practice, 35(3), 38-54. https://doi.org/10.1111/emip.12115
48.
SinharayS. (2021). Score reporting for examinees with incomplete data on large-scale assessments. Educational Measurement: Issues and Practice, 40(1), 79-91. https://doi.org/10.1111/emip.12396
49.
SinharayS.SternH. S.RussellD. (2001). The use of multiple imputation for the analysis of missing data. Psychological Methods, 6(4), 317-329. https://doi.org/10.1037/1082-989X.6.4.317
50.
SinharayS.WanP.ChoiS. W.KimD. (2015). Assessing individual-level impact of interruptions during online testing. Journal of Educational Measurement, 52(1), 80-105. https://doi.org/10.1111/jedm.12064
51.
SinharayS.WanP.WhitakerM.KimD.ZhangL.ChoiS. W. (2014). Determining the overall impact of interruptions during online testing. Journal of Educational Measurement, 51(4), 419-440. https://doi.org/10.1111/jedm.12052
52.
SmitsN.MellenberghG. J.VorstH. C. M. (2002). Alternative missing data techniques to grade point average: Imputing unavailable grades. Journal of Educational Measurement, 39(3), 187-206. https://doi.org/10.1111/j.1745-3984.2002.tb01173.x
53.
StoutW. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52(4), 589-617. https://doi.org/10.1007/BF02294821
54.
SulisI.PorcuM. (2017). Handling missing data in item response theory. Assessing the accuracy of a multiple imputation procedure based on latent class analysis. Journal of Classification, 34(2), 327-359. https://doi.org/10.1007/s00357-017-9220-3
55.
ThissenD.PommerichM.BilleaudK.WilliamsV. (1995). Item response theory for scores on tests including polytomous items with ordered responses. Applied Psychological Measurement, 19(1), 39-49. https://doi.org/10.1177/014662169501900105
VriensM.SinharayS. (2006). Dealing with missing data in surveys and databases. In The handbook of marketing research (pp. 178-191). Sage. https://doi.org/10.4135/9781412973380.n10
58.
XiaoJ.BulutO. (2020). Evaluating the performances of missing data handling methods in ability estimation from sparse data. Educational and Psychological Measurement, 80(5), 932-954. https://doi.org/10.1177/0013164420911136
