Sage Journals: Discover world-class research

Abstract

Takers of educational tests often receive proficiency levels instead of or in addition to scaled scores. For example, proficiency levels are reported for the Advanced Placement (AP^®) and U.S. Medical Licensing examinations. Technical difficulties and other unforeseen events occasionally lead to missing item scores and hence to incomplete data on these tests. The reporting of proficiency levels to the examinees with incomplete data requires estimation of the performance of the examinees on the missing part and essentially involves imputation of missing data. In this article, six approaches from the literature on missing data analysis are brought to bear on the problem of reporting of proficiency levels to the examinees with incomplete data. Data from several large-scale educational tests are used to compare the performances of the six approaches to the approach that is operationally used for reporting proficiency levels for these tests. A multiple imputation approach based on chained equations is shown to lead to the most accurate reporting of proficiency levels for data that were missing at random or completely at random, while the model-based approach of Holman and Glas performed the best for data that are missing not at random. Several recommendations are made on the reporting of proficiency levels to the examinees with incomplete data.

Keywords

IRT models multiple imputation regression imputation

Get full access to this article

View all access options for this article.

References

Agresti

(2013). Categorical data analysis (3rd ed.). John Wiley Publisher.

Allen

N. A.

Donoghue

J. R.

Schoeps

T. L.

(2001). The NAEP 1998 technical report (NCES No. 2001-452) . U.S. Department of Education, Institute of Education Sciences, Office for Educational Research and Improvement.

Baker

F. B.

Kim

H. S.

(2004). Item response theory: Parameter estimation techniques (2nd ed.). Marcel Dekker.

Bernaards

C. A.

Sijtsma

(2000). Influence of imputation and EM methods on factor analysis when item nonresponse in questionnaire data is nonignorable. Multivariate Behavioral Research, 35, 321–364.

Bolsinova

Maris

(2016). Can IRT solve the missing data problem in test equating? Frontiers in Psychology, 6. https://doi.org/doi=10.3389/fpsyg.2015.01956

Box

G. E. P.

Draper

N. R.

(1987). Empirical model-building and response surfaces. Wiley.

Braun

H. I.

Holland

P. W.

(1982). Observed score test equating: A mathematical analysis of some ETS equating procedures. In Holland

P. W.

Rubin

D. B.

(Eds.), Test equating (pp. 9–49). Academic Press.

Byrne

M. R.

(2017). Decisions, concerns, and questions pertaining to two 2017 statewide assessment events involving Algebra I and English II EOCs of the Missouri assessment program. Report submitted to Governor Eric Grietens. https://www.moagainstcommoncore.com/2017EOCAssessmentIssues-10-23-17.pdf

Cetin-Berber

D. D.

Sari

H. I.

Huggins-Manley

A. C.

(2019). Imputation methods to deal with missing responses in computerized adaptive multistage testing. Educational and Psychological Measurement, 79, 495–511.

10.

Chalmers

R. P.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29.

11.

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.

12.

College Board. (n.d.). AP student datafile for schools and districts 2020 layout format. Retrieved September 20, 2021, from https://apcentral.collegeboard.org/pdf/ap-datafile-layout-2020.pdf

13.

De Ayala

R. J.

Plake

B. S.

Impara

J. C.

(2001). The impact of omitted responses on the accuracy of ability estimation in item response theory. Journal of Educational Measurement, 38, 213–234.

14.

Educational Testing Service. (2020). The Praxis test information bulletin 2020-21. ETS.

15.

Educational Testing Service. (n.d.). The Praxis study companion for Spanish: World language. Retrieved September 20, 2021, from https://www.ets.org/s/praxis/pdf/5195.pdf

16.

Edwards

J. M.

Finch

W. H.

(2018). Recursive partitioning methods for data imputation in the context of item response theory: A Monte Carlo simulation. Psicológica Journal, 39, 88–117.

17.

Enders

C. K.

(2011). Missing not at random models for latent growth curve analyses. Psychological Methods, 16, 1–16.

18.

Enders

C. K.

Keller

B. T.

(2020). A model-based imputation procedure for multilevel regression models with random coeffcients, interaction effects, and non-linear terms. Psychological Methods, 25, 88–112.

19.

Enders

C. K.

Keller

B. T.

Levy

(2018). A fully conditional specification approach to multilevel imputation of categorical and continuous variables. Psychological Methods, 23, 298–317.

20.

Fayers

P. M.

Hays

R. D.

(2014). Should linking replace regression when mapping from profile-based measures to preference-based measures? Value in Health, 17, 261–265.

21.

Feinberg

(2021). Estimating classification decisions for incomplete tests. Educational Measurement: Issues and Practice, 40(2), 96–105.

22.

Finch

(2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45, 225–245.

23.

Gayle

(2017). Dozens of high school students’ AP exams missing multiple choice sections, parents angry. Retrieved May 25, 2021, from https://wjla.com/news/local/dozens-of-high-school-students-ap-exams-missing-multiple-choice-sections-parents-angry

24.

Glas

C. A. W.

Pimentel

J. L.

(2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68, 907–922.

25.

Graham

J. W.

(2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576.

26.

Graham

J. W.

(2012). Missing data. Springer.

27.

Grund

Lüdtke

Robitzsch

(2020). On the treatment of missing data in background questionnaires in educational large-scale assessments: An evaluation of different procedures. Journal of Educational and Behavioral Statistics. Advance online publication. https://doi.org/10.3102/1076998620959058

28.

Hambleton

R. K.

Han

(2005). Assessing the fit of IRT models to educational and psychological test data: A five step plan and several graphical displays. In Lenderking

W. R.

Revicki

(Eds.), Advances in health outcomes research methods, measurement, statistical analysis, and clinical applications (pp. 57–78). Degnon Associates.

29.

Hastie

Tibshirani

Friedman

J. H.

(2009). The elements of statistical learning: Data mining, inference, and prediction. Springer.

30.

Holman

Glas

C. A. W.

(2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58, 1–17.

31.

Horton

N. J.

Lipsitz

S. R.

(2001). Multiple imputation in practice. The American Statistician, 55, 244–254.

32.

Huisman

(1999). Item nonresponse: Occurrence, causes, and imputation to missing answers to test items. DSWO Press.

33.

Huisman

Molenaar

I. W.

(2001). Imputation of missing scale data with item response models. In Boomsma

van Duijn

M. A. J.

Snijders

T. A. B.

(Eds.), Essays on item response theory (pp. 221–244). Springer.

34.

Jodoin

M. G.

Rubright

J. D.

(2020). When examinees cannot test: The pandemic’s assault on certification and licensure. Educational Measurement: Issues and Practice, 39(3), 31–33.

35.

Kadengye

D. T.

Cools

Ceulemans

Van den Noortgate

(2012). Simple imputation methods versus direct likelihood analysis for missing item scores in multilevel educational data. Behavior Research Methods, 44, 516–531.

36.

Kang

Chen

T. T.

(2008). Performance of the generalized

S - χ^{2}

item-fit index for polytomous IRT models. Journal of Educational Measurement, 45, 391–406.

37.

Köhler

Pohl

Carstensen

C. H.

(2017). Dealing with item nonresponse in large-scale cognitive assessments: The impact of missing data methods on estimated explanatory relationships. Journal of Educational Measurement, 54, 397–419.

38.

Kolen

M. J.

Brennan

R. L.

(2014). Test equating, scaling, and linking (3rd ed.). Springer.

39.

Korobko

O. B.

Glas

C. A. W.

Bosker

R. J.

Luyten

J. W.

(2008). Comparing the difficulty of examination subjects with item response theory. Journal of Educational Measurement, 45, 139–157.

40.

Little

R. A.

Rubin

(2002). Statistical analyses with missing data (2nd ed.). John Wiley & Sons.

41.

Lord

F. M.

Wingersky

M. S.

(1984). Comparison of IRT true-score and equipercentile observed-score “equatings.” Applied Psychological Measurement, 8, 453–461.

42.

Mislevy

R. J.

P.-K.

(1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing (ETS Research Report Series No. RR-96-30-ONR) . Educational Testing Service.

43.

Moustaki

Knott

(2000). Weighting for item non-response in attitude scales by using latent variable models with covariates. Journal of the Royal Statistical Society: Series A, 163, 445–459.

44.

Muraki

(1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.

45.

Orlando

Thissen

(2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50–64.

46.

Patterson

B. F.

Ewing

(2013). Validating the use of AP exam scores for college course placement (College Board Research Report No. 2013-2) . College Board.

47.

Pohl

Graefe

Rose

(2014). Dealing with omitted and not-reached items in competence tests. Educational and Psychological Measurement, 74, 423–452.

48.

Qualls

A. L.

(1995). Estimating the reliability of a test containing multiple item formats. Applied Measurement in Education, 8, 111–120.

49.

Raghunathan

Lepkowski

J. M.

van Hoewyk

Solenberger

P. W.

(2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27, 85–96.

50.

Revelle

(2020). psych (R package version 2.0.9). Evanston, Illinois: Northwestern University.

51.

Rose

von Davier

Nagengast

(2017). Modeling omitted and not-reached items in IRT models. Psychometrika, 82, 795–819.

52.

Rose

von Davier

(2010). Modeling nonignorable missing data with item response theory (IRT) (ETS Research Report No. RR-10-11) . Educational Testing Service.

53.

Rose

Wagner

Mayer

Nagengast

(2019). Model-based manifest and latent composite scores in structural equation models. Collabra: Psychology, 5(1), 9.

54.

Rubin

D. B.

(1987). Multiple imputation for nonresponse in surveys. John Wiley & Sons, Inc.

55.

Sarle

W. S.

(1998). Prediction with missing inputs. In Wang

P. P.

(Ed.), JCIS 98 proceedings (pp. 399–402). Research Triangle Park.

56.

Schafer

J. L.

(1997). Analysis of incomplete multivariate data. Chapman & Hall.

57.

Schafer

J. L.

Graham

J. W.

(2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177.

58.

Shin

(2009). How to treat omitted responses in Rasch model-based equating. Practical Assessment, Research, and Evaluation, 14, 1–8.

59.

Sijtsma

van der Ark

L. A.

(2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research, 38, 505–528.

60.

Sinharay

(2021a). Estimating probabilities of passing for examinees with incomplete data in mastery tests. Educational and Psychological Measurement. Advance online publication. https://doi.org/doi=10.1177/00131644211023797

61.

Sinharay

(2021b). Score reporting for examinees with incomplete data on large-scale assessments. Educational Measurement: Issues and Practice, 40(1), 79–91.

62.

Sinharay

Stern

H. S.

Russell

(2001). The use of multiple imputation for the analysis of missing data. Psychological Methods, 6, 317–329.

63.

Smits

Mellenbergh

G. J.

Vorst

H. C. M.

(2002). Alternative missing data techniques to grade point average: Imputing unavailable grades. Journal of Educational Measurement, 39, 187–206.

64.

Sulis

Porcu

(2017). Handling missing data in item response theory. Assessing the accuracy of a multiple imputation procedure based on latent class analysis. Journal of Classification, 34, 327–359.

65.

U.S. Medical Licensing Examination. (2020). 2020 bulletin of information. U.S. Medical Licensing Examination. https://www.usmle.org/pdfs/bulletin/bulletin2020.pdf

66.

van Buuren

Groothuis-Oudshoorn

(2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45, 1–67.

67.

van Ginkel

J. R.

Sijtsma

van der Ark

L. A.

Vermunt

J. K.

(2010). Incidence of missing item scores in personality measurement, and simple item-score imputation. Methodology, 6, 17–30.

68.

Williamson

D. M.

Breyer

F. J.

(2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13.

69.

Wolkowitz

A. A.

Skorupski

W. P.

(2013). A method for imputing response options for missing data on multiple-choice assessments. Educational and Psychological Measurement, 73, 1036–1053.

70.

Xiao

Bulut

(2020). Evaluating the performances of missing data handling methods in ability estimation from sparse data. Educational and Psychological Measurement, 80, 932–954.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB

Reporting Proficiency Levels for Examinees With Incomplete Data

Abstract

Keywords

Get full access to this article

References

Supplementary Material