Is the Area Under Curve Appropriate for Evaluating the Fit of Psychometric Models?

Abstract

In the literature of modern psychometric modeling, mostly related to item response theory (IRT), the fit of model is evaluated through known indices, such as χ², M2, and root mean square error of approximation (RMSEA) for absolute assessments as well as Akaike information criterion (AIC), consistent AIC (CAIC), and Bayesian information criterion (BIC) for relative comparisons. Recent developments show a merging trend of psychometric and machine learnings, yet there remains a gap in the model fit evaluation, specifically the use of the area under curve (AUC). This study focuses on the behaviors of AUC in fitting IRT models. Rounds of simulations were conducted to investigate AUC’s appropriateness (e.g., power and Type I error rate) under various conditions. The results show that AUC possessed certain advantages under certain conditions such as high-dimensional structure with two-parameter logistic (2PL) and some three-parameter logistic (3PL) models, while disadvantages were also obvious when the true model is unidimensional. It cautions researchers about the dangers of using AUC solely in evaluating psychometric models.

Keywords

goodness of fit absolute fit indices AUC item response theory multidimensional structures

Get full access to this article

View all access options for this article.

References

Akaike

(1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723.

Asparouhov

Muthén

(2018). SRMR in Mplus. http://www.statmodel.com/download/SRMR2.pdf

Bartholomew

D. J.

Leung

S. O.

(2002). A goodness of fit test for sparse 2p contingency tables. British Journal of Mathematical and Statistical Psychology, 55, 1–15. https://doi.org/10.1348/000711002159617

Bentler

P. M.

(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), Article 238.

Bergner

Droschler

Kortemeyer

Rayyan

Seaton

Pritchard

D. E.

(2012). Model-based collaborative filtering analysis of student response data: Machine-learning item response theory. International Educational Data Mining Society.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In: Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley.

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29–51.

Bowers

A. J.

Zhou

(2019). Receiver operating characteristic (ROC) area under the curve (AUC): A diagnostic measure for evaluating the accuracy of predictors of education outcomes. Journal of Education for Students Placed at Risk, 24(1), 20–46.

Bozdogan

(1987). Model selection and Akaike s information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52, 345–370.

10.

Cai

Hansen

(2013). Limited-information goodness-of-fit testing of hierarchical item factor models. British Journal of Mathematical and Statistical Psychology, 66, 245–276. https://doi.org/10.1111/j.2044-8317.2012.02050.x

11.

Cai

Maydeu-Olivares

Coffman

D. L.

Thissen

(2006). Limited-information goodness-of-fit testing of item response theory models for sparse 2 tables. British Journal of Mathematical and Statistical Psychology, 59, 173–194. https://doi.org/10.1348/000711005X66419

12.

Cai

Monroe

(2014). A new statistic for evaluating item response theory models for ordinal data (CRESST Report 839). National Center for Research on Evaluation, Standards, and Student Testing. https://eric.ed.gov/?id=ED555726

13.

Carlson

S. E.

(2018). Identifying students at risk of dropping out: Indicators and thresholds using ROC analysis (Doctor of Education (EdD), 114). George Fox University. https://digitalcommons.georgefox.edu/edd/114

14.

Chalmers

R. P.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06

15.

Chalmers

R. P.

(2017). Plausible-value imputation statistics for detecting item misfit. Applied Psychological Measurement, 41, 372–387. https://doi.org/10.1177/0146621617692079

16.

Cheng

Liu

Chen

Huang

Chen

(2019, November 3–7). Dirt: Deep learning enhanced item response theory for cognitive diagnosis. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (pp. 2397–2400). Association for Computing Machinery.

17.

Chernyshenko

O. S.

Stark

Chan

K. Y.

Drasgow

Williams

(2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36(4), 523–562.

18.

DeMars

C. E.

(2016). Partially compensatory multidimensional item response theory models: Two alternate model forms. Educational and Psychological Measurement, 76(2), 231–257. https://doi.org/10.1177/0013164415589595

19.

Drasgow

Levine

M. V.

Williams

E. A.

(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67–86.

20.

Embretson

S. E.

Reise

S. P.

(2013). Item response theory for psychologists. Lawrence Erlbaum Associates. (Original work published 2000)

21.

Fawcett

(2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874.

22.

Ferri

Flach

Hernández-Orallo

Senad

(2005). Modifying ROC curves to incorporate predicted probabilities. In Proceedings of the Second Workshop on ROC Analysis in Machine Learning (pp. 33–40). http://dmip.webs.upv.es/ROCML2005/papers/ferriCRC.pdf

23.

Finlayson

K. J.

Parker

C. N.

Miller

Gibb

Kapp

Ogrin

. . . Edwards

H. E.

(2018). Predicting the likelihood of venous leg ulcer recurrence: The diagnostic accuracy of a newly developed risk assessment tool. International Wound Journal, 15(5), 686–694.

24.

Gonzalez

(2021). Psychometric and machine learning approaches for diagnostic assessment and tests of individual classification. Psychological Methods, 26(2), 236–254. https://doi.org/10.1037/met0000317

25.

Hosmer

D. W.

Lemeshow

(2000). Applied logistic regression (2nd ed.). New York, NY: John Wiley and Sons.

26.

Huang

Ling

C. X.

(2005). Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 17(3), 299–310. https://doi.org/10.1109/TKDE.2005.50

27.

Huggins-Manley

A. C.

Han

(2017). Assessing the sensitivity of weighted least squares model fit indexes to local dependence in item response theory models. Structural Equation Modeling: A Multidisciplinary Journal, 24(3), 331–340.

28.

Immekus

J. C.

Snyder

K. E.

Ralston

P. A.

(2019). Multidimensional item response theory for factor structure assessment in educational psychology research. Frontiers in Education, 4, Article 45. https://www.frontiersin.org/article/10.3389/feduc.2019.00045

29.

Jiang

Fitzgerald

S. R.

Walker

K. W.

(2019a). Modeling time-to-trigger in library demand-driven acquisitions via survival analysis. Library & Information Science Research, 41(3), Article 100968.

30.

Jiang

Walker

Shi

(2019b). Applying AdaBoost to improve diagnostic accuracy. Methodology, 15, 77–87.

31.

Johnson

Semmelroth

(2010). The predictive validity of the early warning system tool. NASSP Bulletin, 94(2), 120–134. https://doi.org/10.1177/0192636510380789

32.

Kang

Chen

T. T.

(2008). Performance of the generalized S-X2 Item fit index for polytomous IRT models. Journal of Educational Measurement, 45(4), 391–406. https://doi.org/10.1111/j.1745-3984.2008.00071.x

33.

Kang

Cohen

A. S.

(2007). IRT model selection methods for dichotomous items. Applied Psychological Measurement, 31(4), 331–358. https://doi.org/10.1177/0146621606292213

34.

Kang

Cohen

A. S.

Sung

H. J.

(2009). Model selection indices for polytomous items. Applied Psychological Measurement, 33(7), 499–518. https://doi.org/10.1177/0146621608327800

35.

Khajah

Lindsey

R. V.

Mozer

M. C.

(2016). How deep is knowledge tracing?https://arxiv.org/abs/1604.02416

36.

Koehler

K. J.

Larntz

(1980). An empirical investigation of goodness-of-fit statistics for sparse multinomials. Journal of the American Statistical Association, 75(370), 336–344. https://doi.org/10.1080/01621459.1980.10477473

37.

Kuhn

(2021). caret: Classification and regression training (R package version 6.0-90). https://CRAN.R-project.org/package=caret

38.

C. V.

Pardos

Z. A.

Meyer

S. D.

Thorp

(2018). Communication at scale in a MOOC using predictive engagement analytics. In Penstein Rosé

Martínez-Maldonado

Hoppe

H. U.

Luckin

Mavrikis

Porayska-Pomsta

McLaren

du Boulay

(Eds.), Artificial intelligence in education (pp. 239–252). Springer. https://doi.org/10.1007/978-3-319-93843-1_18

39.

Lee

(2019). Estimating student ability and problem difficulty using item response theory (IRT) and TrueSkill. Information Discovery and Delivery, 47(2), 67–75. https://doi.org/10.1108/IDD-08-2018-0030

40.

Lemeshow

Hosmer

D. W.

Jr. (1982). A review of goodness of fit statistics for use in the development of logistic regression models. American Journal of Epidemiology, 115(1), 92–106. https://doi.org/10.1093/oxfordjournals.aje.a113284

41.

Lin

T. H.

Dayton

C. M.

(1997). Model selection information criteria for non-nested latent class models. Journal of Educational and Behavioral Statistics, 22, 249–264.

42.

Maydeu-Olivares

(2013). Goodness-of-fit assessment of item response theory models. Measurement, 11, 71–101.

43.

Maydeu-Olivares

Joe

(2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71(4), 713–732. https://doi.org/10.1007/s11336-005-1295-9

44.

Maydeu-Olivares

Joe

(2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49(4), 305–328. https://doi.org/10.1080/00273171.2014.911075

45.

McKinley

R. L.

Mills

C. N.

(1985). A comparison of several goodness-of-fit statistics. Applied Psychological Measurement, 9(1), 49–57. https://doi.org/10.1177/014662168500900105

46.

Mersmann

Trautmann

Steuer

Bornkamp

(2018). truncnorm: Truncated normal distribution (Version 1.0-8). https://CRAN.R-project.org/package=truncnorm

47.

Millard

S. P.

(2013). EnvStats: An R package for environmental statistics. Springer. https://doi.org/10.1007/978-1-4614-8456-1

48.

Nicholls

Wolfe

Besterfield-Sacre

Shuman

(2010). Predicting stem degree outcomes based on eighth grade data and standard test scores. Journal of Engineering Education, 99(3), 209–223. https://doi.org/10.1002/j.2168-9830.2010.tb01057.x

49.

Niemeijer

Feskens

Krempl

Koops

Brinkhuis

M. J. S.

(2020, March 23–27). Constructing and predicting school advice for academic achievement: A comparison of item response theory and machine learning techniques. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 462–471). Association for Computing Machinery. https://doi.org/10.1145/3375462.3375486

50.

Orlando

Thissen

(2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24(1), 50–64. https://doi.org/10.1177/01466216000241003

51.

Park

J. Y.

Cornillie

van der Maas

H. L.

Van Den Noortgate

(2019). A multidimensional IRT approach for dynamically monitoring ability growth in computerized practice environments. Frontiers in Psychology, 10, Article 620. https://doi.org/10.3389/fpsyg.2019.00620

52.

Pham

B. T.

Luu

Van Phong

Trinh

P. T.

Shirzadi

Renoud

. . . Clague

J. J.

(2021). Can deep learning algorithms outperform benchmark machine learning algorithms in flood susceptibility modeling? Journal of Hydrology, 592, Article 125615.

53.

Piech

Spencer

Huang

Ganguli

Sahami

Guibas

Sohl-Dickstein

(2015). Deep knowledge tracing. https://arxiv.org/abs/1506.05908

54.

Pliakos

Joo

S. H.

Park

J. Y.

Cornillie

Vens

Van den Noortgate

(2019). Integrating machine learning into item response theory for addressing the cold start problem in adaptive learning systems. Computers & Education, 137, 91–103. https://doi.org/10.1016/j.compedu.2019.04.009

55.

Rasch

(1960). Studies in mathematical psychology: I—Probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.

56.

R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

57.

Robin

Turck

Hainard

Tiberti

Lisacek

Sanchez

J.-C.

Müller

(2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12(1), Article 77. https://doi.org/10.1186/1471-2105-12-77

58.

Schwarz

(1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.

59.

Silva

Spalenza

Bourguet

J. R.

de Oliveira

(2020). Towards a tailored hybrid recommendation-based system for computerized adaptive testing through clustering and IRT. In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020) (Vol. 1, pp. 260–268). https://doi.org/10.5220/0009419902600268

60.

Steiger

J. H.

Lind

J. C.

(1980) May. Statistically based tests for the number of common factors [Paper presentation]. Annual Meeting of the Psychometric Society, Iowa City, IA, United States.

61.

Stuit

O’Cummings

Norbury

Heppen

Dhillon

Lindsay

Zhu

(2016). Identifying early warning indicators in three Ohio school Districts (REL 2016-118). Regional Educational Laboratory Midwest. https://ies.ed.gov/ncee/edlabs

62.

Liu

Huang

Yin

Chen

. . . Hu

(2018, April 26). Exercise-enhanced sequential modeling for student performance prediction. In. Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1). https://ojs.aaai.org/index.php/AAAI/article/view/11864

63.

Svicher

Romanazzo

De Cesaris

Benemei

Geppetti

Cosci

(2019). Mental Pain Questionnaire: An item response theory analysis. Journal of Affective Disorders, 249, 226–233.

64.

Thissen

Steinberg

(2009). Item response theory. In Millsap

R. E.

Maydeu-Olivares

(Eds.), The SAGE handbook of quantitative methods in psychology (pp. 148–177). SAGE. http://dx.doi.org/10.4135/9780857020994.n7

65.

Tucker

L. R.

Lewis

(1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38(1), 1–10. https://doi.org/10.1007/BF02291170

66.

Walker

K. W.

Jiang

(2019). Application of adaptive boosting (AdaBoost) in demand-driven acquisition (DDA) prediction: A machine-learning approach. The Journal of Academic Librarianship, 45(3), 203–212.

67.

Windle

R. C.

(2017). The measurement of adolescent alcohol problems via item response theory and their 15-year prospective associations with alcohol and other psychiatric disorders. Alcoholism: Clinical and Experimental Research, 41(2), 399–406.

68.

Wright

Masters

(1982). Rating Scale analysis. MESA Press.

69.

Chen

Liu

(2017, April 3–7). Knowledge or gaming? Cognitive modelling based on multiple-attempt response. In Proceedings of the 26th International Conference on World Wide Web Companion (pp. 321–329). https://doi.org/10.1145/3041021.3054156

70.

Ioannidis

N. M.

Zou

(2020). Predicting target genes of non-coding regulatory variants with IRT. Bioinformatics, 36(16), 4440–4448.

71.

Xiao

Shi

Gao

Zhao

Liu

Huang

(2019). A new nomogram from the seer database for predicting the prognosis of gallbladder cancer patients after surgery. Annals of Translational Medicine, 7(23), Article 738. https://doi.org/10.21037/atm.2019.11.112

72.

Paek

Xia

(2017). Investigating the behaviors of M2 and RMSEA2 in fitting a unidimensional model to multidimensional data. Applied Psychological Measurement, 41(8), 632–644.

73.

Yen

W. M.

(1981). Using simulation results to choose a latent trait model. Applied Psychological Measurement, 5(2), 245–262. https://doi.org/10.1177/014662168100500212