Evaluating the Performance of a Regularized Differential Item Functioning Method for Testlet-Based Polytomous Items

Abstract

This study investigated the effect of testlets on regularization-based differential item functioning (DIF) detection in polytomous items, focusing on the generalized partial credit model with lasso penalization (GPCMlasso) DIF method. Five factors were manipulated: sample size, magnitude of testlet effect, magnitude of DIF, number of DIF items, and type of DIF-inducing covariates. Model performance was evaluated using false-positive rate (FPR) and true-positive rate (TPR). Results showed that the simulation had effective control of FPR across conditions, while the TPR was differentially influenced by the manipulated factors. Generally, the small testlet effect did not noticeably affect the GPCMlasso model’s performance regarding FPR and TPR. The findings provide evidence of the effectiveness of the GPCMlasso method for DIF detection in polytomous items when testlets were present. The implications for future research and limitations were also discussed.

Keywords

DIF GPCMlasso polytomous testlet

Get full access to this article

View all access options for this article.

References

Ayilara

O. F.

Sajobi

T. T.

Barclay

Bohm

Jafari Jozani

Lix

L. M.

(2022). A comparison of methods to address item non-response when testing for differential item functioning in multidimensional patient-reported outcome measures. Quality of Life Research, 31(9), 2837–2848. https://doi.org/10.1007/s11136-022-03129-8

Baghaei

Ravand

(2016). Modeling local item dependence in cloze and reading comprehension test items using testlet response theory. Psicológica, 37(1), 85–104.

Bauer

D. J.

Belzak

W. C.

Cole

V. T.

(2020). Simplifying the assessment of measurement invariance over multiple background variables: Using regularized moderated nonlinear factor analysis to detect differential item functioning. Structural Equation Modeling: A Multidisciplinary Journal, 27(1), 43–55. https://doi.org/10.1080/10705511.2019.1642754

Belzak

W. C.

(2019). A regularization approach for testing differential item functioning between two groups [Master’s thesis, The University of North Carolina at Chapel Hill]. ProQuest Dissertations & Theses Global.

Belzak

W. C.

Bauer

D. J.

(2020). Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning. Psychological Methods, 25(6), 673–690. https://doi.org/10.1037/met0000253

Beretvas

S. N.

Walker

C. M.

(2012). Distinguishing differential testlet functioning from differential bundle functioning using the multilevel measurement model. Educational and Psychological Measurement, 72(2), 200–223. https://doi.org/10.1177/0013164411412768

Bollmann

Berger

Tutz

(2018). Item-focused trees for the detection of differential item functioning in partial credit models. Educational and Psychological Measurement, 78(5), 781–804. https://doi.org/10.1177/0013164417722179

Boyd

A. M.

Dodd

Fitzpatrick

(2013). A comparison of exposure control procedures in cat systems based on different measurement models for testlets. Applied Measurement in Education, 26(2), 113–135. https://doi.org/10.1080/08957347.2013.765434

Carter

J. V.

Pan

Rai

S. N.

Galandiuk

(2016). ROC-ing along: Evaluation and interpretation of receiver operating characteristic curves. Surgery, 159(6), 1638–1645.

10.

Carter

N. T.

Zickar

M. J.

(2011). A comparison of the LR and DFIT frameworks of differential functioning applied to the generalized graded unfolding model. Applied Psychological Measurement, 35(8), 623–642. https://doi.org/10.1177/0146621611427898

11.

Chalmers

R. P.

Counsell

Flora

D. B.

(2016). It might not make a big DIF: Improved differential test functioning statistics that account for sampling variability. Educational and Psychological Measurement, 76(1), 114–140. https://doi.org/10.1177/0013164415584576

12.

Chen

Thissen

(1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289. https://doi.org/10.3102/10769986022003265

13.

De Boeck

Cho

S. J.

Wilson

. (2011). Explanatory secondary dimension modeling of latent differential item functioning. Applied Psychological Measurement, 35(8), 583–603. https://doi.org/10.1177/0146621611428446

14.

Dell-Ross

T. L.

(2021). Investigating the performance of (multiple-factor) multiple-group methods for the detection of differential item functioning [Doctoral dissertation, Georgia State University]. ProQuest Dissertations & Theses Global.

15.

Ebrahimi

Bagheri

Shayan

Jafari

(2021). A machine learning approach to assess differential item functioning in psychometric questionnaires using the elastic net regularized ordinal logistic regression in small sample size groups. BioMed Research International, 2021, 1–12. https://doi.org/10.1155/2021/6854477

16.

Eckes

(2014). Examining testlet effects in the TestDaF listening section: A testlet response theory modeling approach. Language Testing, 31(1), 39–61. https://doi.org/10.1177/0265532213492969

17.

El-Komboz

Zeileis

Strobl

(2014). Detecting differential item and step functioning with rating scale and partial credit trees (Technical Reports). LMU Department of Statistics. https://epub.ub.uni-muenchen.de/17984/

18.

Elosua

Wells

C. S.

(2013). Detecting DIF in polytomous items using MACS, IRT and ordinal logistic regression. Psicológica, 34(2), 327–342.

19.

Fukuhara

Kamata

(2011). A bifactor multidimensional item response theory model for differential item functioning analysis on testlet-based items. Applied Psychological Measurement, 35(8), 604–622. https://doi.org/10.1177/0146621611428447

20.

Garrett

(2009). A Monte Carlo study investigating missing data, differential item functioning, and effect size [Doctoral dissertation, Georgia State University]. ProQuest Dissertations & Theses Global.

21.

Goodman

J. T.

Willse

J. T.

Allen

N. L.

Klaric

J. S.

(2011). Identification of differential item functioning in assessment booklet designs with structurally missing data. Educational and Psychological Measurement, 71(1), 80–94. https://doi.org/10.1177/0013164410387341

22.

Hanley

(2020). Assessing the performance of two procedures for detecting differential item functioning within the multilevel partial credit model [Doctoral dissertation, University of Kentucky]. UK Institutional Repository.

23.

Hua

(2019). Differential item functioning for polytomous response items using hierarchical generalized linear model [Doctoral dissertation, University of Pittsburgh]. ProQuest Dissertations & Theses Global.

24.

Huang

H. Y.

(2014). Effects of the common scale setting in the assessment of differential item functioning. Psychological Reports, 114(1), 104–125. https://doi.org/10.2466/03.PR0.114k11w0

25.

Huang

P. H.

(2018). A penalized likelihood method for multi-group structural equation modelling. British Journal of Mathematical and Statistical Psychology, 71(3), 499–522. https://doi.org/10.1111/bmsp.12130

26.

Jafari

Mehrabani-Zeinabad

Javadi

Ghanizadeh

Bagheri

(2022). A machine learning approach to assess differential item functioning of the KINDL quality of life questionnaire across children with and without ADHD. Child Psychiatry & Human Development, 53(5), 980–991. https://doi.org/10.1007/s10578-021-01179-6

27.

Jeon

Rijmen

Rabe-Hesketh

(2013). Modeling differential item functioning using a generalization of the multiple-group bifactor model. Journal of Educational and Behavioral Statistics, 38(1), 32–60. https://doi.org/10.3102/1076998611432173

28.

Jiang

(2019). Regularization methods for detecting differential item functioning [Doctoral dissertation, Boston College]. ProQuest Dissertations & Theses Global.

29.

Jiao

Zhang

(2015). Polytomous multilevel testlet models for testlet-based assessments with complex sampling designs. British Journal of Mathematical and Statistical Psychology, 68(1), 65–83. https://doi.org/10.1111/bmsp.12035

30.

Jin

Kang

(2016). Comparing DIF methods for data with dual dependency. Large-scale Assessments in Education, 4(1), 1–20. https://doi.org/10.1186/s40536-016-0033-3

31.

Kang

H. A.

Han

Kim

Kao

S. C.

(2022). Polytomous testlet response models for technology-enhanced innovative items: Implications on model fit and trait inference. Educational and Psychological Measurement, 82(4), 811–838. https://doi.org/10.1177/00131644211032261

32.

Kogar

E. Y.

(2021). Comparison of testlet effect on parameter estimates using different item response theory models. Journal of Measurement and Evaluation in Education and Psychology, 12(3), 254–266. https://doi.org/10.21031/epod.948227

33.

Komboz

Strobl

Zeileis

(2018). Tree-based global model tests for polytomous rasch models. Educational and Psychological Measurement, 78(1), 128–166. https://doi.org/10.1177/0013164416664394

34.

Koziol

N. A.

Bovaird

J. A.

(2018). The impact of model parameterization and estimation methods on tests of measurement invariance with ordered polytomous data. Educational and Psychological Measurement, 78(2), 272–296. https://doi.org/10.1177/0013164416683754

35.

Kristjansson

Aylesworth

Mcdowell

Zumbo

B. D.

(2005). A comparison of four methods for detecting differential item functioning in ordered response items. Educational and Psychological Measurement, 65(6), 935–953. https://doi.org/10.1177/0013164405275668

36.

Lee

Y. S.

Cohen

Toro

(2009). Examining type I error and power for detection of differential item and testlet functioning. Asia Pacific Education Review, 10(3), 365–375. https://doi.org/10.1007/s12564-009-9039-7

37.

Bolt

D. M.

(2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 30(1), 3–21. https://doi.org/10.1177/0146621605275414

38.

Liang

Jacobucci

(2020). Regularized structural equation modeling to detect measurement bias: Evaluation of lasso, adaptive lasso, and elastic net. Structural Equation Modeling: A Multidisciplinary Journal, 27(5), 722–734. https://doi.org/10.1080/10705511.2019.1693273

39.

Magis

Tuerlinckx

De Boeck

(2015). Detection of differential item functioning using the lasso approach. Journal of Educational and Behavioral Statistics, 40(2), 111–135. https://doi.org/10.3102/1076998614559747

40.

Pakarinen

Lerkkanen

M. K.

Poikkeus

A. M.

Kiuru

Siekkinen

Rasku-Puttonen

Nurmi

J. E.

(2010). A validation of the classroom assessment scoring system in Finnish kindergartens. Early Education and Development, 21(1), 95–124. https://doi.org/10.1080/10409280902858764

41.

Pianta

R. C.

La Paro

K. M.

Hamre

B. K.

(2008). Classroom assessment scoring system™: Manual K-3. Paul H Brookes Publishing.

42.

R Core Team. (2023). R: A language and environment for statistical computing (Version 4.3.2) [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org

43.

Raschka

Liu

Y. H.

Mirjalili

Dzhulgakov

(2022). Machine learning with PyTorch and Scikit-Learn: Develop machine learning and deep learning models with Python. Packt Publishing.

44.

Resnik

Borgia

Heinemann

A. W.

Stevens

Clark

M. A.

(2022). The upper extremity functional scale for prosthesis users (UEFS-P): Subscales for one and two-handed tasks. Disability and Rehabilitation, 45(22), 3768–3778. https://doi.org/10.1080/09638288.2022.2138572

45.

Resnik

Borgia

M. L.

Clark

M. A.

Graczyk

Segil

(2021). Structural validity and reliability of the patient experience measure: A new approach to assessing psychosocial experience of upper limb prosthesis users. PLoS One, 16(12), 1–19. https://doi.org/10.1371/journal.pone.0261865

46.

Schauberger

(2022). GPCMlasso: Differential item functioning in generalized partial credit models (Version 0.1-6) [Computer software]. https://CRAN.R-project.org/package=GPCMlasso

47.

Schauberger

Mair

(2020). A regularization approach for the detection of differential item functioning in generalized partial credit models. Behavior Research Methods, 52(1), 279–294. https://doi.org/10.3758/s13428-019-01224-2

48.

Schneider

Strobl

Zeileis

Debelak

(2022). An R toolbox for score-based measurement invariance tests in IRT models. Behavior Research Methods, 54(5), 2101–2113. https://doi.org/10.3758/s13428-021-01689-0

49.

Scott

(2014). Analytic selection of a valid subtest for DIF analysis when DIF has multiple potential causes among multiple groups [Doctoral dissertation, University of South Florida]. ProQuest Dissertations & Theses Global.

50.

Soysal

Kogar

E. Y.

(2022). Item parameter recovery via traditional 2PL, testlet and bi-factor models for testlet-based tests. International Journal of Assessment Tools in Education, 9(1), 254–266. https://doi.org/10.21449/ijate.948182

51.

Spray

Ackerman

(1987). The effect of item response dependency on trait or ability dimensionality (ACT Research Report Series). American College Testing. https://www.act.org/content/dam/act/unsecured/documents/ACT_RR87-10.pdf

52.

StataCorp. (2021). Stata statistical software (Version 17) [Computer software]. StataCorp LLC.

53.

Tang

K. L.

(1996). Polytomous item response theory (IRT) models and their applications in large-scale testing programs: Review of literature. Educational Testing Service. https://www.ets.org/Media/Research/pdf/RM-96-08.pdf

54.

Tibshirani

(1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x

55.

Tibshirani

Saunders

Rosset

Zhu

Knight

(2005). Sparsity and smoothness via the fused lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(1), 91–108. https://doi.org/10.1111/j.1467-9868.2005.00490.x

56.

Toland

M. D.

Sulis

Giambona

Porcu

Campbell

J. M.

(2017). Introduction to bifactor polytomous item response theory analysis. Journal of School Psychology, 60, 41–63. https://doi.org/10.1016/j.jsp.2016.11.001

57.

Tutz

Schauberger

(2015). A penalty approach to differential item functioning in rasch models. Psychometrika, 80(1), 21–43. https://doi.org/10.1007/s11336-013-9377-6

58.

Wainer

Bradlow

Wang

(2007). Testlet response theory and its application. Cambridge University Press.

59.

Wainer

Wang

(2000). Using a new statistical model for testlets to score TOEFL. Journal of Educational Measurement, 37(3), 203–220. https://doi.org/10.1111/j.1745-3984.2000.tb01083.x

60.

Wallin

Chen

Moustaki

(2024). DIF analysis with unknown groups and anchor items. Psychometrika, 89(1), 267–295. https://doi.org/10.1007/s11336-024-09948-7

61.

Wang

Zhu

(2023). Using lasso and adaptive lasso to identify DIF in multidimensional 2PL models. Multivariate Behavioral Research, 58(2), 387–407. https://doi.org/10.1080/00273171.2021.1985950

62.

Wang

Tay

Drasgow

(2013). Detecting differential item functioning of polytomous items for an ideal point response process. Applied Psychological Measurement, 37(4), 316–335. https://doi.org/10.1177/0146621613476156

63.

Wang

W. C.

Wilson

(2005). Assessment of differential item functioning in testlet-based items using the rasch testlet model. Educational and Psychological Measurement, 65(4), 549–576. https://doi.org/10.1177/0013164404268677

64.

Weese

J. D.

Turner

R. C.

Liang

Ames

Crawford

(2022). Implementing a standardized effect size in the POLYSIBTEST Procedure. Educational and Psychological Measurement, 83(2), 401–427. https://doi.org/10.1177/00131644221081011

65.

Wijayanto

Bucur

I. G.

Mul

Groot

van Engelen

B. G.

Heskes

(2023). Semi-automated rasch analysis with differential item functioning. Behavior Research Methods, 55(6), 3129–3148. https://doi.org/10.3758/s13428-022-01947-9

66.

Wurm

M. J.

Rathouz

P. J.

Hanlon

B. M.

(2021). Regularized ordinal regression and the ordinalNet R package. Journal of Statistical Software, 99(6), 1–42. https://doi.org/10.18637/jss.v099.i06

67.

Yesiltas

Paek

(2020). A log-linear modeling approach for differential item functioning detection in polytomously scored items. Educational and Psychological Measurement, 80(1), 145–162. https://doi.org/10.1177/0013164419853000

68.

Zhang

C.-H.

(2010). Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics, 38(2), 894–942. https://doi.org/10.1214/09-AOS729