This study provides an overview of the effect of differential item functioning (DIF) on measurement precision, test information function (TIF), and test effectiveness in computer adaptive tests (CATs). Simulated data for the study was produced and analyzed with the Rstudio. During the data generation process, item pool size, DIF type, DIF percentage, item selection method for CAT, and the test termination rules were considered changed conditions. Sample size and ability parameter distribution, Item Response Theory (IRT) model, DIF size, ability estimation method, test starting rule, and item usage frequency method regarding CAT conditions were considered fixed conditions. To examine the effect of DIF, measurement precision, TIF and test effectiveness were calculated. Results show DIF has negative effects on measurement precision, TIF, and test effectiveness. In particular, statistically significant effects of the percentage DIF items and DIF type are observed on measurement precision.
BabcockB.WeissD. J. (2012). Termination criteria in computerized adaptive tests: Do variable-length CAT’s provide efficient and effective measurement?Journal of Computerized Adaptive Testing, 1(1), 1–18. https://doi.org/10.7333/1212-0101001
2.
BarradaJ. R.OleaJ.PonsodaV.AbadF. J. (2010). A method for the comparison of ıtem selection rules in computerized adaptive testing. Applied Psychological Measurement, 34(6), 438–452. https://doi.org/10.1177/0146621610370152
3.
BattauzM. (2017). On wald tests for differential item functioning detection. Statistical Methods and Applications, 28(1), 103–118. https://doi.org/10.1007/s10260-018-00442-w
4.
BerganJ. R. (2010). Assessing the relative fit of alternative item response theory models to the data. Assessment Technology Inc.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley.
7.
BockR. D.MislevyR. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431–444. https://doi.org/10.1177/014662168200600405
8.
Boztunc-OzturkN.DoganN. (2015). Investigating item exposure control methods in computerized adaptive testing. Educational Sciences: Theory and Practice, 15(1), 85–98. https://doi.org/10.12738/estp.2015.1.2593
9.
BringsjordE. L. (2001). Computerized-adaptive versus paper-and-pencil testing environments: An experimental analysis of examinee experience. (Doctoral Dissertation). ProQuest Dissertations and Theses Database. (UMI No. 3014369).
10.
BudgellG. R.RajuN. S.QuartettiD. A. (1995). Analysis of differential item functioning in translated assessment instruments. Applied Psychological Measurement, 19(4), 309–321. https://doi.org/10.1177/014662169501900401
11.
BulutO. (2015). An empirical analysis of gender-based DIF due to test booklet effect. European Journal of Radiology Extra, 3(1), 7–16.
12.
BulutO.LeiM.GuoQ. (2016). Item and testlet position effects in computer-based alternate assessments for students with disabilities. International Journal of Research and Method in Education, 41(2), 169–183. https://doi.org/10.1080/1743727X.2016.1262341
13.
CamilliG. (2006). Test fairness. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 221–256). American Council on Education & Praeger Publishers.
14.
CamilliG.ShepardL. A. (1994). Methods for identifying biased test items (4th ed.). Sage Publications, Inc.
15.
ChangH.YingZ. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3), 213–229. https://doi.org/10.1177/014662169602000303
ChenS.-Y.AnkenmannR. D.ChangH.-H. (2000). A comparison of item selection rules at the early stages of computerized adaptive testing. Applied Psychological Measurement, 24(3), 241–255. https://doi.org/10.1177/01466210022031705
18.
ChiuP. C. (2012, April). The effect of item position on state mathematics assessment. In Paper presented at the annual meeting of the American Educational Research Association. American Educational Research Association.
19.
ChiuP. C.IrwinP. M. (2011, April). Chronological item ordering: Does it make a difference on a state history and government assessment? In Paper presented at the annual meeting & training sessions of the national council on measurement in education. ERIC.
20.
ChuM. W.LaiH. (2013). Detecting biased items using CATSIB to increase fairness in computer adaptive tests. Alberta Journal of Educational Research, 59(4), 630–643. https://doi.org/10.55016/ojs/ajer.v59i4.55750
21.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
22.
DebeerD.JanssenR. (2013). Modeling item‐position effects within an IRT framework. Journal of Educational Measurement, 50(2), 164–185. https://doi.org/10.1111/jedm.12009
23.
DengH.AnsleyT.ChangH. H. (2010). Stratified and maximum information item selection procedures in computer adaptive testing. Journal of Educational Measurement, 47(2), 202–226. https://doi.org/10.1111/j.1745-3984.2010.00109.x
24.
DesjardinsC. D.BulutO. (2018). Handbook of educational measurement and pyschometrics using R. Taylor & Francis Group.
25.
DoddB. G.KochW. R.de AyalaR. J. (1993). Computerized adaptive testing using the partial credit Model effects of item pool characteristics and different stopping rules. Educational and Psychological Measurement, 53(1), 61–77. https://doi.org/10.1177/0013164493053001005
26.
EconomidesA. A. (2005). Computer adaptive testing quality requirements. In RichardsG. (Ed.), Proceedings E-learn 2005 world conference on E-learning in corporate, government, healthcare, and higher education (pp. 288–295). AACE (Association for the Advancement of Computing in Education).
27.
EggenT. J. H. M. (1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement, 23(3), 249–261. https://doi.org/10.1177/01466219922031365
28.
EmbrestonS. E.ReiseS. P. (2000). Item response theory for psychologists. Erlbaum Ass., Inc.
29.
Erdem-KaraB.DoganN. (2022). The Effect of ratio of items indicating differential item functioning on computer adaptive and multi-stage tests. International Journal of Assessment Tools in Education, 9(3), 682–696. https://doi.org/10.21449/ijate.1105769
30.
EroğluM. G. (2013). Bireyselleştirilmiş bilgisayarlı test uygulamalarında farklı sonlandırma kurallarının ölçme kesinliği ve test uzunluğu açısından karşılaştırılması [Comparison of different test termination rules in terms of measurement precision and test length in computerized adaptive testing]. [Unpublished Doctoral Dissertation]. Hacettepe University.
31.
FeinbergR. A.RubrightJ. D. (2016). Conducting simulation studies in psychometrics. Educational Measurement: Issues and Practice, 35(2), 36–49. https://doi.org/10.1111/emip.12111
32.
FinchW. H. (2016). Detection of differential item functioning for more than two groups: A Monte Carlo comparison of methods. Applied Measurement in Education, 29(1), 30–45. https://doi.org/10.1080/08957347.2015.1102916
33.
FinchW. H.FrenchB. F. (2007). Detection of crossing differential item functioning: A comparison of four methods. Educational and Psychological Measurement, 67(4), 565–582. https://doi.org/10.1177/0013164406296975
34.
FlaugherR. (2000). Item pools. In WainerH. (Ed.), Computerized adaptive testing: A primer (2nd ed., pp. 37–60). Lawrence Erlbaum Associates.
35.
FrenchB. F.MallerS. J. (2007). Iterative purification and effect size use with logistic regression for differential item functioning detection. Educational and Psychological Measurement, 67(3), 373–393. https://doi.org/10.1177/0013164406294781
36.
FrenchB. F.ThompsonT. D. (2003). The evaluation of exposure control procedures for an operational CAT. In The annual meeting of the American Educational Research Association. American Educational Research Association.
37.
GaoX. (2019). A comparison of six DIF detection methods. [Unpublished Master’s Thesis]. University of Connecticut Graduate School.
38.
GierlM. J.LaiH.LiJ. (2013). Identifying differential item functioning in multi-stage computer adaptive testing. Educational Research and Evaluation, 19(2-3), 188–203. https://doi.org/10.1080/13803611.2013.767622
39.
HambletonR. K.SwaminathanH.RogersH. J. (1991a). Fundamentals of item response theory. Sage Publications Inc.
40.
HambletonR. K.ZaalJ. N.PietersJ. P. M. (1991b). Computerized adaptive testing: Theory, applications, and standards. In Advances in educational and psychological testing: Theory and applications (pp. 341–366). Springer. https://doi.org/10.1007/978-94-009-2195-5_12
41.
HanK. C. (2011). User’s manual: SimulCAT. Graduate Management Admission Council.
42.
HanK. C. T. (2018). Conducting simulation studies for computerized adaptive testing using SimulCAT: An instructional piece. Journal of Educational Evaluation for Health Professions, 15(20), 1–11. https://doi.org/10.3352/jeehp.2018.15.20
JinK.ChenH.WangW. (2015). Assessing differential item functioning in multiple grouping variables with factorial logistic regression. In MillsapB.WangA. (Eds.), Quantitative psychology research (pp. 243–259). Springer.
47.
JinK.-Y.ChenH.-F.WangW.-C. (2018). Using odds ratios to detect differential item functioning. Applied Psychological Measurement, 42(8), 613–629. https://doi.org/10.1177/0146621618762738
48.
KengL. (2008). A comparison of the performance of testlet-based computer adaptive tests and multistage tests. [Doctoral Dissertation]. The University of Texas at Austin.
49.
KimJ. (2010). Controlling type 1 error rate in evaluating differential item functioning for four DIF methods: Use of three procedures for adjustment of multiple item testing. [Unpublised Doctoral Dissertation]. Georgia State University.
50.
KingsburyG. G.ZaraA. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2(4), 359–375. https://doi.org/10.1207/s15324818ame0204_6
51.
LeiP. W.ChenS. Y.YuL. (2006). Comparing methods of assessing differential item functioning in a computerized adaptive testing environment. Journal of Educational Measurement, 43(3), 245–264. https://doi.org/10.1111/j.1745-3984.2006.00015.x
52.
LiY. H.LissitzR. W. (2004). Applications of the analytically derived asymptotic standard errors of item response theory item parameter estimates. Journal of Educational Measurement, 41(2), 85–117. https://doi.org/10.1111/j.1745-3984.2004.tb01109.x
53.
Lopez RivasG. E. (2012). Detection and classification of DIF types using parametric and nonparametric methods: A comparison of the IRT-likelihood ratio test, crossing-SIBTEST, and logistic regression procedures. [Unpublished Doctoral Dissertation]. University of South Florida.
54.
LuechtR. M.NungesterR. J. (1998). Some practical examples of computer-adaptive sequential testing. Journal of Educational Measurement, 35(3), 229–249. https://doi.org/10.1111/j.1745-3984.1998.tb00537.x
55.
LunzM. E.StahlJ. A.BergstromB. A. (1993, April). Targeting, test length, test precision, and decision accuracy for computerized adaptive tests. In Paper presented at the annual meeting of the American educational research association. American Educational Research Association.
56.
MagisD. (2022). Collection of methods to detect dichotomous differential item functioning (DIF). Springer. R Package Version, 5.1.
57.
MakranskyG.GlasC. A. W. (2013). Modeling differential item functioning with group-specific item parameters: A computerized adaptive testing application. Measurement, 46(9), 3228–3237. https://doi.org/10.1016/j.measurement.2013.06.020
58.
McDonaldP. L. (2002). Computer adaptive test for measuring personality factors using item response theory. [Unpublished Doctoral Dissertation]. IACAT.
59.
MeijerR. R.NeringM. L. (1999). Computerized adaptive testing: Overview and introduction. Applied Psychological Measurement, 23(3), 187–194. https://doi.org/10.1177/01466219922031310
60.
MellenbergG. J. (1989). Item bias and item response theory. International Journal of Educational Research: Applications of Item Response Theory, 13(2), 127–143. https://doi.org/10.1016/0883-0355(89)90002-5
61.
MundformD. J.SchafferJ.KimM. J.ShawD.ThongteeraparpA.SupawanP.SupawanP. (2011). Number of replications required in Monte Carlo simulation studies: A synthesis of four studies. Journal of Modern Applied Statistical Methods, 10(1), 19–28. https://doi.org/10.22237/jmasm/1304222580
62.
NarayananP.SwaminathanH. (1994). Performance of the Mantel-Haenszel and simultaneous item bias procedures for detecting differential item functioning. Applied Psychological Measurement, 18(4), 315–328. https://doi.org/10.1177/014662169401800403
OshimaT. C.WrightK.WhiteN. (2015). Multiple-group noncompensatory differential item functioning in Raju’s differential functioning of items and tests. International Journal of Testing, 15(3), 254–273. https://doi.org/10.1080/15305058.2015.1009980
65.
PenfieldR. D. (2001). Assessing differential item functioning among multiple groups: A comparison of three mantel-Haenszel procedures. Applied Measurement in Education, 14(3), 235–259. https://doi.org/10.1207/S15324818AME1403_3
66.
PiromsombatC. (2014). Differential item functioning in computerized adaptive testing: Can CAT self-adjust enough? [Doctoral Dissertation]. University of Minnesota.
67.
RockoffD. (2018). A randomization test for the detection of differential item functioning. [Unpublished Doctoral Dissertation]. The University of Arizona.
68.
RogersH. J.SwaminathanH. (1993). A comparison of logistic regression and Mantel-Haenszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17(2), 105–116. https://doi.org/10.1177/014662169301700201
69.
RollinsJ. D.III. (2018). A comparison of observed score approaches to detecting differential item functioning among multiple groups. (Unpublished Doctoral Dissertation). The University of North Carolina at Greensboro, Greensboro.
70.
SaatciogluO. (2023). Çoklu grup değişen madde fonksiyonu belirleme tekniklerinin farkli simülasyon koşullarinda karşilaştirilmasi [Comparing the techniques for determining multiple-group differential item functioning under different simulation conditions]. [Unpublished Doctoral Dissertation]. Ankara University.
71.
SariH. I.Huggins-ManleyA. C. (2017). Examining content control in adaptive tests: Computerized adaptive testing vs. Computerized adaptive multistage testing. Educational Sciences: Theory and Practice, 17, 1759–1781. https://doi.org/10.12738/estp.2017.5.0484
72.
SchaubergerG.TutzG. (2016). Detection of differential item functioning in Rasch models by boosting techniques. British Journal of Mathematical and Statistical Psychology, 69(1), 80–103. https://doi.org/10.1111/bmsp.12060
73.
SuY.H.ChangJ.TsaiH. (2018). Using Credible Intervals to Detect Differential Item Functioning in IRT Models. In Quantitative Psychology (pp. 297–304). Springer.
74.
SuY.-H.TsaiH. (2019). Detection of differential item functioning via the credible intervals and odds ratios methods. In Springer proceedings in mathematics & statistics (Vol. 265, pp. 319–330). Springer.
75.
SulakS. (2013). Bireyselleştirilmiş bilgisayarlı test uygulamalarında kullanılan madde seçme yöntemlerinin karşılaştırılması [Comparision of item selection methods in computerized adaptive testing]. [Unpublished Doctoral Dissertation]. Hacettepe University.
76.
SvetinaD.RutkowskiL. (2014). Detecting differential item functioning using generelized logistic regression in the context of large-scale assessments. Large Scale Assessments in Education, 2(4), 1–17. https://doi.org/10.1186/s40536-014-0004-5
77.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
78.
ThissenD.MislevyR. J. (2000). Testing algorithms. In WainerH. (Ed.), Computerized adaptive testing: A primer (2nd ed.). Lawrence Erlbaum Associates.
79.
ThompsonN. A. (2007). A practitioner’s guide for variable-length computerized classification testing. Practical Assessment, Research and Evaluation, 12(1), 1–13. https://doi.org/10.7275/fq3r-zz60
80.
ThompsonN. A.WeissD. A. (2011). A framework for the development of computerized adaptive tests. Practical Assessment, Research and Evaluation, 16(1), 1–9. https://doi.org/10.7275/wqzt-9427
81.
TolandM. (2008). Determining the accuracy of item parameter standard error of estimates in BILOG-MG 3. [Unpublished Doctoral Dissertation]. The Graduate College at the University of Nebraska.
82.
TrendtelM.RobitzschA. (2018). Modeling item position effects with a Bayesian item response model applied to PISA 2009–2015 data. Psychological Test and Assessment Modeling, 60(2), 241–263. https://doi.org/10.3102/1076998620931016
83.
TutzG.SchaubergerG. (2015). A penalty approach to differential item functioning in rasch models. Psychometrika, 80(1), 21–43. https://doi.org/10.1007/s11336-013-9377-6
van der LindenW. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63(2), 201–216. https://doi.org/10.1007/bf02294775
86.
van der LindenW. (2003). Some alternatives to Sympson-Hetter item-exposure control in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 28(3), 249–265. https://doi.org/10.3102/10769986028003249
87.
van der LindenW. J.GlasG. A. W. (Eds.), (2010). Computerized adaptive testing: Theory and practice. Springer.
88.
WainerH. (2000). Introduction and history. In WainerH. (Ed.), Computerized adaptive testing: A primer (2nd ed., p. 1–22). Lawrence Erlbaum Associates.
WangH. P.KuoB. C.TsaiY. H.LiaoC. H. (2012). A Cerf-Based computerized testing system for Chinese proficiency. TOJET: The Turkish Journal of Educational Technology, 11(4), 1–12.
91.
WangK. (2017). A fair comparison of the performance of computerized adaptive testing and multistage adaptive testing. [Doctoral Dissertation]. Michigan State University.
92.
WangS.JiaoH.YoungM. J.BrooksT. E.OlsonJ. (2007). A meta-analysis of testing mode effects in Grade K–12 mathematics tests. Educational and Psychological Measurement, 67(2), 219–238. https://doi.org/10.1177/0013164406288166
93.
WangS.JiaoH.YoungM. J.BrooksT. E.OlsonJ. (2008). Comparability of computer based and paper-and-pencil testing in K-12 reading assessment: A meta-analysis of testing mode effects. Educational and Psychological Measurement, 68(1), 5–24. https://doi.org/10.1177/0013164407305592
94.
WangS.WangT. (2001). Precision of warm’s weighted likelihood estimates for a polytomous model in computerized adaptive testing. Applied Psychological Measurement, 25(4), 317–331. https://doi.org/10.1177/01466210122032163
95.
WeissD. J. (2004). Computerized adaptive testing for effective and efficient measurement in counseling and education. Measurement and Evaluation in Counseling and Development, 37(2), 70–84. https://doi.org/10.1080/07481756.2004.11909751
96.
WeissD. J.KingsburyG. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x
97.
WiseS. L.KingsburyG. G. (2000). Practical issues in developing and maintaining a computerized adaptive testing program. Psicológica, 21(1), 135–155.
98.
WoodsC. M. (2008). Likelihood-ratio DIF testing: Effects of nonnormality. Applied Psychological Measurement, 32(7), 511–526. https://doi.org/10.1177/0146621607310402
99.
YaoL. (2013). Comparing the performance of five multidimensional CAT selection procedures with different stopping rules. Applied Psychological Measurement, 37(1), 3–23. https://doi.org/10.1177/0146621612455687
100.
ZumboB. D.ThomasD. R. (1996). A measure of DIF effect size using logistic regression procedures. In Paper presented at the national board of medical examiners. Philadelphia. Retrieved from:https://www.educ.ubc.ca/faculty/zumbo/cv.htm
101.
ZumboB. D. A. (1999). Handbook on the theory and methods of differential item functioning: Logistic regression modeling as a unitary framework for binary and likert-type item scores. Directorate of Human Resources Research and Evaluation, Department of National Defense.
102.
ZwickR. (2000). The assessment of Differential item functioning in computer adaptive tests. In van der LindenW. J.GlasC. A. W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 221–244). Kluwer Academic Publishers.
103.
ZwickR. (2010). The investigation of differential item functioning in adaptive tests. In van der LindenW. J.GlasC. A. W. (Eds.), Elements of adaptive testing (pp. 331–352). Springer.
104.
ZwickR.BridgemanB. (2014). Evaluating validity, fairness, and differential item functioning in multistage testing. In YanD.von-DavierA. A.LewisC. (Eds.), Computerized multistage testing (pp. 309–322). CRC Press; Taylor & Francis Group.
105.
ZwickR.ThayerD. T. (2002). Application of an empirical Bayes enhancement of Mantel-Haenszel differential item functioning analysis to a computerized adaptive test. Applied Psychological Measurement, 26(1), 57–76. https://doi.org/10.1177/0146621602026001004
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
