In common item equating, the existence of item outliers may impact the accuracy of equating results and bring significant ramifications to the validity of test score interpretations. Therefore, common item equating should involve a screening process to flag outlying items and exclude them from the common item set before equating is conducted. The current simulation study demonstrated that the sampling variance associated with the item response theory (IRT) item parameter estimates can help detect outliers in the common items under the 2-PL and 3-PL IRT models. The results showed the proposed sampling variance statistic (SV) outperformed the traditional displacement method with cutoff values of 0.3 and 0.5 along a variety of evaluation criteria. Based on the favorable results, item outlier detection statistics based on estimated sampling variability warrant further consideration in both research and practice.
AltmanD. G.BlandJ. M. (1994). Statistics notes: Diagnostic tests 1: Sensitivity and specificity. British Medical Journal, 308(6943), 1552. https://doi.org/10.1136/bmj.308.6943.1552
2.
CookL. L.PatersonN. S. (1987). Problems related to the use of conventional and item response theory equating methods in less than optimal circumstances. Applied Psychological Measurement, 11(3), 225–244. https://doi.org/10.1177/014662168701100302
3.
DonoghueJ. R.IshamS. P. (1998). A comparison of procedures to detect item parameter drift. Applied Psychological Measurement, 22(1), 33–51. https://doi.org/10.1177/01466216980221002
4.
DoransN. J.LiuJ.HammondS. (2008). Anchor test type and population invariance: An exploration across subpopulations and test administrations. Applied Psychological Measurement, 32(1), 81–97. https://doi.org/10.1177/0146621607311580
5.
GoldsteinH. (1983). Measuring changes in educational attainment over time: Problems and possibilities. Journal of Educational Measurement, 20(4), 369–377. https://doi.org/10.1111/j.1745-3984.1983.tb00214.x
6.
HambletonR. K.RogersH. J. (1989). Detecting potentially biased test items: Comparison of IRT area and Mantel-Haenszel methods. Applied Measurement in Education, 2(4), 313–334. https://doi.org/10.1207/s15324818ame0204_4
7.
HansonB. A.ZengL. (1995). ST: A computer program for IRT scale transformation [Computer Software]. ACT.
8.
HansonB. A.ZengL. (2004). PIE: A computer program for IRT equating [Computer Software]. ACT.
9.
HeY.CuiZ.FangY.ChenH. (2013). Using a linear regression method to detect outliers in IRT common item equating. Applied Psychological Measurement, 37(7), 522–540. https://doi.org/10.1177/0146621613483207
10.
HeY.CuiZ.OsterlindS. J. (2015). New robust scale transformation methods in the presence of outlying common items. Applied Psychological Measurement, 39(8), 613–626. https://doi.org/10.1177/0146621615587003
11.
HoggR. V. (1979). Statistical robustness: One view on its use in applications today. The American Statistician, 33(3), 108–115. https://doi.org/10.2307/2683810
12.
HuangC. Y.ShyvC. Y. (2003). The impact of item parameter drift on equating. [Paper presentation]. Annual Meeting of the National Council on Measurement in Education, Chicago, IL, 22–24 April 2003.
13.
HuynhH.MeyerP. (2010). Use of robust z in detecting unstable items in item response theory models. Practical Assessment, Research and Evaluation, 15, 2. https://doi.org/10.7275/ycx6-e864
14.
KleinL. W.JarjouraD. (1985). The importance of content representation for common-item equating with nonrandom groups. Journal of Educational Measurement, 22(3), 197–206. https://doi.org/10.1111/j.1745-3984.1985.tb01058.x
15.
KolenM. J.BrennanR. L. (2014). Test equating, scaling, and linking. Springer-Verlag.
16.
LiuC.JurichD. (2021). Comparison of outlier detection methods in NEAT design. In WibergM.MolenaarD.GonzálezJ.BöckenholtU.KimJ. S. (Eds). Quantitative Psychology. Springer Proceedings in Mathematics & Statistics, vol 353. Springer, Cham. https://doi.org/10.1007/978-3-030-74772-5_20
17.
LordF. M. (1977). A study of item bias, using item characteristic curve theory. In PoortingaY. H. (Ed.), Basic problems in cross-cultural psychology (pp. 19–29). Swets & Zeitlinger.
18.
LordF. M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates.
19.
LuoX.KimD. (2018). A top-down approach to designing the computerized adaptive multistage test. Journal of Educational Measurement, 55(2), 243–263. https://doi.org/10.1111/jedm.12174
20.
MannaV. F.GuL. (2019). Different methods of adjusting for form difficulty under the Rasch model: Impact on consistency of assessment results (Research Report No. RR-19-08). Educational Testing Service.
21.
MillerG. E.RotouO.TwingJ. S. (2004). Evaluation of the 0.3 logit screening criterion in common item equating. Journal of Applied Measurement, 5(2), 172–177.
22.
MurakiE.EngelhardG. (1989). Examining differential item functioning with BIMAIN. [Paper presentation]. Annual meeting of the American Educational Research Association, San Francisco, CA, 27–31 March 1989.
23.
MurphyS.LittleI.FanM.LinC.KirkpatrickR. (2010). The impact of different anchor stability methods on equating results and student performance. [Paper presentation]. Annual meeting of the National Council on Measurement in Education, Washington, DC.
RajuN. S. (1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14(2), 197–207. https://doi.org/10.1177/014662169001400208
26.
SmithR. M.SuhK. K. (2003). Rasch fit statistics as a test of the invariance of item parameter estimates. Journal of Applied Measurement, 4(2), 153–163.
27.
StockingM. L.LordF. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7(2), 201–210. https://doi.org/10.1177/014662168300700208
28.
WellsC. S.HambletonR. K.KirkpatrickR.MengY. (2014). An examination of two procedures for identifying consequential item parameter drift. Applied Measurement in Education, 27(3), 214–231. https://doi.org/10.1080/08957347.2014.905786
29.
WrightB. D.StoneM. H. (1979). Best test design. MESA Press.
