Continuously administered examination programs, particularly credentialing programs that require graduation from educational programs, often experience seasonality where distributions of examine ability may differ over time. Such seasonality may affect the quality of important statistical processes, such as item response theory (IRT) item calibration and equating. The lead time required for producing pre-equated test forms in the continuous testing framework further complicates issues. This study examines the effect of seasonality in test data on Rasch IRT item parameter estimates. Data came from four credentialing examination programs that represented both programs with and without seasonality, as well as medium and low examinee volume. Results showed that calibrating items during certain times can lead to quite poor item parameter estimates. While certain programs could conduct IRT calibrations without waiting for the full examination cycle to be completed, other types of programs should wait as long as possible before calibrating items.
BabcockB.AlbanoA. D. (2012). Rasch scale stability in the presence of item parameter and trait drift. Applied Psychological Measurement, 36, 555-580. doi:10.1177/0146621612455090
2.
BabcockB.WeissD. J. (2012). Termination criteria in computerized adaptive tests: Do variable-length CATs provide efficient and effective measurement?Journal of Computerized Adaptive Testing, 1, 1-18. doi:10.7333/1212-0101001
3.
BanJ.HansonB. A.WangT.YiQ.HarrisD. J. (2001). A comparative study of on-line pretest item – Calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38, 191-212. doi:10.1111/j.1745-3984.201.tb01123.x
4.
BanJ.HansonB. A.YiQ.HarrisD. J. (2002). Data spareness and on-line pretest item calibration-scaling methods in CAT. Journal of Educational Measurement, 39, 207-218. doi:10.1111/j.1745-3984.2002.tb01174.x
5.
BondT. G.FoxC. M. (2007). Applying the Rasch model. Mahwah, NJ: Erlbaum.
6.
de AyalaR. J. (2009). The theory and practice of item response theory. New York, NY: Guilford Press.
7.
DrasgowF. (1989). An evaluation of marginal maximum likelihood estimation for the two-parameter logistic model. Applied Psychological Measurement, 13, 77-90. doi:10.1177/014662168901300108
8.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
9.
EngelhardG. (2013). Invariant measurement. New York, NY: Routledge.
10.
FischerG. H.MolenaarI. W. (1995). Rasch models: Foundations, recent developments, and applications. New York, NY: Springer-Verlag.
11.
HarwellM. R.JanoskyJ. E. (1991). An empirical study of the effects of small datasets and varying prior variances on item parameter estimation in BILOG. Applied Psychological Measurement, 15, 279-291. doi:10.1177/014662169101500308
HulinC. L.LissakR. I.DrasgowF. (1982). Recovery of two- and three-parameter logistic item characteristic curves: A Monte Carlo study. Applied Psychological Measurement, 6, 249-260. doi:10.1177/014662168200600301
KimS. (2006). A comparative study of IRT fixed parameter calibration methods. Journal of Educational Measurement, 43, 355-381. doi:10.1111/j.1745-3984.2006.00021.x
16.
KingsburyG. G. (2009). Adaptive item calibration: A process for estimating item parameters within a computerized adaptive test. In WeissD. J. (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing.
17.
KirisciL.HsuT.-C.YuL. (2001). Robustness of item parameter estimation programs to assumptions of unidimensionality and normality. Applied Psychological Measurement, 25, 146-162. doi:10.1177/01466210122031975
18.
KolenM. J.BrennanR. L. (2004). Test equating, scaling, and linking (2nd ed.). New York, NY: Springer-Verlag.
19.
LinacreJ. M. (2001). WINSTEPS Rasch measurement computer program (Version 3.31) [Computer software]. Chicago, IL: Winsteps.com.
20.
LuechtR. M.SireciS. G. (2011). A review of models for computer-based testing (College Board Research Report 2011-12). New York, NY: College Board.
MeyerJ. PHaileyE. (2012). A study of Rasch, partial credit, and rating scale model parameter recovery in WINSTEPS and jMetrik. Journal of Applied Measurement, 13, 248-258.
23.
O’NeillTPeabodyM.TanR.DuY. (2013). How much item drift is too much?Rasch Measurement Transaction, 27, 1423-1424.
24.
ParshallC. G.SprayJ. A.KalohnJ. C.DaveyT. (2002). Practical considerations in computer-based testing. New York, NY: Springer.
25.
SchmeiserC. B.WelchC. J. (2006). Test development. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 307-353). Washington, DC: American Council on Education.
26.
SeongT. (1990). Sensitivity of marginal maximum likelihood estimation of item and ability parameters to the characteristics of the prior ability distributions. Applied Psychological Measurement, 14, 299-311. doi:10.1177/014662169001400307
27.
SkaggsG.StevensonJ. (1989). A comparison of pseudo-Bayesian and joint maximum likelihood procedures for estimating item parameters in the three-parameter IRT model. Applied Psychological Measurement, 13, 391-402. doi:10.1177/014662168901300405
28.
SmithR. M. (2004). Detecting item bias with the Rasch model. Journal of Applied Measurement, 5, 430-449.
29.
StockingM. L.LordF. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, 201-210. doi:10.1177/014662168300700208
30.
StoneC. A. (1992). Recovery of marginal maximum likelihood estimates in the two-parameter logistic response model: An evaluation of MULTILOG. Applied Psychological Measurement, 16, 1-16. doi:10.1177/014662169201600101
SwaminathanH.GiffordJ. A. (1982). Bayesian estimation in the Rasch model. Journal of Educational Statistics, 7, 175-191. doi:10.3102/10769986007003175
33.
SwaminathanH.GiffordJ. A. (1983). Estimation of parameters in the three-parameter latent trait model. In WeissD. J. (Ed.), New horizons in testing (pp. 9-30). New York, NY: Academic Press.
34.
SwaminathanH.GiffordJ. A. (1985). Bayesian estimation in the two-parameter logistic model. Psychometrika, 50, 349-364. doi:10.1007/BF02294110
35.
ThompsonN. A. (2007). A practitioner’s guide for variable-length computerized classification testing. Practical Assessment, Research and Evaluation, 12(1). Retrieved from http://pareonline.net/pdf/v12n1.pdf
36.
TongY.WuS.XuM. (2008, March). A comparison of pre-equating and post-equating using large-scale assessment data. Paper presented at the annual meeting of the American Educational Research Association. New York, NY.
37.
van der LindenW. J. (2005). Linear models for optimal test design. New York, NY: Springer.
38.
van den WollenbergA. L.WierdaF. W.JansenP. G. W. (1988). Consistency of Rasch model parameter estimation: A simulation study. Applied Psychological Measurement, 12, 307-313. doi:10.1177/014662168801200308
39.
WangW.-C.ChenC.-T. (2005). Item parameter recovery, standard error estimates, and fit statistics of the WINSTEPS program for the family of Rasch models. Educational and Psychological Measurement, 65, 376-404. doi:10.1177/0013164404268673
40.
WrightB. D.DouglasG. A. (1977). Conditional versus unconditional procedures for sample-free item analysis. Educational and Psychological Measurement, 37, 573-586. doi:10.1177/001316447703700301
41.
WyseA. E. (2013). DIF cancellation in the Rasch model. Journal of Applied Measurement, 14, 118-128.
42.
YenW. M. (1987). A comparison of the efficiency and accuracy of BILOG and LOGIST. Psychometrika, 52, 275-291. doi:10.1007/BF02294241
43.
ZwindermanA. H.van den WollenbergA. L. (1990). Robustness of marginal maximum likelihood estimation in the Rasch model. Applied Psychological Measurement, 14, 1, 73-81. doi:10.1177/014662169001400107
