Comparability of paper and online versions of a statewide assessment was evaluated for the purposes of test score reporting, to appropriately adjust equated score conversion tables for students testing online as warranted. In particular, comparability of online and paper tests was designed with a propensity score matching method. The propensity score was used to match the online test sample with the paper–pencil test sample. The results of this study suggest that neither a separate assessment nor a conversion table would be required in a state’s assessment. The online test appears to provide a credible and comparable option to the paper and pencil modality.
American Educational Research Association (AERA), American Psychological Association (APA), and the National Council on Measurement in Education (NCME). (1999). Standards for educational and psychological testing. Washington, DC: AERA.
2.
BennettR. E.BraswellJ.OranjeA.SandeneB.KaplanB.YanF. (2008) Does it matter if I take my mathematics test on computer? A second empirical study of mode effects in NAEP. Journal of Technology, Learning, and Assessment6(9): Retrieved from http://www.jtla.org.
3.
Choi, S. W., & Tinkler, T. (2002). Evaluating comparability of paper and computer-based assessment in a K-12 setting. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA.
4.
CohenJ. (1988) Statistical power analysis for the behavioral sciences, 2nd ed. Hillsdale, NJ: Erlbaum.
5.
EmbretsonS. E.ReiseS. P. (2000) Item response theory for psychologists, Mahwah, NJ: Erlbaum.
HoD.ImaiK.KingG.StuartE. (2007) Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis15: 199–236. Retrieved from http://gking.harvard.edu/files/abs/matchp-abs.shtml.
8.
HollandP. W.ThayerD. T. (1988) Differential item performance and the Mantel-Haenszel procedure. In: WainerH.BraunH. I. (eds) Test validity, Hillsdale, NJ: Lawrence Erlbaum, pp. 129–145.
9.
Ito, K., & Sykes, R. C. (2004). Comparability of scores from norm-referenced paper-and-pencil and Web-based linear tests for grades 4-12. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.
10.
Karkee, T., Kim, D., & Fatica, K. (2010). Comparability study of online and paper and pencil tests using modified internally and externally matched criteria. Paper presented at the Annual meeting of the American Educational Research Association, Denver, CO.
11.
KimD.-H.HuynhH. (2007) Comparability of computer and paper-and-pencil versions of algebra and biology assessments. Journal of Technology, Learning, and Assessment6(4): ). Retrieved from http://www.jtla.org.
12.
LinacreJ. M. (2005) WINSTEPS version 3.59.1 [Computer Software], Chicago, IL: Author.
13.
LottridgeS. M.NicewanderW. A.MitzelH. C. (2011) A comparison of paper and online tests using a within-subjects design and propensity score matching study. Multivariate Behavioral Research46: 544–566.
14.
LoydB. H.HooverH. D. (1980) Vertical equating using the Rasch model. Journal of Educational Measurement17: 179–193.
15.
MasonB. J.PatryM.BernsteinD. J. (2001) An examination of the equivalence between non-adaptive computer-based and traditional testing. Journal of Educational Computing Research24(1): 29–39.
16.
MeadA. D.DrasgowF. (1993) Equivalence of computerized and paper cognitive ability tests: A meta-analysis. Psychological Bulletin114(3): 449–458.
17.
OlsonJ. F.FremerJ. (2013) TILSA test security guidebook, Washington, DC: CCSSO.
PoggioJ.GlasnappD. R.YangX.PoggioA. J. (2005) A comparative evaluations of score results from computerized and paper and pencil mathematics testing in a large scale state assessment program. Journal of Technology, Learning, and Assessment3(6): Retrieved from http://www.jtla.org.
21.
PommerichM. (2004) Developing computerized versions of paper-and-pencil tests: Mode effects for passage-based tests. Journal of Technology, Learning, and Assessment2(6): Retrieved from http://www.jtla.org.
22.
RajuN. S.OshimaT. C.WolachA. (2005) Differential functioning of items and tests (DFIT): Dichotomous and polytomous [Computer program], Chicago: Illinois Institute of Technology.
23.
R Development Core Team (2008). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria, ISBN 3-9000051-07-0, Retrieved from http://www.R-project.org.
24.
RaschG. (1960) Probabilistic models for some intelligence and attainment tests, Copenhagen: Danish Institute for Educational Research.
25.
RosenbaumP. R. (2010) Observational studies, 2nd ed. New York, NY: Springer-Verlag.
26.
RosenbaumP. R.RubinD. B. (1983) The central role of the propensity score in observational studies for causal effects. Biometrika70(1): 41–45.
27.
WangS.JiaoH.YoungM. J.BrooksT.OlsonJ. (2008) Comparability of computer-based and paper-and-pencil testing in k-12 reading assessments. Educational and Psychological Measurement62(1): 5–24.
28.
Way, W. D., Davis, L. L., & Fitzpatrick, S. (2006). Score comparability of online and paper administration of the Texas Assessment of Knowledge and Skills. Paper presented at the Annual Meeting of the National Council on Measurement in Education, San Francisco, CA.
29.
Way, W. D., Um, K., Lin, C., & McClarty, K. L. (2007, April). An evaluation of a matched samples method for assessing the comparability of online and paper test performance. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
30.
WellsC. S.SubkoviakM. J.SerlinR. C. (2002) The effect of item parameter drift on examinee ability estimates. Applied Psychological Measurement26: 77–87.
31.
WrightB. D.StoneM. H. (1979) Best test design, Chicago, IL: MESA Press.
32.
YuL.LivingstonS. A.LarkinK. C.BonettJ. (2004) Investigating differences in examinee performance between computer-based and handwritten essays (RR-04-18), Princeton, NJ: Educational Testing Service.
33.
ZiekyM. (1993) Practical questions in the use of DIF statistics in item development. In: HollandP. W.WainerH. (eds) Differential item functioning, Hillsdale, NJ: Lawrence Erlbaum, pp. 337–348.
