Interest in end-of-year accountability exams has increased dramatically since the passing of the No Child Left Behind Act in 2001. With this increased interest comes a desire to use student data collected throughout the year to estimate student proficiency and predict how well they will perform on end-of-year exams. This article uses student performance on the Assistment System, an online mathematics tutor, to show that replacing percentage correct with an Item Response Theory estimate of student proficiency leads to better fitting prediction models. In addition, it uses other tutor performance metrics to further increase prediction accuracy. Prediction error bounds are also calculated to attain an absolute measure to which the models can be compared.
Anozie, N.O., & Junker, B.W. (2006). Predicting end-of-year accountability assessment scores from monthly student records in an online tutoring system. In Proceedings of the American Association for Artificial Intelligence Workshop on Educational Data Mining (AAAI-06), July 17, 2006 , Boston, MA (Technical Report WS-06-05, pp. 1-6). Menlo Park, CA: AAAI Press.
2.
Ayers, E., & Junker, B.W. (2006). Do skills combine additively to predict task difficulty in eighth grade mathematics? In Proceedings of the American Association for Artificial Intelligence Workshop on Educational Data Mining (AAAI-06), July 17, 2006, Boston, MA (Technical Report WS-06-05, pp. 14-20). Menlo Park, CA : AAAI Press.
3.
Bishop, J.H. (1998). The effect of curriculum-based external exit exam systems on student achievement. The Journal of Economic Education , 29(2), 171-182.
4.
Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
5.
Embretson, S.E. (1999). Generating items during testing: Psychometric issues and models. Psychometrika, 64, 407-433.
6.
Farooque, P., & Junker, B.W. (2005). Behavior of skills within MCAS and Assistment main problems. Final project poster, Department of Statistics, Carnegie Mellon University.
7.
Feng, M., Heffernan, N.T., & Koedinger, K.R. (2006). Predicting state test scores better with intelligent tutoring systems: Developing metrics to measure assistance required. In M. Ikeda, K. Ashley, & T.-W. Chan (Eds.), Proceedings of the 8th International Conference on Intelligent Tutoring Systems (pp. 31-40). Berlin: Springer-Verlag.
8.
Fischer, G.H. (1974). Einführung in die Theorie psychologischer Tests: Grundlagen und Anwendungen [Introduction to the theory of psychological tests: Foundations and applications]. Bern, Switzerland: Verlag Hans Huber.
9.
Fischer, G.H., & Molenaar, I.W. (1995). Rasch models: Foundations, recent developments, and applications. New York: Springer-Verlag.
10.
Gelman, A., Carlin, J., Stern, H., & Rubin, D.B. (2004). Bayesian data analysis (2nd ed.). New York: Chapman & Hall.
11.
Gelman, A., Meng, X.-L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies [with discussion]. Statistica Sinica , 6, 733-807.
12.
Haist, S.A., Witzke, D.B., Quinlivan, S., Murphy-Spencer, A., & Wilson, J.F. (2003). Clinical skills as demonstrated by a comprehensive clinical performance examination: Who performs better-men or women? Advances in Health Sciences Education, 8, 189-199.
13.
Heffernan, N.T., Koedinger, K.R., & Junker, B.W. (2001). Using Web-based cognitive assessment systems for predicting student performance on state exams. Research proposal to the Institute of Educational Statistics, U.S. Department of Education. Department of Computer Science at Worcester Polytechnic Institute, Worcester County, MA.
14.
Janssen, R., & De Boeck, P. (2006). A random-effects version of the LLTM. Technical report, Department of Psychology, University of Leuven, Belgium.
15.
Junker, B.W. (in press). Using on-line tutoring records to predict end-of-year exam scores: Experience with the ASSISTments Project and MCAS 8th grade mathematics . In R. W. Lissitz (Ed.), Assessing and modeling cognitive development in school: Intellectual growth and standard setting. Maple Grove, MN: JAM Press.
16.
Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
17.
Maccini, P., & Hughes, C.A. (2000). Effects of a problem-solving strategy on the introductory algebra performance of secondary students with learning disabilities. Learning Disabilities Research and Practice, 15, 10-21.
18.
Massachusetts Department of Education. (2004 ). 2004 MCAS technical report. Retrieved December 2005 from http://www.doe.mass.edu/mcas/2005/news/04techrpt.pdf
19.
Nuthall, G., & Alton-Lee, A. (1995). Assessing classroom learning: How students use their knowledge and experience to answer classroom achievement test questions in science and social studies. American Educational Research Journal, 31(1), 185-223.
20.
Olson, L. (2005, November 30). State test programs mushroom as NCLB mandate kicks in. Education Week, pp. 10-14.
21.
Patz, R.J., & Junker, B.W. (1999). A straightforward approach to Markov Chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146-178.
22.
Protter, M.H., & Morrey, C.B., Jr. (1991). A first course in real analysis (2nd ed.). New York: Springer.
23.
R Development Core Team. (2004). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing (ISBN 3-900051-07-0). Available from http://www.R-project.org
24.
Rothman, S. (2001). 2001 MCAS reporting workshop: The second generation of MCAS results. Massachusetts Department of Education. Retrieved January 2007 from http://www.doe.mass.edu/mcas/2001/news/reproting_wkshp.pps
25.
Schofield, L. (2007, June 10). Using cognitive test scores in social science research. Thesis proposal, Department of Statistics, Carnegie Mellon University.
26.
Spiegelhalter, D.J., Thomas, A., & Best, N.G. (2003). WinBUGS: Bayesian inference using Gibbs sampling, manual version 1.4. Cambridge, UK: Medical Research Council Biostatistics Unit.
27.
van der Linden, W.J., & Hambleton, R.K. (1997). Handbook of modern item response theory. New York: Springer-Verlag .
