International large-scale assessments (ILSAs) transitioned from paper-based assessments to computer-based assessments (CBAs) facilitating the use of new item types and more effective data collection tools. This allows implementation of more complex test designs and to collect process and response time (RT) data. These new data types can be used to improve data quality and the accuracy of test scores obtained through latent regression (population) models. However, the move to a CBA also poses challenges for comparability and trend measurement, one of the major goals in ISLAs. We provide an overview of current methods used in ILSAs to examine and assure the comparability of data across different assessment modes and methods that improve the accuracy of test scores by making use of new data types provided by a CBA.
AgrestiA. (1990). Categorical data analysis. New York, NY: Wiley.
2.
AkaikeH. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723.
3.
BeckerN.SchmitzF.GöritzA. S.SpinathF. M. (2016). Sometimes more is better, and sometimes less is better: Task complexity moderates the response time accuracy correlation. Journal of Intelligence, 4, 11.
4.
BirnbaumA. (1968). On the estimation of mental ability(Series Report No. 15). Randolph Air Force Base, TX: USAF School of Aviation Medicine.
BozdoganH. (1987). Model selection and Akaike’s information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52, 345–370.
7.
BreimanL. (2001). Random forests. Machine Learning, 45, 5–32. doi:10.1023/A:1010933404324
8.
DietterichT. (2000). An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization. Machine Learning, 40, 139–157.
9.
DodonovaY. A.DodonovY. S. (2013). Faster on easy items, more accurate on difficult ones: Cognitive ability and performance on a task of varying difficulty. Intelligence, 41, 1–10.
10.
DongG.JianP. (2007). Sequence data mining. New York, NY: Springer.
11.
EfronB.HastieT.JohnstoneI.TibshiranR. (2004). Least angle regression. Annals of Statistics 2004, 32, 407–499.
12.
FormanG. (2003). An extensive empirical study of feature selection metrics for text classification. Journal of Machine Learning Research, 3, 1289–1305.
13.
FoxJ. P.MariantiS. (2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51, 540–553.
14.
FraleyC.RafteryA. E. (2002). Model-based clustering, discriminant analysis, and density estimation. Journal of the American Statistical Association, 97, 611–631.
15.
GlasC. A. W.PimentelJ. L.LamersS. M. A. (2015). Nonignorable data in IRT models: Polytomous responses and response propensity models with covariates. Psychological Test and Assessment Modeling, 57, 523–541.
16.
GoldhammerF.NaumannJ.GreiffS. (2015). More is not always better: The relation between item response and item response time in Raven’s matrices. Journal of Intelligence, 3, 21–40.
17.
GoldhammerF.NaumannJ.SelterA.TothK.RolkeH.KliemeE. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106, 608–626.
18.
GreiffS.WüstenbergS.AvvisatiF. (2015). Computer-generated log-file analyses as a window into students’ minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers & Education, 91, 92–105.
19.
GuoH.RiosJ. A.HabermanS.LiuO. L.WangJ.PaekI. (2016). A new procedure for detection of students’ rapid guessing responses using response time. Applied Measurement in Education, 29, 173–183.
20.
HanZ.HeQ.von DavierM. (2016). Predictive feature generation and selection from process data in simulation-based environment: An implementation of Random Forest. Paper presented at the 78th National Council on Measurement in Education (NCME) Conference, Washington, DC.
21.
HastieT.TaylorJ.TibshiraniR.WatlherG. (2007). Forward stagewise regression and the monotone LASSO. Electronic Journal of Statistics, 1, 1–29.
22.
HastieT.TibshiraniR.WainwrightM. (2015). Statistical learning with sparsity: The LASSO and generalizations. Boca Raton, FL: CRC Press.
23.
HeQ.GlasC. A. W.KosinskiM.StillwellD. J.VeldkampB. P. (2014). Predicting self-monitoring skills using textual posts on Facebook. Computers in Human Behavior, 33, 69–78.
24.
HeQ.VeldkampB. P.de VriesT. (2012). Screening for posttraumatic stress disorder using verbal features in self narratives: A text mining approach. Psychiatry Research, 198, 441–447.
25.
HeQ.VeldkampB. P.GlasC. A. W.de VriesT. (2017). Automated assessment of patients’ self-narratives for posttraumatic stress disorder screening using natural language processing and text mining. Assessment, 24, 157–172.
26.
HeQ.von DavierM. (2015). Identifying feature sequences from process data in problem-solving items with n-grams. In van der ArkA.BoltD.ChowS.DouglasJ.WangW. (Eds.), Quantitative psychology research: Proceedings of the 79th annual meeting of the psychometric society (pp. 173–190). New York, NY: Springer.
27.
HeQ.von DavierM. (2016). Analyzing process data from problem-solving items with n-grams: Insights from a computer-based large-scale assessment. In RosenY.FerraraS.MosharrafM. (Eds.), Handbook of research on technology tools for real-world skill development (pp. 749–776). Hershey, PA: Information Science Reference.
28.
KhorramdelL.von DavierM. (2016). Item response theory as a framework for test construction. In SchweizerK.DistefanoC. (Eds.), Principles and methods of test construction: Standards and recent advancements (pp. 52–80). Göttingen, Germany: Hogrefe.
29.
KudenkoD.HirshH. (1998). Feature generation for sequence categorization. In AAAI ‘98/IAAI ‘98: Proceedings of the 15th National/10th Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence (pp. 733–738). Menlo Park, CA: American Association for Artificial Intelligence.
30.
LeeY.-H.ChenH. (2011). A review of recent response-time analyses in educational testing. Psychological Test and Assessment Modeling, 53, 359–379.
31.
LeeY.-H.JiaY. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-Scale Assessments in Education, 2. doi:10.1186/s40536-014-0008-1
32.
LittleR. J. A.RubinD. B. (2002). Statistical analysis with missing data (2nd ed.). New York, NY: Wiley.
33.
ManningC. D.SchützeH. (1999). Foundations of statistical natural language processing. Cambridge, MA: MIT Press.
34.
MarisE. (1993). Additive and multiplicative models for gamma distributed variables, and their application as psychometric models for response times. Psychometrika, 58, 445–469.
35.
MazzeoJ.JohnsonE.BowkerD.FongY. F. (1992). The use of collateral information in proficiency estimation for the trial state assessment. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA, April 20–24.
MeyerJ. P. (2010). A mixture Rasch model with item response time components. Applied Psychological Measurement, 34, 521–538.
38.
MillsapR. E. (2010). Testing measurement invariance using item response theory in longitudinal data: An introduction. Child Development Perspectives, 4, 5–9. doi:10.1111/j.1750-8606.2009.00109
39.
MislevyR. J. (1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56, 177–196.
40.
MolenaarD.OberskiD.VermuntJ.De BoeckP. (2016). Hidden Markov IRT models for responses and response times. Multivariate Behavioral Research, 51, 606–626.
41.
MolenaarD.TuerlinckxF.van der MaasH. L. (2015). A bivariate generalized linear item response theory modeling framework to the analysis of responses and response times. Multivariate Behavioral Research, 50, 56–74.
42.
NigamK.McCallumA. K.ThurnS.MitchellT. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39, 103–134.
43.
OakesM.GaizauskasR.FowkesH.JonssonW. A. V.BeaulieuM. (2001). A method based on chi-square test for document classification. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 440–441). New York, NY: ACM.
PartchevI.De BoeckP. (2012). Can fast and slow intelligence be differentiated?Intelligence, 40, 23–32.
48.
RaschG. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Nielsen & Lydiche (Expanded edition, Chicago, University of Chicago Press, 1980).
49.
RoseN.von DavierM.NagengastB. (2016). Modeling omitted and not-reached items in IRT models. Psychometrika. doi:10.1007/s11336-016-9544-7
50.
RouderJ. N.SunD.SpeckmanP. L.LuJ.ZhouD. (2003). A hierarchical Bayesian statistical framework for response time distributions. Psychometrika, 68, 589–606.
51.
SchwarzG. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.
52.
ShinH. J.KhorramdelL.von DavierM.RobinF.YamamotoK. (2018). Incorporating response time into population modeling for large-scale assessments. Paper presented at the 11th Conference of the International Test Commission (ITC), Montreal, Canada.
53.
ShinH.von DavierM. (2017). Understanding time usage patterns and their associations with proficiencies in international large-scale assessments. Paper presented at the Timing Impact on Measurement in Education (TIME) conference held by National Board of Medical Examiners, Philadelphia, PA.
54.
SireciS. G.ThissenD.WainerH. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28, 237–247.
55.
ThomasN. (2002). The role of secondary covariates when estimating latent trait population distributions. Psychometrika, 67, 33–48.
56.
TibshiraniR. (1996). Regression shrinkage and selection via the LASSO. Journal of Royal Statistical Society. Series B, 58, 267–288.
57.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308.
58.
van der LindenW. J.BreithauptK.ChuahS. C.ZhangY. (2007). Detecting differential speededness in multistage testing. Journal of Educational Measurement, 44, 117–130.
59.
van der LindenW. J.GuoF. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73, 365–384.
60.
van der LindenW. J.Klein EntinkR. H.FoxJ.-P. (2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 34, 327–347.
61.
VerhagenA. J.FoxJ. P. (2012). Bayesian tests of measurement invariance. British Journal of Mathematical and Statistical Psychology, 66, 383–401.
62.
von DavierM. (2005). A general diagnostic model applied to language testing data. (ETS Research Report No. RR-05-16). Princeton, NJ: Educational Testing Service.
63.
von DavierM. (2008). The mixture general diagnostic model. In HancockG. R.SamuelsenK. M. (Eds.), Advances in latent variable mixture models (pp. 255–274). Charlotte, NC: Information Age.
64.
von DavierM. (2013). Imputing proficiency data under planned missingness in population models. In RutkowskiL.von DavierM.RutkowskiD. (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis. Boca Raton, FL: CRC Press (Chapman & Hall).
65.
von DavierM.GonzalezE.MislevyR. (2009). What are plausible values and why are they useful? In IERI Monograph Series: Issues and Methodologies in Large Scale Assessments, Vol. 2. Retrieved from IERI website:http://www.ierinstitute.org/IERI_Monograph_Volume_02_Chapter_01.pdf
66.
von DavierM.SinharayS. (2014). Analytics in international large-scale assessments: Item response theory and population models. In RutkowskiL.von DavierM.RutkowskiD. (Eds.). Handbook international large-scale assessment: Background, technical issues, and methods of data analysis. Boca Raton, FL: CRC Press (Chapman & Hall).
67.
von DavierM.SinharayS.OranjeA.BeatonA. (2006). Statistical procedures used in the National Assessment of Educational Progress (NAEP): Recent developments and future directions. In RaoC. R.SinharayS. (Eds.), Handbook of statistics (Vol. 26): Psychometrics (pp. 1039–1055). Amsterdam, the Netherlands: Elsevier.
68.
von DavierM.von DavierA. A. (2007). A unified approach to IRT scale linking and scale transformations. Methodology, 3, 115–124.
69.
von DavierM.XuX.CarstensenC. H. (2011). Measuring growth in a longitudinal large-scale assessment with a general latent variable model. Psychometrika, 76, 318–336.
70.
WainerH.BradlowE. T.WangX. (2007). Testlet response theory and its applications. New York, NY: Cambridge University Press.
71.
WeeksJ.von DavierM.YamamotoK. (2016). Using response time data to inform the coding of omitted responses. Psychological Test and Assessment Modeling, 58, 671–701.
72.
WeisbergS. (1980). Applied linear regression. New York, NY: Wiley.
73.
WetzelE.XuX.von DavierM. (2015). An alternative way to model population ability distributions in large-scale educational surveys. Educational and Psychological Measurement 2015, 75, 739–763.
74.
WiseS. L.DeMarsC. E. (2006). An application of item response time: The effort- moderated IRT model. Journal of Educational Measurement, 43, 19–38.
75.
WiseS. L.KongX. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.
76.
XingZ.PeiJ.KeoghE. (2010). A brief survey on sequence classification. SIGKDD Explorations Newsletter, 12, 40–48.
77.
YamamotoK.EversonH. (1997). Modeling the effects of test length and test time on parameter estimation using the hybrid model. In RostJ. (Ed.), Applications of latent trait and latent class models in the social sciences (pp. 89–98). Münster, Germany: Waxmann.
78.
YamamotoK.LennonM. L. (2018). Understanding and detecting data fabrication in large-scale assessments. Quality Assurance in Education, 26, 196–212.
