Variable-Length Cognitive Diagnostic Computerized Adaptive Testing in Small-Scale Assessments

Abstract

This study examines innovative procedures for cognitive diagnostic computerized adaptive testing (CD-CAT) in small-scale assessments. Traditional CD-CAT methods, based on parametric cognitive diagnostic models (CDMs), often struggle with small calibration samples, leading to overfitting and overestimated reliability. Nonparametric alternatives, while more robust in small-scale settings, lack reliability information, limiting classification certainty and variable-length adaptive testing. To address these challenges, we propose four CD-CAT procedures using the parsimonious restricted deterministic input, noisy “and” gate (R-DINA) model, a parametric CDM tailored for small samples. Two of these procedures use a calibration sample (R-GDI and R-NPS), while the other two are calibration-free methods (R-NPS_ML and R-NPS_BM). Through a simulation study, where calibration sample size, number of attributes, and item quality were manipulated, we compare these methods to the conventional CD-CAT based on the DINA model. Results indicate that R-GDI and R-NPS consistently outperform the conventional CD-CAT in terms of more accurate posterior probability recovery, classification accuracy, and balanced item usage, although they administer a larger number of items. The calibration-free methods also perform satisfactorily but exhibit reliability overestimation with low-quality items. Overall, the proposed procedures offer practical solutions for formative assessments in educational contexts characterized by small sample sizes and time constraints. We provide recommendations for the use and scalability of these methods in real educational settings.