A testlet is a cluster of items that share a common passage, scenario, or other context. These items might measure something in common beyond the trait measured by the test as a whole; if so, the model for the item responses should allow for this testlet trait. But modeling testlet effects that are negligible makes the model unnecessarily complicated and risks capitalization on chance, increasing the error in parameter estimates. Checking each testlet to see if the items within the testlet share something beyond the primary trait could therefore be useful. This study included (a) comparison between a model with no testlets and a model with testlet g, (b) comparison between a model with all suspected testlets and a model with all suspected testlets except testlet g, and (c) a test of essential unidimensionality. Overall, Comparison b was most useful for detecting testlet effects. Model comparisons based on information criteria, specifically the sample-size adjusted Bayesian Information Criteria (SSA-BIC) and BIC, resulted in fewer false alarms than statistical significance tests. The test of essential unidimensionality had true hit rates and false alarm rates similar to the SSA-BIC when the testlet effect was zero for all testlets except the studied testlet. But the presence of additional testlet effects in the partitioning test led to higher false alarm rates for the test of essential unidimensionality.
AckermanT. A. (1987, April). The robustness of LOGIST and BILOG IRT estimation programs to violations of local independence. Paper presented at the annual meeting of the American Educational Research Association, Washington, DC.
2.
AkaikeH. (1987). Factor analysis and AIC. Psychometrika, 52, 317-332.
3.
AndrichD.KreinerS. (2010). Quantifying response dependence between two dichotomous items using the Rasch model. Applied Psychological Measurement, 34, 181-192.
4.
BergerM. P. F.KnolD. L. (1990). On the assessment of dimensionality in multidimensional item response theory models. Enschede, Netherlands: University of Twente.
5.
BishopN. S.OmarM. H. (2002, April). Comparing vertical scales derived from dichotomous and polytomous IRT models for a test composed of testlets. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
6.
BockR. D.GibbonsR. (2010). Factor analysis of categorical item responses. In NeringM. L.OstiniR. (Eds.), Handbook of polytomous item response theory models (pp. 155-184). New York, NY: Routledge.
BozdoganH. (1987). Model selection and Akaike’s information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52, 345-370.
9.
BradlowE. T.WainerH.WangX. (1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153-168.
10.
CaiL. (2010). A two-tier full-information item factor analysis model with applications. Psychometrika, 75, 581-612.
11.
ChenW.-H.ThissenD. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289.
12.
De ChamplainA.GessaroliM. E. (1998). Assessing the dimensionality of item response matrices with small sample sizes and short test lengths. Applied Measurement in Education, 11, 231-253.
13.
DeMarsC. E. (2003). Detecting multidimensionality due to curricular differences. Journal of Educational Measurement, 40, 29-51.
14.
DeMarsC. E. (2006). Application of the bi-factor multidimensional item response theory model to testlet-based tests. Journal of Educational Measurement, 43, 145-168.
GibbonsR. D.HedekerD. R. (1992). Full-information bi-factor analysis. Psychometrika, 57, 423-436.
17.
HaughtonD. M. A. (1988). On the choice of a model to fit data from an exponential family. Annals of Statistics, 16, 342-355.
18.
HayashiK.BentlerP. M.YuanK.-H. (2007). On the likelihood ratio test for the number of factors in exploratory factor analysis. Structural Equation Modeling, 14, 505-526.
19.
IpE. H. (2010a). Empirically indistinguishable multidimensional IRT and locally dependent unidimensional item response models. British Journal of Mathematical and Statistical Psychology, 63, 395-416.
20.
IpE. H. (2010b). Interpretation of the three-parameter testlet response model and information function. Applied Psychological Measurement, 34, 467-482.
21.
JangE. E.RoussosL. (2007). An investigation into the dimensionality of TOEFL using conditional covariance-based nonparametric approach. Journal of Educational Measurement, 44, 1-21.
22.
JanssenR.De BoeckP. (1999). Confirmatory analyses of componential test structure using multidimensional item response theory. Multivariate Behavioral Research, 34, 245-268.
23.
KangT.CohenA. S. (2007). IRT model selection methods for dichotomous items. Applied Psychological Measurement, 31, 331-358.
24.
KangT.CohenA. S.SungH.-J. (2009). Model selection indices for polytomous items. Applied Psychological Measurement, 33, 499-518.
25.
KimD.De AyalaR. J.FerdousA. A.NeringM. L. (2011). The comparative performance of conditional independence indices. Applied Psychological Measurement, 35, 447-471.
26.
LeeG.KolenM. J.FrisbieD. A.AnkenmannR. D. (2002). Comparison of dichotomous and polytomous item response models in equating scores from tests composed of testlets. Applied Psychological Measurement, 25, 357-372.
27.
LiF.CohenA. S.KimS.-H.ChoS.-J. (2009). Model selection methods for mixture dichotomous IRT models. Applied Psychological Measurement, 33, 353-373.
28.
LiY.BoltD. M.FuJ. (2005). A test characteristic curve linking method for the testlet model. Applied Psychological Measurement, 29, 340-356.
29.
LiY.BoltD. M.FuJ. (2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 30, 3-21.
30.
LiY.RuppA. A. (2011). Performance of the S-χ2 statistic for full-information bifactor models. Educational and Psychological Measurement, 71, 986-1005.
31.
MaraisI. D.AndrichD. (2008a). Effects of varying magnitude and patterns of local dependence in the unidimensional Rasch model. Journal of Applied Measurement, 9, 105-124.
32.
MaraisI. D.AndrichD. (2008b). Formalising dimension and response violations of local independence in the unidimensional Rasch model. Journal of Applied Measurement, 9, 200-215.
33.
McKinleyR. L. (1989). Confirmatory analysis of test structure using multidimensional item response theory (ETS-RR-89-31). Princeton, NJ: ETS.
34.
McKinleyR. L.WayW. D. (1992). The feasibility of modeling secondary TOEFL ability dimensions using multidimensional IRT models (ETS-RR-92–16). Princeton, NJ: ETS.
35.
NylundK. L.AsparouhovT.MuthénB. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling, 14, 535-569.
36.
RijmenF. (2010). Formal relations and an empirical comparison among the bi-factor, the testlet, and a second-order multidimensional IRT model. Journal of Educational Measurement, 47, 361-372.
37.
RosenbaumP. R. (1988). Item bundles. Psychometrika, 53, 349-359.
38.
SchwarzG. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.
39.
ScloveS. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333-343.
40.
SemmesR.DavisonM. L.CloseC. (2011). Modeling individual differences in numerical reasoning speed as a random effect of response time limits. Applied Psychological Measurement, 35, 433-446.
41.
SireciS. G.ThissenD.WainerH. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28, 237-247.
42.
StoutW. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.
43.
StoutW. (2005). Dimtest (Version 2.0) [Computer software]. Champaign, IL: William Stout Institute for Measurement.
44.
StoutW.FroelichA.GaoF. (2001). Using resampling methods to produce an improved DIMTEST procedure. In BoomsmaA.van DuijnM. A. J.SnijdersT. A. B. (Eds.), Essays on item response theory (pp. 357-375). New York, NY: Springer-Verlag.
45.
TateT. (2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203.
46.
WainerH.BradlowE. T.DuZ. (2000).Testlet response theory: An analog for the 3PL model useful in testlet-based adaptive testing. In van der LindenW. J.GlasC. A. W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 245-269). Dordrecht, Netherlands: Kluwer.
47.
WainerH.BradlowE. T.WangX. (2007). Testlet response theory and its applications. New York, NY: Cambridge University Press.
48.
WainerH.WangC. (2000).Using a new statistical model for testlets to score TOEFL. Journal of Educational Measurement, 37, 203-220.
49.
WoodroofeM. (1982). On model selection and the arc sine laws. Annals of Statistics, 10, 1182-1194.
50.
YangC.-C. (2006). Evaluating latent class analysis models in qualitative phenotype identification. Computational Statistics & Data Analysis, 50, 1090-1104.
51.
YangC.-C.YangC.-C. (2007). Separating latent classes by information criteria. Journal of Classification, 24, 183-203.
52.
YenW. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145.
53.
YenW. M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30, 187-213.
54.
ZeniskyA.HambletonR. K. L.SireciS. G. (2002). Identification and evaluation of local item dependencies in the Medical College Admissions Test. Journal of Educational Measurement, 39, 291-309.
