The Optimal Design of Bifactor Multidimensional Computerized Adaptive Testing with Mixed-format Items

Abstract

Multidimensional computerized adaptive testing (MCAT) using mixed-format items holds great potential for the next-generation assessments. Two critical factors in the mixed-format test design (i.e., the order and proportion of polytomous items) and item selection were addressed in the context of mixed-format bifactor MCAT. For item selection, this article presents the derivation of the Fisher information matrix of the bifactor graded response model and the application of the bifactor dimension reduction method to simplify the computation of the mutual information (MI) item selection method. In a simulation study, different MCAT designs were compared with varying proportions of polytomous items (0.2–0.6, 1), different item-delivering formats (DPmix: delivering polytomous items at the final stage; RPmix: random delivering), three bifactor patterns (low, middle, and high), and two item selection methods (Bayesian D-optimality and MI). Simulation results suggested that a) the overall estimation precision increased with a higher bifactor pattern; b) the two item selection methods did not show substantial differences in estimation precision; and c) the RPmix format always led to more precise interim and final estimates than the DPmix format. The proportions of 0.3 and 0.4 were recommended for the RPmix and DPmix formats, respectively.

Keywords

multidimensional computerized adaptive testing bifactor model mixed-format design item selection bifactor dimension reduction

Get full access to this article

View all access options for this article.

References

Cai

Yang

Hansen

(2011). Generalized full-information item bifactor analysis. Psychological Methods, 16(3), 221–248. https://doi.org/10.11037/a0023350

Cham

Hughes

J. N.

West

S. G.

M. H.

(2015). Effect of retention in elementary grades on Grade 9 motivation for educational attainment. Journal of School Psychology, 53(1), 7–24. https://doi.org/10.1016/j.jsp.2014.10.001

Dombrowski

S. C.

Canivez

G. L.

Watkins

M. W.

Beaujean

A. A.

(2015). Exploratory bifactor analysis of the Wechsler intelligence scale for children—fifth edition with the 16 primary and secondary subtests. Intelligence, 53(8), 194–201. https://doi.org/10.1016/j.intell.2015.10.009

Ercikan

Schwarz

R. D.

Julian

M. W.

Burket

G. R.

Weber

M. M.

Link

(1998). Calibration and scoring of tests with multiple-choice and constructed-response item types. Journal of Educational Measurement, 35(2), 137–154. https://doi.org/10.1111/j.1745-3984.1998.tb00531.x

Furtner

M. R.

Rauthmann

J. F.

Sachse

(2015). Unique self-leadership: A bifactor model approach. Leadership, 11(1), 105–125. https://doi.org/10.1177/1742715013511484

Gibbons

R. D.

Rush

A. J.

Immekus

J. C.

(2009). On the psychometric validity of the domains of the PDSQ: An illustration of the bi-factor item response theory model. Journal of Psychiatric Research, 43(4), 401–410. https://doi.org/10.1016/j.jpsychires.2008.04.013

Grady

Dodd

B. G.

(2009). A comparison of exposure control procedures for mixed-format adaptive tests. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.

Holzinger

K. J.

Swineford

(1937). The bi-factor method. Psychometrika, 2(1), 41–54. https://doi.org/10.1007/bf02287965

Kim

Chuang

Dodd

B. G.

Park

(2012). Panel Design Variations in the multistage test using the mixed-format tests. Educational and Psychological Measurement, 72(4), 574–588. https://doi.org/10.1177/0013164411428977

10.

Kinsey

T. L.

(2003). A comparison of IRT and Rasch procedures in a mixed-item format test (Accession No. 3106604) [Doctoral dissertation, University of North Texas]. ProQuest Dissertations Publishing.

11.

Lee

Y. H.

E. H.

Fuh

C. D.

(2008). A strategy for controlling item exposure in multidimensional computerized adaptive testing. Educational and Psychological Measurement, 68(2), 215–232. https://doi.org/10.1177/0013164407307007

12.

Lin

(2012). Item selection methods in multidimensional computerized adaptive testing adopting polytomously-scored items under multidimensional generalized partial credit model. Unpublished doctor dissertation. University of Illinois at Urbana-Champaign.

13.

Mao

Zhang

Xin

(2019). Application of dimension reduction to CAT item selection under the bifactor model. Applied Psychological Measurement, 43(6), 419–434. https://doi.org/10.1177/0146621618813086

14.

Mulder

van der Linden

W. J.

(2009). Multidimensional adaptive testing with optimal design criteria. Psychometrika, 74(2), 273–296. https://doi.org/10.1007/s11336-008-9097-5

15.

Nandakumar

H. H.

Stout

(1998). Assessing unidimensionality of polytomous data. Applied Psychological Measurement, 22(2), 99–115. https://doi.org/10.1177/01466216980222001

16.

Park

Kim

Chuang

Dodd

B. G.

(2014). Enhancing pool utilization in constructing the multistage test using mixed-format tests. Applied Psychological Measurement, 38(4), 268–280. https://doi.org/10.1177/0146621613515545

17.

Passos

V. L.

Berger

M. P.

Tan

F. E.

(2008). The D-optimality item selection criterion in the early stage of CAT: A study with the graded response model. Journal of Educational and Behavioral Statistics, 33(1), 88–110. https://doi.org/10.3102/1076998607302631

18.

Reckase

M. D.

(2017). A tale of two models: Sources of confusion in achievement testing (ETS research report no. RR-17-44). Education Testing Service. https://sharepoint.etslan.org/rd/rrpts/RR/RR-17-44.pdf#search=reckase

19.

Rosa

Swygert

K. A.

Nelson

Thissen

(2001). Item response theory applied to combinations of multiple-choice and constructed-response items-scale scores for patterns of summed scores. In Thissen

Wainer

(Eds.), Test scoring (pp. 253–292). Erlbaum.

20.

Samejima

(1976). Graded response model of the latent trait theory and tailored testing. In Clark

C. K.

(Ed.), Proceedings of the first conference on computerized adaptive testing (pp. 5–17). U.S. Government Printing Office.

21.

Seo

D. G.

Weiss

D. J.

(2015). Best design for multidimensional computerized adaptive testing with the bifactor model. Educational and Psychological Measurement, 75(6), 954–978. https://doi.org/10.11177/0013164415575147

22.

van Rijin

P. W.

Eggen

T. J. H. M.

Hemker

B. T.

Sanders

P. F.

(2002). Evaluation of selection procedures for computerized adaptive testing with polytomous items. Applied Psychological Measurement, 26(4), 393–411. https://doi.org/10.1177/014662102237796

23.

Veldkamp

B. P.

van der Linden

W. J.

(2002). Multidimensional adaptive testing with constraints on test content. Psychometrika, 67(4), 575–588. https://doi.org/10.1007/bf02295132

24.

Wang

Chang

H. H.

Boughton

K. A.

(2011). Kullback–Leibler information and its applications in multi-dimensional adaptive testing. Psychometrika, 76(1), 13–39. https://doi.org/10.1007/s11336-010-9186-0

25.

Watkins

M. W.

Beaujean

A. A.

(2014). Bifactor structure of the Wechsler preschool and primary scale of intelligence –fourth edition. School Psychology Quarterly, 29(1), 52–63. https://doi.org/10.1037/spq0000038

26.

Weiss

D. J.

Gibbons

R. D.

(2007). Computerized adaptive testing with the bifactor model. Paper presented at the New CAT Models session at the 2007 GMAC Conference on Computerized Adaptive Testing. http://publicdocs.iacat.org/cat2010/cat07weiss&gibbons.pdf

27.

Wiesner

Schanding

(2013). Exploratory structural equation modeling, bifactor models, and standard confirmatory factor analysis models: Application to the BASC-2 behavioral and emotional screening system teacher form. Journal of School Psychology, 51(6), 751–763. https://doi.org/10.1016/j.jsp.2013.09.001

28.

Yao

(2014). Multidimensional item response theory for score reporting. In Cheng

Chang

H.-H.

(Eds.), Advances in modern international testing: Transition from summative to formative assessment. Information Age.

29.

Yao

Schwarz

R. D.

(2006). A multidimensional partial credit model with associated item and test statistics: An application to mixed-format tests. Applied Psychological Measurement, 37(6), 3–23. https://doi.org/10.1177/0146621605284537

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.20 MB