Mathematical programming has been widely used by professionals in testing agencies as a tool to automatically construct equivalent test forms. This study introduces the linear programming capabilities (modeling language plus solvers) of SAS Operations Research as a platform to rigorously engineer tests on specifications in an automated manner. To that end, real items from a medical licensing test are used to demonstrate the simultaneous assembly of multiple parallel test forms under two separate linear programming scenarios: (a) constraint satisfaction (one problem) and (b) combinatorial optimization (three problems). In the four problems from the two scenarios, the forms are assembled subjected to various content and psychometric constraints. Assembled forms are next assessed using psychometric methods to ensure equivalence about all test specifications. Results from this study support SAS as a reliable and easy-to-implement platform for form assembly. Annotated codes are provided to promote further research and operational work in this area.
BreithauptK.HareD. R. (2007). Automated simultaneous assembly of multistage testlets for a high-stakes licensing examination. Educational and Psychological Measurement, 67, 5-20.
5.
ChenP. H.ChangH. H.WuH. (2012). Item selection for the development of parallel forms from an IRT-based seed test using a sampling and classification approach. Applied Psychological Measurement, 72, 933-953.
6.
ChoeE. M. (2017). Advancements in test security: Preventive test assembly methods and change-point detection of compromised items in adaptive testing (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign.
7.
ChoeE. M.DenbleykerJ. (2014). Quality psychometrics of Common Block Assembly: Summary report. Chicago, IL: National Board of Osteopathic Medical Examiners.
8.
ChuahS. C.DrasgowF.LuechtR. (2006). How big is big enough? Sample size requirements for CAST item parameter estimation. Applied Measurement in Education, 19, 241-255.
9.
CorK.AlvesC.GierlM. J. (2008). Conducting automated test assembly using the Premium Solver Platform Version 7.0 with Microsoft Excel and the large-scale LP/QP solver engine add-in. Applied Psychological Measurement, 32, 652-663.
10.
CorK.AlvesC.GierlM. J. (2009). Three applications of automated test assembly within a user-friendly modeling environment. Practical Assessment, Research and Evaluation, 14, 1-23.
11.
de AyalaR. J. (2009). The theory and practice of Item Response Theory. New York, NY: Guilford Press.
12.
DiaoQ.van der LindenW. J. (2011). Automated test assembly using lp_Solve version 5.5 in R. Applied Psychological Measurement, 35, 398-409.
13.
DrasgowF.LuechtR. M.BennettR. E. (2006). Technology and testing. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 471-515). Westport, CT: Praeger.
14.
FinkelmanM.KimW.RoussosL. A. (2009). Automated test assembly for cognitive diagnosis models using a genetic algorithm. Journal of Educational Measurement, 46, 273-292.
15.
LinacreJ. M. (2018). Winsteps® (Version 4.2.0) [Computer software]. Beaverton, Oregon: Winsteps.com. Available from https://www.winsteps.com/
16.
LuechtR. M. (1998). Computer-assisted test assembly using optimization heuristics. Applied Psychological Measurement, 22, 224-236.
17.
LuechtR. M.BrumfieldT.BreithauptK. (2006). A testlet assembly design for adaptive multistage tests. Applied Measurement in Education, 19, 189-202.
18.
LuechtR. M.ChamplainA. D.NungesterR. J. (1998). Maintaining content validity in computerized adaptive testing. Advances in Health Sciences Education, 3, 29-41.
19.
LuechtR. M.HirschT. M. (1992). Item selection using an average growth approximation of target information functions. Applied Psychological Measurement, 16, 41-51.
R Core Team. (2018). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/
27.
SAS Institute. (2017a). SAS/OR 14.3 user’s guide: Mathematical programming. Cary, NC: Author.
28.
SAS Institute. (2017b). System requirements for SAS 9.4 Foundation for Microsoft Windows for × 64. Cary, NC: Author.
29.
SAS Institute. (2018a). SAS 9.4 language reference: Concepts (6th ed.). Cary, NC: Author.
StockingM. L.SwansonL.PearlmanM. (1993). The application of an automated item selection method to real data. Applied Psychological Measurement, 17, 167-176.
33.
SunK.-T.ChenY.-J.TsaiS.-Y.ChengC.-F. (2008). Creating IRT-based parallel test forms using the genetic algorithm method. Applied Measurement in Education, 21, 141-161.
34.
TheunissenT. J. J. M. (1985). Binary programming and test design. Psychometrika, 50, 411-420.
35.
TheunissenT. J. J. M. (1986). Some applications of optimization algorithms in test design and adaptive testing. Applied Psychological Measurement, 10, 381-389.
36.
van der LindenW. J. (2005). Linear models of optimal test design. New York, NY: Springer.
37.
van der LindenW. J.AdemaJ. J. (1998). Simultaneous assembly of multiple test forms. Journal of Educational Measurement, 35, 185-198.
38.
VeldkampB. P. (1999). Multiple objective test assembly problems. Journal of Educational Measurement, 36, 253-266.
39.
VerschoorA. J. (2007). Genetic algorithms for automated test assembly (Unpublished doctoral thesis). University of Twente, Enschede, The Netherlands.
40.
WooA.GorhamJ. L. (2010). Understanding the impact of enemy items on test validity and measurement precision. CLEAR Exam Review, 21(1), 15-17.
41.
ZhengY.ChangH.-H. (2014). Multistage testing, on-the-fly multistage testing, and beyond. In ChengY.ChangH.-H. (Eds.), Advancing methodologies to support both summative and formative assessments (pp. 21-39). Charlotte, NC: Information Age.
ZhengY.WangC.CulbertsonM. J.ChangH.-H. (2014). Overview of test assembly methods in multistage testing. In YanD.von DavierA. A.LewisC. (Eds.), Computerized multistage testing: Theory and applications (pp. 87-99). New York, NY: CRC Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
