Computerized adaptive testing (CAT) based on item response theory (IRT) is viewed from the perspective of graphical modeling (GM). GM provides methods for making inferences about multifaceted skills and knowledge, and for extracting data from complex performances. However, simply incorporating variables for all sources of variation is rarely successful. Thus, researchers must closely analyze the substance and structure of the problem to create more effective models. Researchers regularly employ sophisticated strategies to handle many sources of variability outside the IRT model. Relevant variables can play many roles without appearing in the operational IRT model per se, e.g., in validity studies, assembling tests, and constructing and modeling tasks. Some of these techniques are described from a GM perspective, as well as how to extend them to more complex assessment situations. Issues are illustrated in the context of language testing.
Get full access to this article
View all access options for this article.
References
1.
Adams, R. , & Wilson, M. R. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1–23.
2.
Albert, J. H. (1992). Bayesian estimation of normal ogive item response curves using Gibbs sampling. Journal of Educational Statistics, 17, 251–269.
3.
Almond, R. G. (1995). Graphical belief modeling. London: Chapman and Hall.
4.
Bachman, L. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
5.
Bachman, L. F. , & Palmer, A. S. (1996). Language testing in practice. Oxford: Oxford University Press.
6.
Bradlow, E. T. , Wainer, H., & Wang, X. (1998). A Bayesian random effects model for testlets (Research Report RR-98-3). Princeton NJ: Educational Testing Service.
7.
Bejar, I. I. (1990). A generative analysis of a threedimensional spatial task. Applied Psychological Measurement, 14, 237–245.
8.
Berger, M. P. F. , & Veerkamp, W. J. J. (1996). A review of selection methods for optimal test design. In G. Engelhard & M. Wilson (Eds.), Objective measurement: Theory into practice(Vol. 3). Norwood NJ: Ablex.
9.
Breese, J. S. , Goldman, R. P., & Wellman, M. P. (1994). Introduction to the special section on knowledge-based construction of probabilistic and decision models. IEEE Transactions on Systems, Man, and Cybernetics, 24, 1577–1579.
10.
Chalifour, C. , & Powers, D. E. (1989). The relationship of content characteristics of GRE analytical reasoning items to their difficulties and discriminations. Journal of Educational Measurement, 26, 120–132.
11.
Collis, J. M. , Tapsfield, P. G. C., Irvine, S. H., Dann, P. L., & Wright, D. (1995) The British Army Recruit Battery goes operational: From theory to practice in computer-based testing using itemgeneration techniques. International Journal of Selection and Assessment, 3, 96–104.
12.
Embretson, S. E. (1993). Psychometric models for learning and cognitive processes. In N. Frederiksen, R. J. Mislevy, & I. I. Bejar (Eds.), Test theory for a new generation of tests(pp. 125–150).Hillsdale NJ: Erlbaum.
13.
ETS, Inc. (1999). TOEFL 1999–2000: Information bulletin for TOEFL, TWE, and TSE. Princeton NJ: Author.
14.
Freedle, R. , & Kostin, I. (1993). The prediction of TOEFL reading item difficulty: Implications for construct validity. Language Testing, 10, 133–170.
15.
Hambleton, R. K. (1989). Principles and selected applications of item response theory. In R. L. Linn (Ed.), Educational measurement(3rd ed., pp. 147–200). New York: American Council on Education/Macmillan.
16.
Heckerman, D. , Horvitz, E., & Middleton, B. (1993). Anapproximate nonmyopic computation for value of information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15, 292–298.
17.
Jöreskog, K. G. , & Sörbom, D. (1979). Advances in factor analysis and structural equation models. Cambridge MA: Abt Books.
18.
Kelderman, H. , & Rijkes, C. P. M. (1994). Loglinear multidimensional IRT models for polytomously scored items. Psychometrika, 59, 149–176.
19.
Lauritzen, S. L. (1996). Graphical models. Oxford: Clarenden Press.
20.
Madigan, D. , & Almond, R. G. (1996). Test selection strategies for belief networks. In D. Fisher and H.-J. Lenz (Eds.), Learning from data: AI and statistics IV(pp. 89–98). New York: Springer.
21.
Mislevy, R. J. (1986). Bayes modal estimation in item response models. Psychometrika, 51, 177–196.
22.
Mislevy, R. J. (1988). Exploiting auxiliary information about items in the estimation of Rasch item difficulty parameters. Applied Psychological Measurement, 12, 281–296.
23.
Mislevy, R. J. , Sheehan, K. M., & Wingersky, M. S. (1993). How to equate tests with little or no data. Journal of Educational Measurement, 30, 55–78.
24.
Owen, R. J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association, 70, 351–356.
25.
Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Mateo CA: Kaufmann.
26.
Schum, D. A. (1994). The evidential foundations of probabilistic reasoning. New York: Wiley.
27.
Segall, D. (1996). Multidimensional adaptive testing. Psychometrika, 61, 331–354.
28.
Shachter, R. D. (1986). Evaluating influence diagrams. Operations Research, 34, 871–882.
29.
Sheehan, K. M. , & Mislevy, R. J. (1990). Integrating cognitive and psychometric models in a measure of document literacy. Journal of Educational Measurement, 27, 255–272.
30.
Spiegelhalter, D. J. , Dawid, A. P., Lauritzen, S. L., & Cowell, R. G. (1993). Bayesian analysis in expert systems. Statistical Science, 8, 219–283.
31.
Spiegelhalter, D. J. , Thomas, A., Best, N., & Gilks, W. (1996a). BUGS 0.5 examples(Vols. 1 & 2). Cambridge, England: MRC Biostatistics Unit, Institute of Public Health.
32.
Spiegelhalter, D. J. , Thomas, A., Best, N., & Gilks, W. (1996b). BUGS 0.5: Bayesian inference using Gibbs sampling(Version 2.0) [Software manual]. Cambridge England: MRC Biostatistics Unit, Institute of Public Health.
33.
Stocking, M. L. , & Swanson, L. (1993). A method for severely constrained item selection in adaptive testing. Applied Psychological Measurement, 17, 277–292.
34.
Tatsuoka, K. K. (1990). Toward an integration of item response theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. G. Shafto, (Eds.), Diagnostic monitoring of skill and knowledge acquisition(pp. 453–488). Hillsdale NJ: Erlbaum.
35.
van der Linden, W. J. (1997). Multidimensional adaptive testing with a minimum error-variance criterion(Research Report 97-03). Enschede, The Netherlands: University of Twente, Department of Educational Measurement and Data Analysis.
36.
van der Linden, W. J. (1998). Optimal test assembly[Special issue]. Applied Psychological Measurement, 22.
37.
Wainer, H. , Dorans, N. J., Flaugher, R., Green, B. F., Mislevy, R. J., Steinberg, L.,& Thissen, D. (1990). Computerized adaptive testing: A primer. Hillsdale NJ: Erlbaum.
38.
Widdowson, H.G. (1978). Teaching language as communication. Oxford: Oxford University Press.
