Bayesian networks (BN) provide a convenient and intuitive framework for specifying complex joint probability distributions and are thus well suited for modeling content domains of educational assessments at a diagnostic level. BN have been used extensively in the artificial intelligence community as student models for intelligent tutoring systems (ITS) but have received less attention among psychometricians. This critical review outlines the existing research on BN in educational assessment, providing an introduction to the ITS literature for the psychometric community, and points out several promising research paths. The online appendix lists 40 assessment systems that serve as empirical examples of the use of BN for educational assessment in a variety of domains.
AlmondR. G. (2010). “I can name that Bayesian network in two matrixes!” International Journal of Approximate Reasoning, 51, 167-178. doi:10.1016/j.ijar.2009.04.005
2.
AlmondR. G.DiBelloL. V.MoulderB.Zapata-RiveraJ.-D. (2007). Modeling diagnostic assessments with Bayesian networks. Journal of Educational Measurement, 44, 341-359. doi:10.1111/j.1745-3984.2007.00043.x
3.
AlmondR. G.MislevyR. J. (1999). Graphical models and computerized adaptive testing. Applied Psychological Measurement, 23, 223-237. doi:10.1177/01466219922031347
4.
AlmondR. G.MulderJ.HematL. A.YanD. (2009). Bayesian network models for local dependence among observable outcome variables. Journal of Educational and Behavioral Statistics, 34, 491-521. doi:10.3102/1076998609332751
5.
AlmondR. G.ShuteV. J.UnderwoodJ. S.Zapata-RiveraJ.-D. (2009). Bayesian networks: A teacher’s view. International Journal of Approximate Reasoning, 50, 450-460. doi:10.1016/j.ijar.2008.04.011
6.
AlmondR. G.YanD.HematL. (2008). Parameter recovery studies with a diagnostic Bayesian network model. Behaviormetrika, 35, 159-185. doi:10.2333/bhmk.35.159
7.
AndersonC. J.YuH.-T. (2007). Log-multiplicative association models as item response models. Psychometrika, 72, 5-23. doi:10.1007/s11336-005-1419-2
8.
BorsboomD.MellenberghG. J.van HeerdenJ. (2003). The theoretical status of latent variables. Psychological Review, 110, 203-219. doi:10.1037/0033-295X.110.2.203
9.
ButzC. J.HuaS.MaguireR. B. (2006). A web-based Bayesian intelligent tutoring system for computer programming. Web Intelligence and Agent Systems, 4(1), 77-97.
10.
CaiL. (2010). Metropolis-Hastings Robbins-Monro Algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35, 307-335. doi:10.3102/1076998609353115
11.
CarmonaC.MillánE.Pérez-de-la CruzJ.TrellaM.ConejoR. (2005). Introducing prerequisite relations in a multi-layered Bayesian student model. In ArdissonoL.BrnaP.MitrovicA. (Eds.), User modeling 2005: 10th international conference (Vol. 3538, pp. 347-356). New York, NY: Springer.
12.
ChangH.-H.YingZ. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229. doi:10.1177/014662169602000303
13.
ChungG. K. W. K.NiemiD.BewleyW. L. (2003, April). Assessment applications of ontologies. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
14.
CollinsJ.GreerJ.HuangS. (1996). Adaptive assessment using granularity hierarchies and Bayesian nets. In FrassonC.GauthierG.LesgoldA. (Eds.), Intelligent tutoring systems: Third international conference (pp. 569-577). New York, NY: Springer.
15.
ConatiC. (2010). Bayesian student modeling. In NkambouR.BourdeauJ.MizoguchiR. (Eds.), Advances in intelligent tutoring systems (pp. 281-299). New York, NY: Springer. doi:10.1007/978-3-642-14363-2_14
16.
ConatiC.GertnerA.VanLehnK. (2002). Using Bayesian networks to manage uncertainty in student modeling. User Modeling and User-Adapted Interaction, 12, 371-417. doi:10.1023/A:1021258506583
17.
ConatiC.GertnerA. S.VanLehnK.DruzdzelM. J. (1997). On-line student modeling for coached problem solving using Bayesian networks. In JamesonA.ParisC.TassoC. (Eds.), User modeling: Proceedings of the sixth international conference (pp. 231-242). New York, NY: Springer.
18.
ConatiC.ZhaoX. (2004). Building and evaluating an intelligent pedagogical agent to improve the effectiveness of an educational game. In NunesN. J.RichC. (Eds.), Proceedings of the 9th international conference on intelligent user interfaces (pp. 6-13). Association for Computing Machinery. doi:10.1145/964445.964446
19.
CorbettA. T.AndersonJ. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278. doi:10.1007/BF01099821
20.
de la TorreJ.DouglasJ. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353. doi:10.1007/BF02295640
21.
DesmaraisM. C.BakerR. S. J. d. (2012). A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction, 22, 9-38. doi:10.1007/s11257-011-9106-8
22.
EggenT. J. H. M. (1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement, 23, 249-261. doi:10.1177/01466219922031365
23.
EpsteinE. S. (1969). A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology, 8, 985-987. doi:10.1175/1520-0450(1969)008<0985:ASSFPF>2.0.CO;2
24.
GertnerA. S.ConatiC.VanLehnK. (1998). Procedural help in Andes: Generating hints using a Bayesian network student model. In MostowJ.RichC. (Eds.), Proceedings of the fifteenth national conference on artificial intelligence (pp. 106-111). American Association for Artificial Intelligence.
25.
GoguadzeG.SosnovskyS.IsotaniS.McLarenB. M. (2011). Towards a Bayesian student model for detecting decimal misconceptions. InYuF.-Y.HirashimaT.SupnithiT.BiswasG. (Eds.), Proceedings of the 19th international conference on computers in education. Asia-Pacific Society for Computers in Education. Retrieved from http://www.nectec.or.th/icce2011/program/proceedings/pdf/C1_F5_221F.pdf
26.
GuzmánE.ConejoR.Pérez-de-la CruzJ.-L. (2007). Adaptive testing for hierarchical student models. User Modeling and User-Adapted Interaction, 17, 119-157. doi:10.1007/s11257-006-9018-1
27.
JamesonA. (1995). Numerical uncertainty management in user and student modeling: An overview of systems and issues. User Modeling and User-Adapted Interaction, 5, 193-251.
28.
KoenigA. D.LeeJ. J.IseliM. R.WainessR. (2010). A conceptual framework for assessing performance in games and simulations (Tech. Rep. No.771). National Center for Research on Evaluation, Standards, and Student Testing. Retrieved from http://www.cse.ucla.edu/products/reports/R771.pdf
29.
LeightonJ. P.GierlM. J.HunkaS. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41, 205-237. doi:10.1111/j.1745-3984.2004.tb01163.x
30.
LevyR.CrawfordA. V. (2009). Bayesian network modeling for student-and domain-level inferences. Unpublished manuscript.
31.
LevyR.MislevyR. J. (2004). Specifying and refining a measurement model for a computer-based interactive assessment. International Journal of Testing, 4, 333-369. doi:10.1207/s15327574ijt04043
32.
LevyR.SvetinaD. (2011). A generalized dimensionality discrepancy measure for dimensionality assessment in multidimensional item response theory. The British Journal of Mathematical and Statistical Psychology, 64, 208-232. doi:10.1348/000711010X500483
33.
LiuC.-L. (2005). Some theoretical properties of mutual information for student assessments in intelligent tutoring systems. In HacidM.-S.MurrayN. V.RasZ. W.TsumotoS. (Eds.), Foundations of intelligent systems: 15th international symposium (pp. 524-534). New York, NY: Springer.
34.
LiuC.-L. (2009). Selecting Baysian-network models based on simulated expectation. Behaviormetrika, 36, 1-25. doi:10.2333/bhmk.36.1
35.
MadiganD.AlmondR. G. (1996). On test selection strategies for belief networks. In FisherD.LenzH.-J. (Eds.), Learning from data: Artificial intelligence and statistics V (pp. 89-98). New York, NY: Springer.
36.
MartinJ.VanLehnK. (1995). A Bayesian approach to cognitive assessment. In NicholsP. D.ChipmanS. F.BrennanR. L. (Eds.), Cognitively diagnostic assessment (pp. 141-165). Mahwah, NJ: Lawrence Erlbaum.
37.
McGlohenM.ChangH.-H. (2008). Combining computer adaptive testing technology with cognitively diagnostic assessment. Behavior Research Methods, 40, 808-821. doi:10.3758/BRM.40.3.808
38.
MillánE.Pérez-de-la CruzJ. L. (2002). A Bayesian diagnostic algorithm for student modeling and its evaluation. User Modeling and User-Adapted Interaction, 12, 281-330. doi:10.1023/A:1015027822614
39.
MillánE.Pérez-de-la CruzJ. L.SuárezE. (2000). Adaptive Bayesian networks for multilevel student modelling. In GauthierG.FrassonC.VanLehnK. (Eds.), Intelligent tutoring systems: 5th international conference (pp. 534-543). New York, NY: Springer.
40.
MislevyR. J. (1994). Probability-based inference in cognitive diagnosis (Tech. Rep. No. RR-94-93). Princeton, NJ: Educational Testing Service.
41.
MislevyR. J.AlmondR. G.DiBelloL.JenkinsF.SteinbergL.YanD.SenturkD. (2002). Modeling conditional probabilities in complex educational assessments (Tech. Rep. No. 580). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing.
42.
MislevyR. J.AlmondR. G.YanD.SteinbergL. S. (2000). Bayes nets in educational assessment: Where do the numbers come from? (Tech. Rep. No. 518). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing.
43.
MislevyR. J.GitomerD. H. (1996). The role of probability-based inference in an intelligent tutoring system. User Modeling and User-Adapted Interaction, 5, 253-282. doi:10.1007/BF01126112
44.
PardosZ.HeffernanN.AndersonB.HeffernanC. (2007). The effect of model granularity on student performance prediction using Bayesian networks. In ConatiC.McCoyK.PaliourasG. (Eds.), User modeling 2007: 11th international conference (pp. 435-439). New York, NY: Springer. doi:10.1007/978-3540-73078-1_60
45.
PattersonJ. A. (2011). Deconstructing a domain into its cognitive attributes: Test construction and data analysis (Unpublished doctoral dissertation). University of Illinois, Urbana–Champaign.
46.
PatzR. J.JunkerB. W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146-178. doi:10.3102/10769986024002146
47.
PearlJ. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Mateo, CA: Morgan Kaufmann.
48.
ReadT.BárcenaE.BarrosB.VerdejoF. (2002). Adaptive modelling of student diagnosis and material selection for on-line language learning. Journal of Intelligent and Fuzzy Systems, 12(3–4), 135-149.
49.
ReckaseM. D. (2009). Multidimensional item response theory. New York, NY: Springer.
50.
ReyeJ. (2004). Student modelling based on belief networks. International Journal of Artificial Intelligence in Education, 14(1), 63-96.
51.
RubinD. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applied statistician. Annals of Statistics, 12, 1151-1172. doi:10.1214/aos/1176346785
52.
RudnerL. M. (2002, April). An examination of decision-theory adaptive testing procedures. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
53.
RuppA. A.TemplinJ.HensonR. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.
54.
SchäferR.WeyrathT. (1997). Assessing temporally variable user properties with dynamic Bayesian networks. In JamesonA.ParisC.TassoC. (Eds.), User modeling: Proceedings of the sixth international conference (pp. 377-388). New York, NY: Springer.
55.
ShuteV. J. (2005). Stealth assessment in computer-based games to support learning. In TobiasS.FletcherJ. D. (Eds.), Computer games and instruction (pp. 503-524). Charlotte, NC: Information Age.
56.
SinharayS. (2006). Model diagnostics for Bayesian networks. Journal of Educational and Behavioral Statistics, 31, 1-33. doi:10.3102/10769986031001001
57.
SinharayS.AlmondR. G. (2007). Assessing fit of cognitive diagnostic models a case study. Educational and Psychological Measurement, 67, 239-257. doi:10.1177/0013164406292025
58.
SpiegelhalterD. J.BestN. G.CarlinB. P.van der LindeA. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64, 583-639. doi:10.1111/1467-9868.00353
59.
TatsuokaK. K. (1983). Rule-space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354. doi:10.1111/j.17453984.1983.tb00212.x
60.
TeppermanJ.LeeS.AlwanA. (2011). A generative student model for scoring word reading skills. IEEE Transactions on Audio, Speech, and Language Processing, 19, 348-360. doi:10.1109/TASL.2010.2047812
61.
TingC.-Y.Phon-AmnuaisukS. (2012). Properties of Bayesian student model for INQPRO. Applied Intelligence, 36, 391-406. doi:10.1007/s10489-010-0267-7
62.
van der GaagL. C.RenooijS.WittemanC. L. M.AlemanB. M. P.TaalB. G. (1999). How to elicit many probabilities. In LaskeyK. B.PradeH.CooperG. F. (Eds.), Uncertainty in artificial intelligence: Proceedings of the fifteenth conference (pp. 647-654). San Francisco, CA: Morgan Kaufmann.
63.
VomlelJ. (2004). Bayesian networks in educational testing. International Journal of Uncertainty, Fuzziness, and Knowledge-Based Systems, 12(Supp. 1), 83-100. doi:10.1142/S021848850400259X
64.
WangC.ChangH.-H. (2011). Item selection in multidimensional computerized adaptive testing—Gaining information from different angles. Psychometrika, 76, 363-384. doi:10.1007/s11336011-9215-7
65.
WangS.JiaoH. (2009). Construct equivalence across grades in a vertical scale for a K-12 large-scale reading assessment. Educational and Psychological Measurement, 69, 760-777. doi:10.1177/0013164409332230
WeissmanA. (2007). Mutual information item selection in adaptive classification testing. Educational and Psychological Measurement, 67, 41-58. doi:10.1177/0013164406288164
68.
WestP.RutsteinD. W.MislevyR. J.LiuJ.LevyR.DicerboK. E.BehrensJ. T. (2012). A Bayesian network approach to modeling learning progressions. In AlonzoA. C.GotwalsA. W. (Eds.), Learning progressions in science: Current challenges and future directions (pp. 257-292). New York, NY: Springer.
69.
WilliamsonD. M.MislevyR. J.AlmondR. G. (2001, April). Model criticism of Bayesian networks with latent variables. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.
70.
YudelsonM. V.MedvedevaO. P.CrowleyR. S. (2008). A multifactor approach to student model evaluation. User Modeling and User-Adapted Interaction, 18, 349-382. doi:10.1007/s11257007-9046-5
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
