Dynamic Bayesian networks (DBNs; Reye, 2004) are a promising tool for modeling student proficiency under rich measurement scenarios (Reichenberg, 2018). These scenarios often present assessment conditions far more complex than what is seen with more traditional assessments and require assessment arguments and psychometric models capable of integrating those complexities. Unfortunately, DBNs remain understudied and their psychometric properties relatively unknown. The current work aimed at exploring the properties of DBNs under a variety of realistic psychometric conditions. A Monte Carlo simulation study was conducted in order to evaluate parameter recovery for DBNs using maximum likelihood estimation. Manipulated factors included sample size, measurement quality, test length, the number of measurement occasions. Results suggested that measurement quality has the most prominent impact on estimation quality with more distinct performance categories yielding better estimation. From a practical perspective, parameter recovery appeared to be sufficient with samples as low as N = 400 as long as measurement quality was not poor and at least three items were present at each measurement occasion. Tests consisting of only a single item required exceptional measurement quality in order to adequately recover model parameters.
AlmondR.G. (2015). An IRT-based parameterization for conditional probability tables. Proceedings of the 12th UAI Bayesian Modeling Applications Workshop (pp. 14-23). Retrieved fromhttp://ceur-ws.org/Vol-1565/bmaw2015_paper4.pdf.
AlmondR. G.MislevyR. J.SteinbergL.YanD.WilliamsonD. (2015). Bayesian networks in educational assessment. Springer.
4.
BaldwinE. (2015). A Monte Carlo simulation study examining statistical power in latent transition analysis. UC Santa Barbara. Retrieved fromhttps://alexandria.ucsb.edu/downloads/q524jn927.
BandalosD. L.GagnéP. (2012). Simulation methods in structural equation modeling. In HoyleR. H. (Ed.), Handbook of structural equation modeling (pp. 92-108). The Guilford Press.
7.
BandalosD. L.LeiteW. (2013). The role of simulation in structural equation modeling. In Structural equation modeling: A second course (2nd ed., pp. 625-666). Information Age.
8.
ChoiY. (2012). Dynamic Bayesian inference networks and hidden Markov models for modeling learning progressions over multiple time points. Retrieved fromhttp://drum.lib.umd.edu/handle/1903/12739.
9.
ChungG.BakerE. L.VendlinskiT. P.BuschangR. E.DelacruzG. C.MichiuyeJ. K.BittickS. J. (2010). Testing instructional design variations in a prototype math game. In AtkinsonR.Chair (Eds,), Current perspectives from three national R&D centers focused on game-based learning: Issues in learning, instruction, assessment, and game design. Structured poster session at the annual meeting of the American Educational Research Association.
10.
CoetzeeD. (2014, June). Choosing sample size for knowledge tracing models. Presented at the Seventh International Conference on Educational Data Mining, London, UK.
11.
CollinsL. M.FidlerP. L.WugalterS. E.LongJ. D. (1993). Goodness-of-fit testing for latent class models. Multivariate Behavioral Research, 28(3), 375–389. https://doi.org/10.1207/s15327906mbr2803_4
12.
CollinsL. M.LanzaS. T. (2013) Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences (Vol. 718). John Wiley & Sons.
13.
CollinsL. M.TracyA. J. (1997). Estimation in complex latent transition models with extreme data sparseness. Kwantitatieve Methoden, 55(18), 57–71.
14.
CollinsL. M.WugalterS. E. (1992). Latent class models for stage-sequential dynamic latent variables. Multivariate Behavioral Research, 27(1), 131–157. https://doi.org/10.1207/s15327906mbr2701_8
15.
CollinsL. M.WugalterS. E.FidlerP. L. (1996). Some practical issues related to the estimation of latent class and latent transition parameters. In Categorical variables in developmental research (pp. 133-146). Academic Press.
16.
CorbettA. T.AndersonJ. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4), 253–278. https://doi.org/10.1007/BF01099821
17.
CorbettA. T.KoedingerK. R.AndersonJ. R. (1997). Intelligent tutoring systems. Handbook of Human-Computer Interaction, 5, 849–874.
FisherC. R.WalshM. M.BlahaL. M.GunzelmannG.VekslerB. (2016). Efficient parameter estimation of cognitive Models for real-time performance monitoring and adaptive interfaces. Retrieved fromhttp://acs.ist.psu.edu/iccm2016/proceedings/fisher2016iccm.pdf.
20.
GhahramaniZ. (2001). An introduction to hidden Markov models and Bayesian networks. International Journal of Pattern Recognition and Artificial Intelligence, 15(01), 9–42. https://doi.org/10.1142/9789812797605_0002
González-BrenesJ.HuangY.BrusilovskyP. (2015). General features in knowledge tracing: Applications to multiple subskills, temporal item response theory, and expert knowledge. Proceedings of the 7th International Conference on Educational Data Mining. Retrieved fromhttp://d-scholarship.pitt.edu/26017/1/84_EDM-2014-Full.pdf.
23.
HarveyA. C. (1990). Forecasting, structural time series models and the Kalman filter. Cambridge University Press.
24.
JamesG.WittenD.HastieT.TibshiraniR. (2013) An introduction to statistical learning (Vol. 112). New York: Springer.
25.
JohnsJ.WoolfB.ParkM. (2006). A dynamic mixture model to detect student motivation and proficiency. In Proceedings of the National Conference on Artificial Intelligence (21, p. 163). Melno Park, CA; Cambridge, MA; London: AAAI Press; MIT Press.
26.
KayaY.LeiteW. L. (2017). Assessing change in latent skills across time with longitudinal cognitive diagnosis modeling: An evaluation of model performance. Educational and Psychological Measurement, 77(3), 369–388. https://doi.org/10.1177/0013164416659314
27.
KlinglerS.KäserT.SolenthalerB.GrossM. (2015). On the performance characteristics of latent-factor and knowledge tracing models. International Educational Data Mining Society. Retrieved fromhttp://eric.ed.gov/?id=ED560586.
28.
LevyR. (2014). Dynamic Bayesian network modeling of game based diagnostic assessments. CRESST report 837. National Center for Research on Evaluation, Standards, and Student Testing (CRESST). Retrieved fromhttp://eric.ed.gov/?id=ED555714.
29.
LiF.CohenA.BottgeB.TemplinJ. (2016). A latent transition analysis model for assessing change in cognitive skills. Educational and Psychological Measurement, 76(2), 181–204. https://doi.org/10.1177/0013164415588946
30.
LunnD. J.ThomasA.BestN.SpiegelhalterD. (2000). WinBUGS–a Bayesian modeling framework: Concepts, structure, and extensibility. Statistics and Computing, 10(4), 325–337. https://doi.org/10.1023/A:1008929526011
31.
MadisonM. J.BradshawL. P. (2018). Assessing growth in a diagnostic classification model framework. Psychometrika, 83(4), 963–990. https://doi.org/10.1007/s11336-018-9638-5
32.
MurphyK. P. (2002). Dynamic Bayesian networks: Representation, inference and learning. Berkeley: University of California.
33.
MuthénL. K.MuthénB. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9(4), 599–620. https://doi.org/10.1207/S15328007SEM0904_8
34.
NeapolitanR. E. (2004) Learning Bayesian networks (Vol. 38). Saddle River, NJ: Pearson Prentice Hall Upper.
35.
NielsenT. D.JensenF. V. (2009). Bayesian networks and decision graphs. Springer Science & Business Media.
PanQ.QinL.KingstonN. (2020). Growth modeling in a diagnostic classification model (DCM) framework -A multivariate longitudinal diagnostic classification model. Frontiers in Psychology, 11, 1714. https://doi.org/10.3389/fpsyg.2020.01714
PardosZ. A.HeffernanN. T. (2010). Modeling individualization in a Bayesian networks implementation of knowledge tracingInternational Conference on User Modeling, Adaptation, and Personalization (pp. 255–266). Springer. Retrieved fromhttp://link.springer.com.ezproxy1.lib.asu.edu/10.1007%2F978-3-642-13470-8_24.
42.
PavlikP. I.CenH.KoedingerK. R. (2009, January). Performance factors analysis -- A new alternative to knowledge tracing. Presented at the 14th International Conference on Artificial Intelligence in Education, Brighton, UK.
43.
PearlJ. (1988). Probabilistic inference in intelligent systems. CA: Morgan Kaufmann San Mateo.
R Core Team (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.R-project.org/.
47.
ReichenbergR. (2018). Dynamic Bayesian networks in educational measurement: Reviewing and advancing the state of the field. Applied Measurement in Education, 31(4), 335–350. https://doi.org/10.1080/08957347.2018.1495217
48.
ReyeJ. (2004). Student modelling based on belief networks. International Journal of Artificial Intelligence in Education, 14(1), 63–96.
49.
RodríguezC. E.WalkerS. G. (2014). Label switching in Bayesian mixture models: Deterministic relabeling strategies. Journal of Computational and Graphical Statistics, 23(1), 25–45. https://doi.org/10.1080/10618600.2012.735624
50.
RuppA. A.TemplinJ.HensonR. A. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.
51.
ScutariM. (2010). Learning Bayesian networks with the bnlearn R package. Journal of Statistical Software, 35(3), 1–22. https://doi.org/10.18637/jss.v035.i03
52.
ShuteV. J. (2011). Stealth assessment in computer-based games to support learning. In Computer Games and Instruction (pp. 503-524). Information Age Publishing.
StephensM. (2000). Dealing with label switching in mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 62(4), 795–809. https://doi.org/10.1111/1467-9868.00265
55.
van de SandeB. (2013). Applying Three Models of Learning to Individual Student Log Data. In D’MelloS. K.CalvoR. A.OlneyA. (Eds), Proceedings of the 6th International Conference on Educational Data Mining (pp. 193-199). International Educational Data Mining Society. Retrieved fromhttp://www.educationaldatamining.org/conferences/index.php/EDM/2013/paper/download/1037/1003.
56.
WangS.YangY.CulpepperS.A.DouglasJ.A. (2018). Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden Markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57–87. https://doi.org/10.3102/1076998617719727
57.
YudelsonM. V.KoedingerK. R.GordonG. J. (2013). Individualized Bayesian knowledge tracing models. In International Conference on Artificial Intelligence in Education (pp. 171–180). Springer. Retrieved fromhttp://link.springer.com/10.1007/978-3-642-39112-5_18.
58.
Zapata-RiveraDBauerM. (2012) Exploring the Role of Games in Educational Assessment. In J. Clarke-Midura, M. Mayrath, and D. Robinson (Eds.), Technology-Based Assessments for 21st Century Skills: Theoretical and Practical Implications from Modern Research (pp. 147-169). Information Age Publishing.
59.
ZhanP.JiaoH.LiaoD.LiF. (2019). A longitudinal higher-order diagnostic classification model. Journal of Educational and Behavioral Statistics, 44(3), 251–281. https://doi.org/10.3102/1076998619827593
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
