Exploring the Interplay of Topic Complexity,Emotional Engagement,and Cognitive Engagement in MOOC Discussions: Using Deep Learning and Topic Modeling

Abstract

Forum discussions in Massive Open Online Courses (MOOCs) play a crucial role in promoting learning engagement and academic achievement. In particular, discussion topics significantly influence learners’ emotional and cognitive engagement. However, the complex interrelationships among these factors remain underexplored. This study introduces an innovative two-step methodological approach to investigate the relationships between topic complexity, emotional engagement, cognitive engagement, and academic achievement in MOOC discussions. Using BERT for engagement detection and developing a Joint Emotion and Cognition Topic Model (JECTM) based on Bayesian networks, we analyzed 27,428 discussion posts from 2857 learners in a psychology MOOC. Our findings reveal three key insights: (1) The proposed two-step approach efficiently detects topics and analyzes their patterns of emotional and cognitive engagement. (2) As topic complexity increases, learners demonstrate higher-order cognitive engagement while experiencing reduced positive emotions along with increased confused and negative emotions. (3) In high-complexity topics, learners who maintain both positive emotions and higher-order cognitive engagement are more likely to achieve academic success than those who have negative emotions or lower-order cognitive engagement. These fine-grained analyses provide valuable insights for optimizing discussion design and interventions. This study also provides implications for the analysis of classroom dialogues and AI tutor-based conversations.

Keywords

MOOCs topic complexity emotional engagement cognitive engagement deep learning topic modeling forum discussion

Get full access to this article

View all access options for this article.

References

Atapattu

Thilakaratne

Vivian

Falkner

(2019). Detecting cognitive engagement using word embeddings within an online teacher professional development community. Computers & Education, 140(20), Article 103594. https://doi.org/10.1016/j.compedu.2019.05.020

Birjali

Kasri

Beni-Hssane

(2021). A comprehensive survey on sentiment analysis: Approaches, challenges and trends. Knowledge-Based Systems, 226(8), Article 107134. https://doi.org/10.1016/j.knosys.2021.107134

Blei

D. M.

A. Y.

Jordan

M. I.

(2003). Latent Dirichlet allocation. The Journal of Machine Learning Research, 3(2), 993–1022.

Bloom

B. S.

(1956). Taxonomy of educational objectives: The classification of educational goals, 1st ed. Longman Group.

Buhr

E. E.

Daniels

L. M.

Goegan

L. D.

(2019). Cognitive appraisals mediate relationships between two basic psychological needs and emotions in a massive open online course. Computers in Human Behavior, 96(9), 85–94. https://doi.org/10.1016/j.chb.2019.02.009

Chen

C.-M.

M.-C.

Chang

W.-C.

Chen

X.-X.

(2021). Developing a topic analysis instant feedback system to facilitate asynchronous online discussion effectiveness. Computers & Education, 163(4), Article 104095. https://doi.org/10.1016/j.compedu.2020.104095

Chen

C. K.

(2020). Sustaining online academic discussions: Identifying the characteristics of messages that receive responses. Computers & Education, 156(1), Article 103938. https://doi.org/10.1016/j.compedu.2020.103938

Chevrier

Muis

K. R.

Trevors

G. J.

Pekrun

Sinatra

G. M.

(2019). Exploring the antecedents and consequences of epistemic emotions. Learning and Instruction, 63(6), Article 101209. https://doi.org/10.1016/j.learninstruc.2019.05.006

Chi

M. T. H.

Wylie

(2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219–243. https://doi.org/10.1080/00461520.2014.965823

10.

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104

11.

Dalipi

Zdravkova

Ahlgren

(2021). Sentiment analysis of students’ feedback in MOOCs: A systematic literature review. Frontiers in Artificial Intelligence, 4(1), Article 728708. https://doi.org/10.3389/frai.2021.728708

12.

Demir

Ö.

Cinar

Keskin

(2023). Participation style and social anxiety as predictors of active participation in asynchronous discussion forums and academic achievement. Education and Information Technologies, 28(9), 1–22. https://doi.org/10.1007/s10639-022-11517-3

13.

Devlin

Chang

M.-W.

Lee

Toutanova

(2019). Bert: Pre-Training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis USA, June 2-7, pp. 4171–4186.

14.

Dhawal

Archisha

(2023). 202K courses, 662M enrollments: Breaking down udemy’s massive catalog. https://www.classcentral.com/report/udemy-by-the-numbers/

15.

Dillon

Bosch

Chetlur

Wanigasekara

Ambrose

G. A.

Sengupta

D’Mello

S. K.

(2016). Student emotion, Co-occurrence, and dropout in a MOOC context. International Educational Data Mining Society . https://eric.ed.gov/?id=ED592723

16.

Ding

(2019). Applying gamifications to asynchronous online discussions: A mixed methods study. Computers in Human Behavior, 91(9), 1–11. https://doi.org/10.1016/j.chb.2018.09.022

17.

D’Mello

Dieterle

Duckworth

(2017). Advanced, analytic, automated (AAA) measurement of engagement during learning. Educational Psychologist, 52(2), 104–123. https://doi.org/10.1080/00461520.2017.1281747

18.

D’Mello

Lehman

Pekrun

Graesser

(2014). Confusion can be beneficial for learning. Learning and Instruction, 29(3), 153–170. https://doi.org/10.1016/j.learninstruc.2012.05.003

19.

Doo

M. Y.

Kim

(2024). The relationship between learning engagement and learning outcomes in online learning in higher education: A meta-analysis study. Distance Education, 45(1), 60–82. https://doi.org/10.1080/01587919.2024.2303484

20.

Dubovi

(2022). Cognitive and emotional engagement while learning with VR: The perspective of multimodal methodology. Computers & Education, 183(4), Article 104495. https://doi.org/10.1016/j.compedu.2022.104495

21.

Ertmer

P. A.

Sadaf

Ertmer

D. J.

(2011). Student-content interactions in online courses: The role of question prompts in facilitating higher-level engagement with course content. Journal of Computing in Higher Education, 23(2–3), 157–186. https://doi.org/10.1007/s12528-011-9047-6

22.

Gašević

Joksimović

Eagan

B. R.

Shaffer

D. W.

(2018). SENS: Network analytics to combine social and cognitive perspectives of collaborative learning. Computers in Human Behavior, 92(3), 562–577. https://doi.org/10.1016/j.chb.2018.07.003

23.

Gašević

Kovanović

Joksimović

(2017). Piecing the learning analytics puzzle: A consolidated model of a field of research and practice. Learning: Research and Practice, 3(1), 63–78. https://doi.org/10.1080/23735082.2017.1286142

24.

Geng

Niu

Feng

Huang

(2020). Understanding the focal points and sentiment of learners in MOOC reviews: A machine learning and SC-LIWC-based approach. British Journal of Educational Technology, 51(5), 1785–1803. https://doi.org/10.1111/bjet.12999

25.

Graesser

A. C.

(2020). Emotions are the experiential glue of learning environments in the 21st century. Learning and Instruction, 70(9), Article 101212. https://doi.org/10.1016/j.learninstruc.2019.05.009

26.

Han

Z.-M.

Huang

C.-Q.

J.-H.

Tsai

C.-C.

(2021). Identifying patterns of epistemic emotions with respect to interactions in massive online open courses using deep learning and social network analysis. Computers in Human Behavior, 122(18), Article 106843. https://doi.org/10.1016/j.chb.2021.106843

27.

(2021). HanLP: Han language processing. [Python]. https://github.com/hankcs/HanLP

28.

Hou

Zhu

Zheng

Zhang

Huang

Zhong

Ker

C. L.

(2024). Prompt-based and fine-tuned GPT models for context-dependent and -independent deductive coding in social annotation. In Proceedings of the 14th Learning Analytics and Knowledge Conference, Tokyo Japan, March 18-22, pp. 518–528. https://doi.org/10.1145/3636555.3636910

29.

Hoyle

Goel

Hian-Cheong

Peskov

Boyd-Graber

Resnik

(2021). Is automated topic model evaluation broken? The incoherence of coherence. Advances in Neural Information Processing Systems, 34(26), 2018–2033. https://proceedings.neurips.cc/paper/2021/hash/0f83556a305d789b1d71815e8ea4f4b0-Abstract.html.

30.

Iqbal

Rakovic

Chen

Ferreira Mello

Fan

Fiorentino

Radi Aljohani

Gasevic

(2023). Towards automated analysis of rhetorical categories in students essay writings using Bloom’s taxonomy. In LAK23: 13th International Learning Analytics and Knowledge Conference, Austin USA, March 13-17, pp. 418–429. https://doi.org/10.1145/3576050.3576112

31.

Kapur

(2011). Temporality matters: Advancing a method for analyzing problem-solving processes in a computer-supported collaborative environment. International Journal of Computer-Supported Collaborative Learning, 6(1), 39–56. https://doi.org/10.1007/s11412-011-9109-9

32.

Kong

Liu

Chen

Liu

Tang

(2025). Exploratory study of an AI-supported discussion representational tool for online collaborative learning in a Chinese university. The Internet and Higher Education, 64(2), Article 100973. https://doi.org/10.1016/j.iheduc.2024.100973

33.

Krathwohl

D. R.

(2002). A revision of Bloom’s taxonomy: An overview. Theory Into Practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2

34.

Lau

J. H.

Newman

Baldwin

(2014). Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, Gothenburg Sweden, April 26-30, pp. 530–539. https://doi.org/10.3115/v1/E14-1056

35.

Lave

Wenger

(1991). Situated Learning: Legitimate Peripheral participation. Cambridge University Press.

36.

Lee

J.-E.

Recker

(2021). The effects of instructors’ use of online discussions strategies on student participation and performance in university online introductory mathematics courses. Computers & Education, 162(5), Article 104084. https://doi.org/10.1016/j.compedu.2020.104084

37.

Liu

Peng

Yang

(2022). Automated detection of emotional and cognitive engagement in MOOC discussions to predict learning achievement. Computers & Education, 181(9), Article 104461. https://doi.org/10.1016/j.compedu.2022.104461

38.

Liu

Sha

Zeng

Gasevic

Liu

(2024). Annotation Guidelines-based knowledge Augmentation: Towards enhancing large language models for educational text classification No. arXiv:2406.00954. arXiv. https://doi.org/10.48550/arXiv.2406.00954

39.

Liu

Zhang

Liu

(2020). Investigating the relationship between learners’ cognitive participation and learning outcome in asynchronous online discussion forums. In Proceedings of the 12th International Conference on Computer Supported Education, Prague Czech Republic, May 2-4, pp. 26–33.

40.

Liu

Tang

Ouyang

Long

Liu

(2024). Profiling students’ learning engagement in MOOC discussions to identify learning achievement: An automated configurational approach. Computers & Education, 219(9), Article 105109. https://doi.org/10.1016/j.compedu.2024.105109

41.

Liu

Yang

Rüdian

Liu

Zhao

Wang

(2019). Temporal emotion-aspect modeling for discovering what students are concerned about in online course forums. Interactive Learning Environments, 27(5–6), 598–627. https://doi.org/10.1080/10494820.2019.1610449

42.

Malinen

(2015). Understanding user participation in online communities: A systematic literature review of empirical studies. Computers in Human Behavior, 46(4), 228–238. https://doi.org/10.1016/j.chb.2015.01.004

43.

Merriënboer

J. J. G. V.

Sweller

(2005). Cognitive load theory and complex learning: Recent developments & future directions. Educational Psychology Review, 17(5), Article 147177.

44.

Mohammed

Omar

(2020). Question classification based on Bloom’s taxonomy cognitive domain using modified TF-IDF and word2vec. PLoS One, 15(3), Article e0230442. https://doi.org/10.1371/journal.pone.0230442

45.

Newman

Lau

J. H.

Grieser

Baldwin

(2010). Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Los Angeles USA, June 1–6, pp. 100–108. https://www.aclweb.org/anthology/N10-1012

46.

Nistor

Dascalu

Tarnai

Trausan-Matu

(2020). Predicting newcomer integration in online learning communities: Automated dialog assessment in blogger communities. Computers in Human Behavior, 105(12), Article 106202. https://doi.org/10.1016/j.chb.2019.106202

47.

Onan

Toçoğlu

(2020). Weighted word embeddings and clustering‐based identification of question topics in MOOC discussion forum posts. Computer Applications in Engineering Education, 29(4), 675–689. https://doi.org/10.1002/cae.22252

48.

O’Riordan

Millard

D. E.

Schulz

(2021). Is critical thinking happening? Testing content analysis schemes applied to MOOC discussion forums. Computer Applications in Engineering Education, 29(4), 690–709. https://doi.org/10.1002/cae.22314

49.

Orji

F. A.

Vassileva

(2023). Predicting the persuasiveness of influence strategies from student online learning behaviour using machine learning methods. Journal of Educational Computing Research, 61(7), 1410–1429. https://doi.org/10.1177/07356331231178873

50.

Ouyang

(2021). Using three social network analysis approaches to understand computer-supported collaborative learning. Journal of Educational Computing Research, 59(7), 1401–1424. https://doi.org/10.1177/0735633121996477

51.

Ouyang

Chen

(2021). Effect of three network visualizations on students’ social-cognitive engagement in online discussions. British Journal of Educational Technology, 52(6), 2242–2262. https://doi.org/10.1111/bjet.13126

52.

Pan

Zhou

Feng

(2022). Are students happier the more they learn? – Research on the influence of course progress on academic emotion in online learning. Interactive Learning Environments, 31(10), 1–21. https://doi.org/10.1080/10494820.2022.2052110

53.

Pekrun

Linnenbrink-Garcia

(2012). Academic emotions and student engagement. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 259–282). Springer US. https://doi.org/10.1007/978-1-4614-2018-7_12

54.

Peng

Han

Ouyang

Liu

(2020). Topic tracking model for analyzing student-generated posts in SPOC discussion forums. International Journal of Educational Technology in Higher Education, 17(1), 35. https://doi.org/10.1186/s41239-020-00211-4

55.

Peng

(2020). Investigating learners’ behaviors and discourse content in MOOC course reviews. Computers & Education, 143, Article 103673. https://doi.org/10.1016/j.compedu.2019.103673

56.

Qiu

Zhu

Jiang

(2013). It is not just what we say, but how we say them: LDA-based behavior-topic model. In Proceedings of the 2013 SIAM International Conference on Data Mining, Austin USA, May 2-4, pp. 794–802. https://doi.org/10.1137/1.9781611972832.88

57.

Ramirez-Arellano

Bory-Reyes

Hernández-Simón

L. M.

(2019). Emotions, motivation, cognitive–metacognitive strategies, and behavior as predictors of learning performance in blended learning. Journal of Educational Computing Research, 57(2), 491–512. https://doi.org/10.1177/0735633117753935

58.

Rivera

D. A.

Frenay

Paquot

(2024). The role of MOOC forum discussion tasks in learners’ cognitive engagement. Journal of Computer Assisted Learning, 40(5), 2103, 2120. https://doi.org/10.1111/jcal.13001

59.

Shaffer

D. W.

Collier

Ruis

A. R.

(2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics, 3(3), Article 3. https://doi.org/10.18608/jla.2016.33.3

60.

Shaikh

Daudpotta

S. M.

Imran

A. S.

(2021). Bloom’s learning outcomes’ automatic classification using LSTM and pretrained word embeddings. IEEE Access, 9(6), 117887–117909. https://doi.org/10.1109/ACCESS.2021.3106443

61.

Soffer

Cohen

(2019). Students’ engagement characteristics predict success and completion of online courses. Journal of Computer Assisted Learning, 35(3), 378–389. https://doi.org/10.1111/jcal.12340

62.

Sun

J. C.-Y.

Rueda

(2012). Situational interest, computer self-efficacy and self-regulation: Their impact on student engagement in distance education. British Journal of Educational Technology, 43(2), 191–204. https://doi.org/10.1111/j.1467-8535.2010.01157.x

63.

Tang

Xing

Pei

(2019). Time really matters: Understanding the temporal dimension of online learning using educational data mining. Journal of Educational Computing Research, 57(5), 1326–1347. https://doi.org/10.1177/0735633118784705

64.

Vygotsky

L. S.

(1978). Mind in society. Harvard University Press. https://book.douban.com/subject/2266607/

65.

Wang

Shan

Zheng

Guo

Chen

(2023). Can chatgpt detect student talk moves in classroom discourse? A preliminary comparison with bert. In Proceedings of the 16th International Conference on Educational Data Mining, Bengaluru India, July 11-14, pp. 515–519. https://doi.org/10.5281/zenodo.8115772

66.

Wang

Wen

Rosé

C. P.

(2016). Towards triggering higher-order thinking behaviors in MOOCs. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, Edinburgh Scotland, April 25-29, pp. 398–407. https://doi.org/10.1145/2883851.2883964

67.

Wang

Yang

Wen

Koedinger

Rosé

C. P.

(2015). Investigating how student’s cognitive behavior in MOOC discussion forums affect learning gains. International Educational Data Mining Society .

68.

Wei

Saab

Admiraal

(2021). Assessment of cognitive, behavioral, and affective learning outcomes in massive open online courses: A systematic literature review. Computers & Education, 163(7), Article 104097. https://doi.org/10.1016/j.compedu.2020.104097

69.

Wigfield

Eccles

J. S.

(2020). 35 years of research on students' subjective task values and motivation: A look back and a look forward. In Elliot

A. J.

(Ed.), Advances in motivation science (7, pp. 161–198). Elsevier. https://doi.org/10.1016/bs.adms.2019.05.002

70.

Wise

A. F.

Cui

(2018). Learning communities in the crowd: Characteristics of content related interactions and social relationships in MOOC discussion forums. Computers & Education, 122(10), 221–242. https://doi.org/10.1016/j.compedu.2018.03.021

71.

Wise

A. F.

Cui

Jin

Vytasek

(2017). Mining for gold: Identifying content-related MOOC discussion threads across domains through linguistic modeling. The Internet and Higher Education, 32(1), 11–28. https://doi.org/10.1016/j.iheduc.2016.08.001

72.

Wise

A. F.

Knight

Shum

S. B.

(2021). Collaborative learning analytics. In Cress

Rosé

Wise

A. F.

Oshima

(Eds.), International handbook of computer-supported collaborative learning (1st ed., pp. 425–443). Springer International Publishing. https://doi.org/10.1007/978-3-030-65291-3_23

73.

Xing

(2023). Mining large open online learning networks: Exploring community dynamics and communities of performance. Journal of Educational Computing Research, 61(2), 390–415. https://doi.org/10.1177/07356331221113613

74.

Yang

Tang

Hao

Rose

J. R.

(2022). Untangling chaos in discussion forums: A temporal analysis of topic-relevant forum posts in MOOCs. Computers & Education, 178, Article 104402. https://doi.org/10.1016/j.compedu.2021.104402

75.

Yao

Zhang

Wei

Jin

Zhang

Chen

(2017). Incorporating knowledge graph embeddings into topic modeling. In Thirty-First AAAI Conference on Artificial Intelligence, San Francisco USA, February 4-9. https://doi.org/10.1609/aaai.v31i1.10951

76.

Zimmerman

B. J.

Schunk

D. H.

(2011). Self-regulated learning and performance: An introduction and an overview. In Handbook of self-regulation of learning and performance. Routledge.

77.

Zou

Pan

Cai

Liu

(2021). Exploring the relationship between social presence and learners’ prestige in MOOC discussion forums using automated content analysis and social network analysis. Computers in Human Behavior, 115(11), 106582. https://doi.org/10.ghmph7