Sage Journals: Discover world-class research

Abstract

While the majority of scholarly investigations concerning pedagogical behavior predominantly underscore verbal communication, the significance of nonverbal behavior remains inadequately examined. This research endeavors to propose a theoretical framework pertaining to educators’ nonverbal communication and introduces two artificial intelligence-based recognition models: GestureNet, which focuses on hand gestures, and JointBDOE, which pertains to body orientation. Utilizing video data derived from 32 high-caliber classroom sessions, the research meticulously analyzes gesture patterns through the application of descriptive statistics, sequence analysis, and variance testing. The results indicate a notable prevalence of descriptive and pointing gestures alongside a predominance of frontal orientations. Marked disparities in gesture utilization were identified across various educational stages and subjects, with primary school educators exhibiting more frequent gesture sequences. These findings contribute to the enhancement of automated behavior analysis and provide practical implications for professional development and the design of classroom interactions.

Keywords

nonverbal communication in teaching gesture recognition automated behavior analysis deep learning in education teacher multimodal behavior classroom analytics

Get full access to this article

View all access options for this article.

References

Aldrup

Carstensen

Klusmann

(2024). The role of teachers’ emotion regulation in teaching effectiveness: A systematic review integrating four lines of research. Educational Psychologist, 59(2), 89–110. https://doi.org/10.1080/00461520.2023.2282446

Alibali

M. W.

Nathan

M. J.

(2012). Embodiment in mathematics teaching and learning: Evidence from learners’ and teachers’ gestures. The Journal of the Learning Sciences, 21(2), 247–286. https://doi.org/10.1080/10508406.2011.611446

Bakeman

Gottman

J. M.

(1997). Observing interaction: An introduction to sequential analysis (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511527685

Baltieri

Vezzani

Cucchiara

(2012). People orientation recognition by mixtures of wrapped distributions on random trees. In Fitzgibbon

Lazebnik

Perona

Sato

Schmid

(Eds.), Computer vision – eccv 2012 (7576, pp. 270–283). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-33715-4_20

Beege

Schneider

Nebel

Rey

G. D.

(2017). Look into my eyes! Exploring the effect of addressing in educational videos. Learning and Instruction, 49, 113–120. https://doi.org/10.1016/j.learninstruc.2017.01.004

Blömeke

Kaiser

(2017). Understanding the development of teachers’ professional competencies as personally, situationally, and societally determined. In Clandinin

D. J.

Husu

(Eds.), International Handbook of Research on Teacher Education (pp. 783–802). Sage.

Bower

M. G.

Moloney

Cavanagh

Sweller

(2013). Assessing preservice teachers’ presentation capabilities: Contrasting the modes of communication with the constructed impression. Australian Journal of Teacher Education, 38(8), 111–131. https://doi.org/10.14221/ajte.2013v38n8.1

Brok

P. D.

Bergen

Brekelmans

(2006). Convergence and divergence between students’ and teachers’ perceptions of instructional behaviour in Dutch secondary education. In Fisher

Khine

M. S.

(Eds.), Contemporary approaches to research on learning environments (pp. 125–160). World Scientific. https://doi.org/10.1142/9789812774651_0006

Burgermeister

Curio

(2022, June). PedRecNet: Multi-task deep neural network for full 3D human pose and orientation estimation. In 2022 IEEE Intelligent Vehicles Symposium (IV) (pp. 441–448). IEEE. https://doi.org/10.1109/IV51971.2022.9827202

10.

Carney

D. R.

Cuddy

A. J. C.

Yap

A. J.

(2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21(10), 1363–1368. https://doi.org/10.1177/0956797610383437

11.

Chen

Huang

Liu

Wang

Wen

Wang

(2024). ST-TGR: Spatio-Temporal representation learning for skeleton-based teaching gesture recognition. Sensors, 24(8), 2589. https://doi.org/10.3390/s24082589

12.

Choi

Lee

B.-J.

Zhang

B.-T.

(2016). Human body orientation estimation using convolutional neural network (version 1). arXiv. https://doi.org/10.48550/ARXIV.1609.01984

13.

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104

14.

Corder

S. P.

(1967). The significance of LEARNER’S errors. IRAL - International Review of Applied Linguistics in Language Teaching, 5(1–4), 161–170. https://doi.org/10.1515/iral.1967.5.1-4.161

15.

Cortes

Vapnik

(1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018

16.

Cover

Hart

(1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27. https://doi.org/10.1109/TIT.1967.1053964

17.

Domingos

Pazzani

(1997). On the optimality of the simple bayesian classifier under zero-one loss. Machine Learning, 29(2-3), 103–130. https://doi.org/10.1023/A:1007413511361

18.

Dong

Chen

Yang

(2023). Childhood emotional neglect and adolescent depression: The role of self-compassion and friendship quality. Current Psychology, 42(28), 24451–24463. https://doi.org/10.1007/s12144-022-03539-4

19.

Dou

Zhang

(2009). Organizational behavior textbook. Tsinghua University Press.

20.

Ekman

Friesen

W. V.

(1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1(1), 49–98. https://doi.org/10.1515/semi.1969.1.1.49

21.

Eryilmaz

Van Der Pol

Ryan

Clark

P. M.

Mary

(2013). Enhancing student knowledge acquisition from online learning conversations. International Journal of Computer-Supported Collaborative Learning, 8(1), 113–144. https://doi.org/10.1007/s11412-012-9163-y

22.

Flanders

N. A.

(1963). Intent, action and feedback: A preparation for teaching. Journal of Teacher Education, 14(3), 251–260. https://doi.org/10.1177/002248716301400305

23.

Gao

(2012). Analysis of the relationship between the behaviors of teacher and student. Educational Research, 33(03), 100–106.

24.

Gibbs

R. W.

(2005). Embodiment and cognitive science (1st ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511805844

25.

Zhou

(2020). Online teaching gestures recognition model based on deep learning. In 2020 International Conference on Networking and Network Applications (NaNA) (pp. 410–416). IEEE. https://doi.org/10.1109/NaNA51271.2020.00076

26.

Wang

(2004). New exploration of classroom analysis techniques to support teacher professional development. China Educational Technology, (07), 18–21.

27.

Girshick

(2015). Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1440–1448). Santiago, Chile.

28.

Zhang

Ren

Sun

(2016). Deep residual learning for image recognition. arxiv, pp. 770–778. https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html

29.

Heller

(1998). Communicate clearly. Dorling Kindersley. https://archive.org/details/communicateclear0000hell

30.

Henderson

M. L.

Schroeder

N. L.

(2021). A systematic review of instructor presence in instructional videos: Effects on learning and affect. Computers and Education Open, 2, 100059. https://doi.org/10.1016/j.caeo.2021.100059

31.

Hostetter

A. B.

Alibali

M. W.

(2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin & Review, 15(3), 495–514. https://doi.org/10.3758/PBR.15.3.495

32.

Jeong

A. C.

(2003). The sequential analysis of group interaction and critical thinking in online. American Journal of Distance Education, 17(1), 25–43. https://doi.org/10.1207/S15389286AJDE1701_3

33.

Keith

L. T.

Tornatzky

L. G.

Pettigrew

L. E.

(1974). An analysis of verbal and nonverbal classroom teaching behaviors. The Journal of Experimental Education, 42(4), 30–38. https://doi.org/10.1080/00220973.1974.11011490

34.

Lachner

Backfisch

Franke

(2024). Towards an integrated perspective of teachers’ technology integration: A preliminary model and future research directions. Frontline Learning Research, 12(1), 1–15. https://doi.org/10.14786/flr.v12i1.1179

35.

LeCun

Bengio

Hinton

(2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539

36.

Leigh

Mead

(2005). Lifting teacher performance. Policy report. In Progressive policy institute. Progressive Policy Institute. https://eric.ed.gov/?id=ED491196

37.

Leong

Byrne

Clackson

Georgieva

Lam

Wass

(2017). Speaker gaze increases information coupling between infant and adult brains. Proceedings of the National Academy of Sciences, 114(50), 13290–13295. https://doi.org/10.1073/pnas.1702493114

38.

Banerjee

N. K.

Dey

(2023). Simultaneous prediction of hand gestures, handedness, and hand keypoints using thermal images. In Das

Binong

Krejcar

Bhattacharjee

(Eds.), Proceedings of international conference on data, electronics and computing (pp. 117–127): Springer Nature. https://doi.org/10.1007/978-981-99-1509-5_12

39.

Saudagar

A. K. J.

Badshah

A. M.

Muhammad

Liu

(2023). Student behavior recognition for interaction detection in the classroom environment. Image and Vision Computing, 136, 104726. https://doi.org/10.1016/j.imavis.2023.104726

40.

(1995). Posture language commonly used by Americans. People’s Education Press.

41.

Liu

Zhang

Chen

Wang

Yuan

Xie

(2022). Categorizing teachers’ gestures in classroom teaching: From the perspective of multiple representations. Social Semiotics, 32(2), 184–204. https://doi.org/10.1186/s12985-022-01919-7

42.

Liu

(2019). Research on intelligent perception and recognition method for quantification of teacher’s credibility. Central China Normal University.

43.

Liu

Chen

Wang

(2019). Automatic instructional pointing gesture recognition by machine learning in the intelligent learning environment. In Proceedings of the 2019 4th international conference on distance education and learning (pp. 153–157). IEEE. https://doi.org/10.1145/3338147.3338163

44.

Liu

Anguelov

Erhan

Szegedy

Reed

C. Y.

Berg

A. C.

(2016). Ssd: Single shot MultiBox detector. In Leibe

Matas

Sebe

Welling

(Eds.), Computer vision – ECCV 2016 (9905, pp. 21–37). Springer International Publishing. https://doi.org/10.1007/978-3-319-46448-0_2

45.

Liu

Han

Zhou

(2025). Joint multi-person body detection and orientation estimation via one unified embedding. In Lin

Cheng

M.-M.

Ubul

Silamu

Zha

Zhou

Liu

C.-L.

(Eds.), Pattern recognition and computer vision (15039, pp. 467–480). Springer Nature. https://doi.org/10.1007/978-981-97-8692-3_33

46.

Mann

H. B.

Whitney

D. R.

(1947). On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, 18(1), 50–60. https://doi.org/10.1214/aoms/1177730491

47.

Mayer

R. E.

(2002). Multimedia learning. The Annual Report of Educational Psychology in Japan, 41(0), 27–29. https://doi.org/10.5926/arepj1962.41.0_27

48.

Mayer

R. E.

(2005). The cambridge handbook of multimedia learning. Cambridge University Press.

49.

McIntyre

N. A.

Mulder

K. T.

Mainhard

M. T.

(2020). Looking to relate: Teacher gaze and culture in student-rated teacher interpersonal behaviour. Social Psychology of Education, 23(2), 411–431. https://doi.org/10.1007/s11218-019-09541-2

50.

McNeill

(1996). Hand and mind: What gestures reveal about thought. University of Chicago Press. https://press.uchicago.edu/ucp/books/book/chicago/H/bo3641188.html

51.

McNeill

(2008). Gesture and thought. In Gesture and thought. University of Chicago Press. https://doi.org/10.7208/9780226514642/html

52.

Mishra

Pandey

Singh

Gupta

Sahu

Keshri

(2019). Descriptive statistics and normality tests for statistical data. Annals of Cardiac Anaesthesia, 22(1), 67–72. https://doi.org/10.4103/aca.ACA_157_18

53.

Liu

Yang

(2020). Instructor presence in video lectures: Eye gaze matters, but not body orientation. Computers & Education, 144, 103713. https://doi.org/10.1016/j.compedu.2019.103713

54.

Pianta

R. C.

La Paro

K. M.

Hamre

B. K.

(2008). Classroom assessment scoring System^TM: Manual K-3 (pp. xi–112). Paul H. Brookes Publishing Co.

55.

Quinlan

J. R.

(1986). Induction of decision trees. Machine Learning, 1(1), 81–106. https://doi.org/10.1007/BF00116251

56.

Redmon

Divvala

Girshick

Farhadi

(2016). You only look once: Unified, real-time object detection. 2016 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 779–788). IEEE. https://doi.org/10.1109/CVPR.2016.91

57.

Sackett

G. P.

(1978). Observing behavior: I. Theory and applications in mental retardation (pp. xv–416). University Park.

58.

Sahlender

Ten Hagen

(2023). Do teachers adapt their gestures in linguistically heterogeneous second language teaching to learners’ language proficiencies? Gesture, 22(2), 189–226. https://doi.org/10.1075/gest.22023.sah

59.

Shehata

Tlili

Huang

Adarkwah

M. A.

Liu

Chang

(2024). How are we doing with student-centered learning facilitated by educational technologies? A systematic review of literature reviews. Education and Information Technologies, 29(7), 7813–7854. https://doi.org/10.1007/s10639-023-12112-w

60.

Shen

(1983). Non-verbal behavior in the classroom. Primary & Secondary Schooling Abroad, (04), 23–27.

61.

Simon

Joo

Matthews

Sheikh

(2017). Hand keypoint detection in single images using multiview bootstrapping. In 2017 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 4645–4653). IEEE. https://doi.org/10.1109/CVPR.2017.494

62.

Smidekova

Janik

Minarikova

Holmqvist

(2018). Teachers’ gaze over space and time in a real-world classroom. Journal of Eye Movement Research, 13(4), 10–16910. https://doi.org/10.16910/jemr.13.4.1

63.

Stull

A. T.

Fiorella

Mayer

R. E.

(2018). An eye-tracking analysis of instructor presence in video lectures. Computers in Human Behavior, 88, 263–272. https://doi.org/10.1016/j.chb.2018.07.019

64.

Stylianou

D. A.

Lee

Ristroph

Knuth

Blanton

Stephens

Gardiner

(2024). Semiotic mediation of gestures in the teaching of early algebra: The case of the equal sign. Educational Studies in Mathematics, 116(2), 257–279. https://doi.org/10.1007/s10649-024-10319-3

65.

Sun

Xiao

Liu

Wang

(2019). Deep high-resolution representation learning for human pose estimation. Open Access, 5693–5703. https://openaccess.thecvf.com/content_CVPR_2019/html/Sun_Deep_High-Resolution_Representation_Learning_for_Human_Pose_Estimation_CVPR_2019_paper.html

66.

Van Gog

(2021). The signaling (or cueing) principle in multimedia learning. In Mayer

R. E.

Fiorella

(Eds.), The cambridge handbook of multimedia learning (3rd ed., pp. 221–230). Cambridge University Press. https://doi.org/10.1017/9781108894333.022

67.

Wang

(2019). The instructor’s gaze guidance in video lectures improves learning. Journal of Computer Assisted Learning, 35(1), 42–50. https://doi.org/10.1111/jcal.12309

68.

Wang

(2017). Research on teaching phenomena based on big data of classroom teaching behavior. E-education Research, 38(4), 77–85. https://doi.org/10.13811/j.cnki.eer.2017.04.012

69.

Wang

(2009). The nature of pedagogical activity: A special life process. Curriculum, Teaching Material and Method, (10), 34–39. https://doi.org/10.19877/j.cnki.kcjcjf.2009.10.007

70.

Weineck

Schultchen

Hauke

Messner

Pollatos

(2020). Using bodily postures to reduce anxiety and improve interoception: A comparison between powerful and neutral poses. Plos One, 15(12), Article e0242578. https://doi.org/10.1371/journal.pone.0242578

71.

Chen

Luo

C.-C.

Dawane

Hanzra

Deng

Liu

Wang

J. Z.

Kuo

(2020). Mebow: Monocular estimation of body orientation in the wild. In 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 3448–3458). IEEE. https://doi.org/10.1109/CVPR42600.2020.00351

72.

(1999). On nonverbal behavior in classroom instruction. Teacher Education Research, (05), 62–66.

73.

Jin

Liu

Qian

Ouyang

Luo

Wang

(2022). ZoomNAS: Searching for whole-body human pose estimation in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4), 5296–5313. https://doi.org/10.1109/TPAMI.2022.3197352

74.

Feng

Zhang

Yang

(2018). Depth based hand gesture recognition for smart teaching. In 2018 international conference on security, pattern analysis, and cybernetics (SPAC) (pp. 387–390). IEEE. https://doi.org/10.1109/SPAC46244.2018.8965567

75.

Yang

Zhu

Guo

(2020). Instructors’ gestures enhance their teaching experience and performance while recording video lectures. Journal of Computer Assisted Learning, 36(2), 189–198. https://doi.org/10.1111/jcal.12397

76.

Yang

Liu

Luo

(2009). Discussion on VICS, a classification system for instructional behaviors. Teacher, (06), 3–4. https://doi.org/10.3969/j.issn.1674-120X.2009.06.001

77.

Yoon

H. Y.

Kang

Kim

(2024). A non‐verbal teaching behaviour analysis for improving pointing out gestures: The case of asynchronous video lecture analysis using deep learning. Journal of Computer Assisted Learning, 40(3), 1006–1018. https://doi.org/10.1111/jcal.12933

78.

Zhao

Chu

Zhu

Wang

Xia

(2023). Mining the relationship between teacher behavior, eye-gaze and gesture in classroom teaching: A multi-dimension analysis approach. In 2023 international conference on intelligent education and intelligent research (IEIR) (pp. 1–6). IEEE. https://doi.org/10.1109/IEIR59294.2023.10391218

79.

Zheng

Liang

Luo

Yin

(2022). Attention assessment based on multi‐view classroom behaviour recognition. IET Computer Vision. https://doi.org/10.1049/cvi2.12146

80.

Zhou

(2003). The quantifying study on teachers’ nonverbal BehaviorIn classroom instruction. Guangxi Normal University.

81.

Zhou

(2006). A brief introduction to the study of teachers’ nonverbal behavior. Publishing House of Minority Nationalities.

Mining the Patterns of Teachers’ Nonverbal Behavior: Automated Recognition and Systematic Exploration

Abstract

Keywords

Get full access to this article

References