Providing consistent, individualized feedback to teachers is essential for improving instruction but can be prohibitively resource-intensive in most educational contexts. We develop M-Powering Teachers, an automated tool based on natural language processing to give teachers feedback on their uptake of student contributions, a high-leverage dialogic teaching practice that makes students feel heard. We conduct a randomized controlled trial in an online computer science course (N = 1,136 instructors), to evaluate the effectiveness of our tool. We find that M-Powering Teachers improves instructors’ uptake of student contributions by 13% and present suggestive evidence that it also improves students’ satisfaction with the course and assignment completion. These results demonstrate the promise of M-Powering Teachers to complement existing efforts in teachers’ professional development.
AlicS.DemszkyD.MancenidoZ.LiuJ.HillH.JurafskyD. (2022). Computationally identifying funneling and focusing questions in classroom discourse. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022) (pp. 224–233). Association for Computational Linguistics.
2.
AlrajhiL.AlamriA.PereiraF. D.CristeaA. I. (2021). Urgency analysis of learners’ comments: An automated intervention priority model for MOOC. In International Conference on Intelligent Tutoring Systems (pp. 148–160). Springer.
3.
AslanS.AlyuzN.TanrioverC.MeteS. E.OkurE.D’MelloS. K.Arslan EsmeA. (2019, May4–9). Investigating the impact of a real-time, multimodal student engagement analytics technology in authentic classrooms [Conference session]. CHI’19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.
4.
BeanR. M.DraperJ. A.HallV.VandermolenJ.ZigmondN. (2010). Coaches and coaching in reading first schools: A reality check. The Elementary School Journal, 111(1), 87–114.
5.
BrophyJ. E. (1984). Teacher behavior and student achievement (No. 73). Institute for Research on Teaching, Michigan State University.
6.
ClarkH. H.SchaeferE. F. (1989). Contributing to discourse. Cognitive Science, 13(2), 259–294.
7.
CohenD. K. (2011). Teaching and its predicaments. Harvard University Press.
8.
CollinsJ. (1982). Discourse style, classroom interaction and differential treatment. Journal of Reading Behavior, 14, 429–437.
9.
DanielsonC. (2007). Enhancing professional practice: A framework for teaching. Association for Supervision and Curriculum Development.
10.
DemszkyD.HillH. (2022). The NCTE transcripts: A dataset of elementary math classroom transcripts. arXiv preprint arXiv:2211.11772.
11.
DemszkyD.LiuJ.MancenidoZ.CohenJ.HillH.JurafskyD.HashimotoT. (2021). Measuring conversational uptake: A case study on student-teacher interactions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL-IJCNLP) (pp. 1638–1653). Association for Computational Linguistics.
12.
DonaldsonM. L.WoulfinS. (2018). From tinkering to going “rogue”: How principals use agency when enacting new teacher evaluation systems. Educational Evaluation and Policy Analysis, 40(4), 531–556.
13.
DonnellyP. J.BlanchardN.OlneyA. M.KellyS.NystrandM.D’MelloS. K. (2017). Words matter: Automatic detection of teacher questions in live classroom discourse using linguistics, acoustics and context. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference —LAK ’17 (pp. 218–227). Association for Computing Machinery.
14.
FirestoneW. A.DonaldsonM. L. (2019). Teacher evaluation as data use: What recent research suggests. Educational Assessment, Evaluation and Accountability, 31(3), 289–314.
15.
GibbonsL. K.CobbP. (2016). Content-focused coaching: Five key practices. The Elementary School Journal, 117(2), 237–260.
16.
GibbonsL. K.CobbP. (2017). Focusing on teacher learning opportunities to identify potentially productive coaching activities. Journal of Teacher Education, 68(4), 411–425.
17.
GodfreyJ. J.HollimanE. C.McDanielJ. (1992). Switchboard: Telephone speech corpus for research and development. In IEEE International Conference on Acoustics, Speech, and Signal Processing (Vol. 1, pp. 517–520). IEEE.
18.
GonzalezK.LynchK.HillH. C. (2022). A meta-analysis of the experimental evidence linking stem classroom interventions to teacher knowledge, classroom instruction, and student achievement. EdWorkingPaper.
19.
HellrungK.HartigJ. (2013). Understanding and using feedback—A review of empirical studies concerning feedback from external evaluations to teachers. Educational Research Review, 9, 174–190.
20.
Herbel-EisenmannB.DrakeC.CirilloM. (2009). “Muddying the clear waters”: Teachers’ take-up of the linguistic idea of revoicing. Teaching and Teacher Education, 25(2), 268–277.
21.
HigginsE. T.RholesW. S. (1978). “Saying is believing”: Effects of message modification on memory and liking for the person described. Journal of Experimental Social Psychology, 14(4), 363–378.
22.
HoA. D.KaneT. J. (2013). The reliability of classroom observations by school personnel [Research paper]. MET Project, Bill & Melinda Gates Foundation.
23.
JacobsJ.ScornavaccoK.HartyC.SureshA.LaiV.SumnerT. (2022). Promoting rich discussions in mathematics classrooms: Using personalized, automated feedback to support reflection and instructional change. Teaching and Teacher Education, 112, 103631.
24.
JensenE.DaleM.DonnellyP. J.StoneC.KellyS.GodleyA.D’MelloS. K. (2020). Toward automated feedback on teacher discourse to enhance teacher learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–13). Association for Computing Machinery.
25.
KaneB. D.RosenquistB. (2019). Relationships between instructional coaches’ time use and district- and school-level policies and expectations. American Educational Research Journal, 56(5), 1718–1768.
26.
KellyS.OlneyA. M.DonnellyP.NystrandM.D’MelloS. K. (2018). Automatically measuring question authenticity in real-world classrooms. Educational Researcher, 47, 451–464. https://doi.org/10.3102/0013189X18785613
27.
KizilcecR. F.LeeH. (2022). Algorithmic fairness in education. In HolmesW.Porayska-PomstaK. (Eds.), The ethics of artificial intelligence in education (pp. 174–202). Routledge.
28.
KoeneckeA.NamA.LakeE.NudellJ.QuarteyM.MengeshaZ.. . .GoelS. (2020). Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14), 7684–7689.
29.
KraftM. A.BlazarD.HoganD. (2018). The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research, 88(4), 547–588. https://doi.org/10.3102/0034654318759268
30.
KraftM. A.GilmourA. F. (2016). Can principals promote teacher development as evaluators? A case study of principals’ views and experiences. Educational Administration Quarterly, 52(5), 711–753.
31.
KraftM. A.HillH. C. (2020). Developing ambitious mathematics instruction through web-based coaching: A randomized field trial. American Educational Research Journal, 57(6), 2378–2414.
32.
LampertM. (2001). Teaching problems and the problems of teaching. Yale University Press.
33.
LiuJ.CohenJ. (2021). Measuring teaching practices at scale: A novel application of text-as-data methods. Educational Evaluation and Policy Analysis, 43, 587–614.
34.
MadaioM.BlodgettS. L.MayfieldE.Dixon-Rom´anE. (2022). Beyond “fairness”: Structural (in) justice lenses on AI for education. In HolmesW.Porayska-PomstaK. (Eds.), The ethics of artificial intelligence in education (pp. 203–239). Routledge.
35.
MadaioM.StarkL.Wortman VaughanJ.WallachH. (2020). Co-designing check lists to understand organizational challenges and opportunities around fairness in AI. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–14). Association for Computing Machinery.
36.
NystrandM.GamoranA.KachurR.PrendergastC. (1997). Opening dialogue. Teachers College Press.
37.
NystrandM.WuL. L.GamoranA.ZeiserS.LongD. (2000). Questions in time: Investigating the structure and dynamics of unfolding classroom discourse. National Research Center on English Learning and Achievement (CELA), The University of Wisconsin–Madison.
38.
O’ConnorM. C.MichaelsS. (1993). Aligning academic task and participation status through revoicing: Analysis of a classroom discourse strategy. Anthropology & Education Quarterly, 24, 318–335.
39.
Oolbekkink-MarchandH. W.HadarL. L.SmithK.HelleveI.UlvikM. (2017). Teachers’ perceived professional space and their agency. Teaching and Teacher Education, 62, 37–46.
40.
PiantaR. C.ParoL. M. K.HamreB. K. (2008). Classroom assessment scoring system™: Manual k–3. Paul H Brookes Publishing.
41.
PiechC.MalikA.JueK.SahamiM. (2021). Code in place: Online section leading for scalable human-centered learning. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (pp. 973–979). Association for Computing Machinery.
42.
PriestleyM.BiestaG. J. J.PhilippouS.RobinsonS. (2015). The teacher and the curriculum: Exploring teacher agency. In WyseD.HaywardL.PandyaJ. (Eds.), The SAGE handbook of curriculum, pedagogy and assessment (pp. 187–201). SAGE.
43.
RigbyJ. G.Larbi-CherifA.RosenquistB. A.SharpeC. J.CobbP.SmithT. (2017). Administrator observation and feedback: Does it lead toward improvement in inquiry-oriented math instruction?Educational Administration Quarterly, 53(3), 475–516.
44.
SameiB.OlneyA. M.KellyS.NystrandM.D’MelloS.BlanchardN.. . .GraesserA. (2014). Domain independent assessment of dialogic properties of classroom discourse. https://eric.ed.gov/?id=ED566380
45.
ScottS. E.CortinaK. S.CarlisleJ. F. (2012). Understanding coach-based professional development in reading first: How do coaches spend their time and how do teachers perceive coaches’ work?Literacy Research and Instruction, 51(1), 68–85.
46.
ShuteV. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.
47.
SteinbergM. P.SartainL. (2015). Does teacher evaluation improve school performance? Experimental evidence from Chicago’s excellence in teaching project. Education Finance and Policy, 10(4), 535–572.
48.
SureshA.JacobsJ.LaiV.TanC.WardW.MartinJ. H.SumnerT. (2021). Using transformers to provide teachers with personalized feedback on their classroom discourse: The talkmoves application [Preprint]. arXiv.
49.
TaieS.GoldringR. (2017). Characteristics of public elementary and secondary school teachers in the United States: Results from the 2015–16 national teacher and principal survey (First Look, NCES 2017-072). National Center for Education Statistics.
50.
TannenD. (1987). Repetition in conversation: Toward a poetics of talk. Language, 63, 574–605.
51.
TaylorE. S.TylerJ. H. (2012). The effect of evaluation on teacher performance. American Economic Review, 102(7), 3628–3651.
52.
WellsG. (1999). Dialogic inquiry: Towards a socio-cultural practice and theory of education. Cambridge University Press.
53.
WellsG.ArauzR. M. (2006). Dialogue in the classroom. The Journal of the Learning Sciences, 15(3), 379–428.
54.
WillsJ. S.Haymore SandholtzJ. (2009). Constrained professionalism: Dilemmas of teaching in the face of test-based accountability. Teachers College Record, 111(4), 1065–1114.
55.
WilsonJ.NazemiM.JacksonK.WilhelmA. G. (2019). Investigating teaching in conceptually oriented mathematics classrooms characterized by African American student success. Journal for Research in Mathematics Education, 50(4), 362–400.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
