This study used a randomized, controlled trial to assess the impact of the automated writing evaluation system MI Write on middle school students from traditionally underserved groups, including Black and Hispanic/Latinx students and those experiencing poverty. The research focused on whether students who used MI Write for one school year (n = 1,260) showed improved argumentative writing performance, increased self-efficacy, greater enjoyment of writing, and stronger beliefs in the recursive nature of writing compared with those who did not use MI Write (n = 1,227). No significant overall treatment effects were found, but significant treatment by district interactions emerged. Perceptions of MI Write were positive overall but varied by district. Findings reveal the limitations and promise of automated writing evaluation.
ArcherA. L.HughesC. A. (2011). Explicit instruction: Effective and efficient teaching. Guilford Press.
2.
BarrotJ. S. (2024). Trends in automated writing evaluation systems research for teaching, learning, and assessment: A bibliometric analysis. Education and Information Technologies, 29, 7155–7199. https://doi.org/10.1007/s10639-023-12083-y
3.
BruningR.DempseyM.KauffmanD. F.McKimC.ZumbrunnS. (2013). Examining dimensions of self-efficacy for writing. Journal of Educational Psychology, 105(1), 25–38. https://doi.org/10.1037/a0029692
4.
ChenC. F. E.ChengW. Y. E. (2008). Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology, 12, 94–112. https://doi.org/10125/44145
5.
DelgadoA.WilsonJ.PalermoC.ZhangS.Cruz CorderoT.MyersM.PotterA.EackerH.ColesJ. (2024). Relationships between middle-school teachers’ perceptions and application of automated writing evaluation and student performance. In ShermisM. D.WilsonJ. (Eds.), The Routledge international handbook of automated essay evaluation (pp. 261–277). Routledge.
6.
Educational Testing Service (ETS). (2024). California Assessment of Student Performance and Progress Smarter Balanced Summative Assessment 2022–23 technical report (Final submitted March 19, 2024; Contract No. CN220002). California Department of Education, Assessment Development & Administration Division.
7.
FerrettiR. P.FanY. (2016). Argumentative writing. In MacArthurC. A.GrahamS.FitzgeraldJ. (Eds.), Handbook of writing research (2nd ed., pp. 301–315). Guilford Press.
8.
FleckensteinJ.LiebenowL. W.MeyerJ. (2023). Automated feedback and writing: A multi-level meta-analysis of effects on students’ performance. Frontiers in Artificial Intelligence, 6, 1162454. https://doi.org/10.3389/frai.2023.1162454
9.
FoltzP. W. (2014, May). Improving student writing through automated formative assessment: Practices and results [Paper presentation]. International Association for Educational Assessment (IAEA), Singapore.
10.
FuQ.-K.ZouD.XieH.ChengG. (2024). A review of AWE feedback: Types, learning, outcomes, and implications. Computer Assisted Language Learning, 37(1–2), 179–221. https://doi.org/10.1080/09588221.2022.2033787
GrahamS.HarrisK. R. (1993). Self-regulated strategy development: Helping students with learning problems develop as writers. The Elementary School Journal, 94(2), 169–181. https://doi.org/10.1086/461758
14.
GrahamS.HebertM.HarrisK. R. (2015). Formative assessment and writing: A meta-analysis. Elementary School Journal, 115(4), 523–547. https://doi.org/10.1086/681947
15.
GrahamS.CapizziA.HarrisK. R.HebertM.MorphyP. (2014). Teaching writing to middle school students: A national survey. Reading and Writing, 27, 1015–1042. https://doi.org/10.1007/s11145-013-9495-7
16.
GrahamS.CaoY.KimY.-S.LeeJ.TateT.CollinsP.ChoM.MoonY.ChungH. Q.OlsonC. B. (2024). Effective writing instruction for students in grades 6 to 12: A best evidence meta-analysis. Reading and Writing, 38, 1–46. https://doi.org/10.1007/s11145-024-10539-2
17.
GrimesD.WarschauerM. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment, 8(6), 1–44. http://www.jtla.org
18.
HuangX.XuW.LiF.YuZ. (2024). A meta-analysis of effects of automated writing evaluation on anxiety, motivation, and second language writing skills. The Asia-Pacific Education Researcher, 33(6), 957–976. https://doi.org/10.1007/s40299-024-00865-y
19.
HuangX.ZouD.ChengG.ChenX.XieH. (2023). Trends, research issues and applications of artificial intelligence in language education. Educational Technology & Society, 26(1), 112–131. https://doi.org/10.30191/ETS.202301_26(1).0009
LandauerT. K.FoltzP. W.LahamD. (1998). An introduction to latent semantic analysis. Discourse Processes, 25(2&3), 259–284. https://doi.org/10.1080/01638539809545028
22.
LiR. (2023). Still a fallible tool? Revisiting effects of automated writing evaluation from activity theory perspective. British Journal of Educational Technology, 54(3), 773–789. https://doi.org/10.1111/bjet.13294
23.
LinkS.MehrzadM.RahimiM. (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 35(4), 605–634. https://doi.org/10.1080/09588221.2020.1743323
24.
MacArthurC. A.PhilippakosZ. A.IanettaM. (2015). Self-regulated strategy instruction in college developmental writing. Journal of Educational Psychology, 107(3), 855–867. https://doi.org/10.1037/edu0000011
25.
MatsumuraL. C.Patthey-ChavezG. G.ValdesR.GarnierH. (2002). Teacher feedback, writing assignment quality, and third-grade students’ revision in lower- and higher-achieving urban schools. Elementary School Journal, 103, 3–25. https://www.jstor.org/stable/1002306
26.
MayfieldE.ButlerS. (2018). Districtwide implementations outperform isolated use of automated feedback in high school writing [Paper presentation]. International Conference of the Learning Sciences, London, United Kingdom. http://ceur-ws.org/Vol-2128/industrial4.pdf
27.
MooreN. S.MacArthurC. A. (2016). Student use of automated essay evaluation technology during revision. Journal of Writing Research, 8, 149–175. https://doi.org/10.17239/jowr-2016.08.01.05
28.
MuthénB. O.SatorraA. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. http://www.jstor.org/stable/271070
National Center for Education Statistics. (2012). The nation’s report card: Writing 2011 (NCES 2012–470). Institute of Education Sciences, U.S. Department of Education.
31.
NgoT. T.-N.ChenH. H.-J.LaiK. K.-W. (2022). The effectiveness of automated writing evaluation in EFL/ESL writing: A three level meta-analysis. Interactive Learning Environments. Advance online publication. https://doi.org/10.1080/10494820.2022.2096642
32.
NunesA.CordeiroC.LimpoT.CastroS. L. (2022). Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020. Journal of Computer Assisted Learning, 38(2), 599–620. https://doi.org/10.1111/jcal.12635
33.
PageE. B. (2003). Project essay grade: PEG. In ShermisM. D.BursteinJ. (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 43–54). Lawrence Erlbaum Associates.
34.
PalermoC.ThomsonM. M. (2018). Teacher implementation of self-regulated strategy development with an automated writing evaluation system: Effects on the argumentative writing performance of middle school students. Contemporary Educational Psychology, 54, 255–270. https://doi.org/10.1016/j.cedpsych.2018.07.002
35.
PalermoC.WilsonJ. (2020). Implementing automated writing evaluation in different instructional contexts: A mixed-methods study. Journal of Writing Research, 12(1), 63–108. https://doi.org/10.17239/jowr-2020.12.01.04
36.
PotterA.WilsonJ. (2021). Statewide implementation of automated writing evaluation: Analyzing usage and associations with state test performance in grades 4–11. Education Technology Research & Development, 69, 1557–1578. https://doi.org/10.1007/s11423-021-10004-9
37.
PumaM. J.OlsenR. B.BellS. H.PriceC. (2009). What to do when data are missing in group randomized controlled trials (NCEE 2009-0049). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
38.
RaudenbushS. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173. https://doi.org/10.1037//1082-989x.2.2.173
39.
RubinD. B. (1987). Multiple imputation for nonresponse in surveys. Wiley.
40.
Sanders-ReioJ.AlexanderP. A.ReioT. G.Jr.NewmanI. (2014). Do students’ beliefs about writing relate to their writing self-efficacy, apprehension, and performance?Learning and Instruction, 33, 1–11. https://doi.org/10.1016/j.learninstruc.2014.02.001
41.
SchererS.GrahamS.BusseV. (2024). How effective is feedback for L1, L2 and FL learners’ writing? A meta-analysis. Learning and Instruction, 93, 101961. https://doi.org/10.1016/j.learninstruc.2024.101961
42.
StroblC.AilhaudE.BenetosK.DevittA.KruseO.ProskeA.RappC. (2019). Digital support for academic writing: A review of technologies and pedagogies. Computers & Education, 131, 33–48. https://doi.org/10.1016/j.compedu.2018.12.005
43.
WangE. L.MatsumuraL. C.CorrentiR.LitmanD.ZhangH.HoweE.MagoodaA.QuintanaR. (2020). eRevis(ing): Students’ revision of text evidence use in an automated writing evaluation system. Assessing Writing, 44, 100449. https://doi.org/10.1016/j.asw.2020.100449
44.
What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance (NCEE). https://ies.ed.gov/ncee/wwc/Handbooks
45.
WilsonJ.CzikA. (2016). Automated essay evaluation software in English language arts classrooms: Effects on teacher feedback, student motivation, and writing quality. Computers and Education, 100, 94–109. https://doi.org/10.1016/j.compedu.2016.05.004
46.
WilsonJ.RoscoeR. D. (2020). Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research, 58(1), 87–125. https://doi.org/10.1177/0735633119830764
47.
WilsonJ.PotterA. H.Cruz CorderoT.MyersM. (2023). Integrating goal-setting and automated feedback to improve writing outcomes: A pilot study. Innovation in Language Learning and Teaching, 17(3), 518–534. https://doi.org/10.1080/17501229.2022.2077348
48.
WilsonJ.HuangY.PalermoC.BeardG.MacArthurC. A. (2021). Automated feedback and automated scoring in the elementary grades: Usage, attitudes, and associations with writing outcomes in a districtwide implementation of MI Write. International Journal of Artificial Intelligence in Education, 31(2), 234–276. https://doi.org/10.1007/s40593-020-00236-w
49.
WilsonJ.AhrendtC.FudgeE.RaicheA.BeardG.MacArthurC. A. (2021). Elementary teachers’ perceptions of automated feedback and automated scoring: Transforming the teaching and learning of writing using automated writing evaluation. Computers & Education, 168, 104208. https://doi.org/10.1016/j.compedu.2021.104208
50.
WilsonJ.ZhangF.PalermoC.Cruz CorderoT.MyersM.EackerH.PotterA.ColesJ. (2024). Predictors of middle school students’ perceptions of automated writing evaluation. Computers & Education, 211, 104985. https://doi.org/10.1016/j.compedu.2023.104985
51.
ZhaiN.MaZ. (2023). The effectiveness of automated writing evaluation on writing quality: A meta-analysis. Journal of Educational Computing Research, 61(4), 875–900. https://doi.org/10.1177/07356331221127300
