Automated writing evaluation (AWE) has been frequently used to provide feedback on student writing. Many empirical studies have examined the effectiveness of AWE on writing quality, but the results were inconclusive. Thus, the magnitude of AWE’s overall effect and factors influencing its effectiveness across studies remained unclear. This study re-examined the issue by meta-analyzing the results of 26 primary studies with a total of 2468 participants from 2010 to 2022. The results revealed that AWE had a large positive overall effect on writing quality (g = 0.861, p < 0.001). Further moderator analyses indicated that AWE was more effective for post-secondary students than for secondary students and had more benefits for English as a Foreign Language (EFL) and English as a Second Language (ESL) learners than for Native English Speaker (NES) learners. When the genre of writing was considered, AWE showed a more significant impact on argumentative writing than on academic and mixed writing genres. However, intervention duration, feedback combination, and AWE platform did not moderate the effect of AWE on writing quality. The implications and recommendations for both research and practice are discussed in depth.
AlexanderO.ArgentS.SpencerJ. (2008). EAP essentials: A teacher's guide to principles and practice. Garnet Education.
2.
BaklaA. (2020). A mixed-methods study of feedback modes in EFL writing. Language Learning & Technology, 24(1), 107–128.
3.
BarkaouiK. (2016). What and when second‐language learners revise when responding to timed writing tasks on the computer: The roles of task type, second language proficiency, and keyboarding skills. The Medico-legal Journal, 100(1), 320–340. https://doi.org/10.1111/modl.12316
4.
BarrotJ. S. (2021). Using automated written corrective feedback in the writing classrooms: Effects on L2 writing accuracy. Computer Assisted Language Learning, 1(1), 1–24.
5.
BorensteinM.HedgesL. V.HigginsJ. P.RothsteinH. R. (2011). Introduction to meta-analysis. John Wiley & Sons.
6.
ChangT. S.LiY.HuangH. W.WhitfieldB. (2021). Exploring EFL students' writing performance and their acceptance of AI-based automated writing feedback. In 2021 2nd International Conference on Education Development and Studies (pp. 31–35). ACM.
7.
ChengG. (2017). The impact of online automated feedback on students' reflective journal writing in an EFL course. The Internet and Higher Education, 34(1), 18–27.
8.
ChewC. S.WuW. C. V.IdrisN.LohE. F.ChuaY. P. (2019). Enhancing summary writing of ESL learners via a theory-based online tool: System development and evaluation. Journal of Educational Computing Research, 58(2), 398–432.
9.
CohenJ. (2013). Statistical power analysis for the behavioral sciences. Routledge.
10.
CrossleyS. (2020). Linguistic features in writing quality and development: An overview. Journal of Writing Research, 11(3), 415–443. https://doi.org/10.17239/jowr-2020.11.03.01
11.
CuijpersP.WeitzE.CristeaI. A.TwiskJ. (2017). Pre-post effect sizes should be avoided in meta-analyses. Epidemiology and Psychiatric Sciences, 26(4), 364–368. https://doi.org/10.1017/s2045796016000809
DeaneP. (2013). On the relation between automated essay scoring and modern views of the writing construct. Assessing Writing, 18(1), 7–24. https://doi.org/10.1016/j.asw.2012.10.002
14.
DikliS.BleyleS. (2014). Automated essay scoring feedback for second language writers: How does it compare to instructor feedback?Assessing Writing, 22, 1–17. https://doi.org/10.1016/j.asw.2014.03.006
15.
EggerM.SmithG. D.SchneiderM.MinderC. (1997). Bias in meta-analysis detected by a simple, graphical test. The British Medical Journal, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629
16.
EllisR.EllisR. R. (2008). The study of second language acquisition (2nd ed.). Oxford University.
17.
FuQ. K.ZouD.XieH.ChengG. (2022). A review of AWE feedback: Types, learning outcomes, and implications. Computer Assisted Language Learning, 1(1), 1–43. https://doi.org/10.1080/09588221.2022.2033787
18.
GarzónJ.AcevedoJ. (2019). Meta-analysis of the impact of Augmented Reality on students’ learning gains. European Respiratory Review: an Official Journal of the European Respiratory Society, 27(1), 244–260. https://doi.org/10.1016/j.edurev.2019.04.001
19.
GrahamS.LiuX.BartlettB.NgC.HarrisK. R.AitkenA.BarkelA.KavanaughC.TalukdarJ. (2018). Reading for writing: A meta-analysis of the impact of reading interventions on writing. Review of Educational Research, 88(2), 243–284. https://doi.org/10.3102/0034654317746927
20.
HanY.ZhaoS.NgL. L. (2021). How technology tools impact writing performance, lexical complexity, and perceived self-regulated learning strategies in EFL academic writing: A comparative study. Frontiers in Psychology, 12(1), 752793.
21.
HassanzadehM.FotoohnejadS. (2021). Implementing an automated feedback program for a Foreign Language writing course: A learner‐centric study: Implementing an AWE tool in a L2 class. Journal of Computer Assisted Learning, 37(5), 1494–1507.
HedgesL. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. (Journal of Environmental Sciences China), 6(2), 107–128. https://doi.org/10.2307/1164588
24.
HewK. F.LoC. K. (2018). Flipped classroom improves student learning in health professions education: A meta-analysis. BMC Medical Education, 18(1), 38. https://doi.org/10.1186/s12909-018-1144-z
25.
HigginsD.HeilmanM. (2014). Managing what we can measure: Quantifying the susceptibility of automated scoring systems to gaming behavior. Educational Measurement: Issues and Practice, 33(3), 36–46. https://doi.org/10.1111/emip.12036
26.
HigginsJ.ThompsonS.G.DeeksJ.J.AltmanD.G. (2003). Measuring inconsistency in meta-analyses. Bmj: British Medical Journal, 327(7414), 557–560. https://doi.org/10.1136/bmj.327.7414.557
HousenA.KuikenF. (2009). Complexity, accuracy, and fluency in second language acquisition. Applied linguistics, 30(4), 461–473. https://doi.org/10.1093/applin/amp048
29.
HuismanB.SaabN.van den BroekP.van DrielJ. (2019). The impact of formative peer feedback on higher education students’ academic writing: A meta-analysis. Assessment & Evaluation in Higher Education, 44(6), 863–880. https://doi.org/10.1080/02602938.2018.1545896
30.
JiangL.YuS.WangC. (2020). Second Language writing instructors’ feedback practice in response to automated writing evaluation: A sociocultural perspective. System, 93, 102302. https://doi.org/10.1016/j.system.2020.102302
31.
KangE.HanZ. (2015). The efficacy of written corrective feedback in improving L2 written accuracy: A meta‐analysis. The Modern Language Journal, 99(1), 1–18. https://doi.org/10.1111/modl.12189
32.
KarimK.NassajiH. (2020). The revision and transfer effects of direct and indirect comprehensive corrective feedback on ESL students’ writing. Language Teaching Research, 24(4), 519–539. https://doi.org/10.1177/1362168818802469
33.
KelloggR. T.WhitefordA. P.QuinlanT. (2010). Does automated feedback help students learn to write?Journal of Educational Computing Research, 42(2), 173–196.
34.
KnightS.ShibaniA.AbelS.GibsonA.RyanP. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12(1), 141–186.
35.
LiJ.LinkS.HegelheimerV. (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing, 27(1), 1–18.
36.
LinkS.MehrzadM.RahimiM. (2020). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 1(1), 1-30.
37.
LongM. H. (1996). The role of the linguistic environment in second language acquisition. In RitchieW.BhatiaT.K. (Eds), Handbook of Second Language acquisition (pp. 413–468). Academic Press.
38.
LuX. (2019). An empirical study on the artificial intelligence writing evaluation system in China CET. Big Data, 7(2), 121–129.
39.
LvX.RenW.XieY. (2021). The effects of online feedback on ESL/EFL writing: A meta-analysis. The Asia-Pacific Education Researcher, 30(6), 1–11. https://doi.org/10.1007/s40299-021-00594-6
40.
MohsenM. A. (2022). Computer-mediated corrective feedback to improve L2 writing skills: A meta-analysis. Journal of Educational Computing Research, 60(5), 1276. https://doi.org/10.1177/07356331211064066
41.
NgoT. T. N.ChenH. H. J.LaiK. K. W. (2022). The effectiveness of automated writing evaluation in EFL/ESL writing: A three-level meta-analysis. Interactive Learning Environments, 1(1), 1–18. https://doi.org/10.1080/10494820.2022.2096642
42.
NunesA.CordeiroC.LimpoT.CastroS. L. (2021). Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020. Journal of Computer Assisted Learning, 38(2), 599–620. https://doi.org/10.1111/jcal.12635
43.
PageM. J.McKenzieJ. E.BossuytP. M.BoutronI.HoffmannT. C.MulrowC. D.ShamseerL.TetzlaffJ. M.AklE. A.BrennanS. E.ChouR.GlanvilleJ.GrimshawJ. M.HrobjartssonA.LaluM. M.LiT.LoderE. W.Mayo-WilsonE.McDonaldS.MoherD. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. International Journal of Surgery, 88(1), 105906. https://doi.org/10.1016/j.ijsu.2021.105906
44.
PalermoC.ThomsonM. M. (2018). Teacher implementation of self-regulated strategy development with an automated writing evaluation system: Effects on the argumentative writing performance of middle school students. Contemporary Educational Psychology, 54(1), 255–270.
45.
PalermoC. J (2017). A framework for deliberate practice: Self-regulated strategy development and an automated writing evaluation program. North Carolina State University. Unpublished doctoral dissertation.
46.
ReynoldsB. L.KaoC. W.HuangY. Y. (2021). Investigating the effects of perceived feedback source on Second Language writing performance: A quasi-experimental study. The Asia-Pacific Education Researcher, 30(6), 585–595.
47.
RichC. S. (2012). The impact of online automated writing evaluation: A case study from dalian. Chinese Journal of Applied Linguistics, 35(1), 63–79.
48.
RoscoeR. D.WilsonJ.JohnsonA. C.MayraC. R. (2017). Presentation, expectations, and experience: Sources of student perceptions of automated writing evaluation. Computers in Human Behavior, 70(1), 207–221. https://doi.org/10.1016/j.chb.2016.12.076
49.
RosenthalR. (1991). Meta-analytic procedures for social research. Sage.
50.
SalavatizadehM.TahririA. (2020). The effect of blended online automated feedback and teacher feedback on EFL learners’ essay writing ability and perception. Teaching English As a Second Language, 39(32), 181–225.
51.
SchmidtR. (1995). Consciousness and Foreign Language learning: A tutorial on the role of attention and awareness in learning. Attention and Awareness in Foreign Language Learning, 9(1), 1–63.
52.
SkehanP. (1998). A cognitive approach to language learning. Oxford University Press.
53.
StevensonM.PhakitiA. (2014). The effects of computer-generated feedback on the quality of writing. Assessing Writing, 19(1), 51–65. https://doi.org/10.1016/j.asw.2013.11.007
54.
SunB.FanT. (2022). The effects of an AWE-aided assessment approach on business English writing performance and writing anxiety: A contextual consideration. Studies in Educational Evaluation, 72(1), 101123.
55.
TanS.ChoY. W.XuW. (2022). Exploring the effects of automated written corrective feedback, computer-mediated peer feedback and their combination mode on EFL learner’s writing performance. Interactive Learning Environments, 1(1), 1–11.
56.
TangJ.RichC. S. (2017). Automated writing evaluation in an EFL setting: Lessons from China. JALT CALL Journal, 13(2), 117–146.
57.
VygotskyL. (1978). Interaction between learning and development. Readings on the Development of Children, 23(3), 34–41.
58.
WangS.LiR. (2019). An empirical study on the impact of an automated writing assessment on Chinese college students’ English writing proficiency. International Journal of Language and Linguistics, 7(5), 218–229.
59.
WangY. J.ShangH. F.BriodyP. (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students' writing. Computer Assisted Language Learning, 26(3), 234–257.
60.
WangZ. (2020). Computer-assisted EFL writing and evaluations based on artificial intelligence: A case from a college reading and writing course. Library Hi Tech, 40(1), 80–97.
61.
WeigleS. C. (2013). English as a second language writing and automated essay evaluation. In ShermisM. D.BursteinJ. (Eds.), Handbook of automated essay evaluation (pp. 58–76). Routledge.
62.
WhiteH. D. (2009). Scientific communication and literature retrieval. The Handbook of Research Synthesis and Meta-Analysis, 2(1), 51–71. https://doi.org/10.7758/9781610448864.7
63.
WilsonJ.CzikA. (2016). Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing quality. Computers & Education, 100(1), 94–109.
64.
WilsonJ.RoscoeR. D. (2020). Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research, 58(1), 87–125.
65.
XuJ.ZhangS. (2021). Understanding AWE feedback and English writing of learners with different proficiency levels in an EFL classroom: A sociocultural perspective. The Asia-Pacific Education Researcher, 1(1), 1–11.
66.
XuZ.BanerjeeM.RamirezG.ZhuG.WijekumarK. (2019). The effectiveness of educational technology applications on adult English language learners’ writing quality: A meta-analysis. Computer Assisted Language Learning, 32(1–2), 132–162. https://doi.org/10.1080/09588221.2018.1501069
67.
ZhaiN.MaX. (2021). Automated writing evaluation (AWE) feedback: A systematic investigation of college students’ acceptance (pp. 1–26). Computer Assisted Language Learning.
68.
ZhuM.LiuO. L.LeeH. S. (2020). The effect of automated feedback on revision behavior and learning gains in formative assessment of scientific argument writing. Computers & Education, 143(1), 103668.
