Research on computer-based writing evaluation has only recently focused on the potential for providing formative feedback rather than summative assessment. This study tests the impact of an automated essay scorer (AES) that provides formative feedback on essay drafts written as part of a series of online teacher education case studies. Seventy preservice teachers in four teacher education classes were assigned to complete two cases. Each student was randomly assigned to either a condition where the AES was available (experimental condition) or a condition where the AES was unavailable (control condition). Students in the experimental condition who opted to use the AES submitted more highly rated final, human-scored essays (in the second of two case studies) and conducted more relevant searches (in both of the two case studies) than students either in the control condition or in the experimental condition who chose not to use the scorer.
Get full access to this article
View all access options for this article.
References
1.
BeachR. (1976). Self-evaluation strategies of extensive revisers and non-revisers. College and Communication, 27, 160–164.
2.
BeachR., & FriedrichT. (2005). Response to writing. In MacArthurC. A.GrahamS., & FitzgeraldJ. (Eds.), Handbook of writing research.Boston: Allyn and Bacon.
3.
BursteinJ. (2003). The e-rater scoring engine: Automated essay scoring with natural language processing. In ShermisM. D. & BursteinJ. (Eds.), Automated essay scoring: A cross-disciplinary perspective.Mahwah, NJ: Lawrence Erlbaum Associates.
4.
BursteinJ.KukichK.WolffS.ChiL., & ChodorowM. (1998). Enriching automated essay scoring using discourse marking. Proceedings of the Workshop on Discourse Relations and Discourse Marking, Annual Meeting of the Association of Computational Linguistics, Montreal, Canada.
5.
BursteinJ.LeacockC.SwartzR. (2001). Automated evaluation of essay and short answers. In DansonM. (Ed.), Proceedings of the Sixth International Computer Assisted Assessment Conference, Loughborough University, Loughborough, UK.
6.
CharmanD., & ElmesA. (1998, November). A computer-based formative assessment strategy for a basic statistics module in geography. Journal of Geography in Higher Education, 22(3), 381–385.
7.
ConradS. M., & GoldsteinL. M. (1999). ESL student revision after teacher-written comments: Text, contexts, and individuals. Journal of Second Language Writing, 8(3), 257–276.
8.
DanoffB.HarrisK. R., & GrahamS. (1993). Incorporating strategy instruction within the writing process in the regular classroom: Effects on the writing of students with and without learning disabilities. Journal of Reading Behavior, 25(3), 295–322.
9.
DexterS. (2002). eTIPS—Educational technology integration and implementation principles. In RodgersP. (Ed.), Designing instruction for technology-enhanced learning (pp. 56–70). New York: Idea Group Publishing.
10.
DukeB. L., & MillerR. (2004, April). The influence of using cognitive strategy instruction through writing rubrics on high school students' writing motivation and achievement. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.
11.
EmigJ. (1971). The composing processes of twelfth graders.Urbana, IL: National Council of Teachers of English.
12.
FerrisD. (2001). Teaching writing for academic purposes. In FlowerdewJ. & PeacockM. (Eds.), Research perspectives on English for academic purposes.New York: Cambridge University Press.
13.
FerrisD. R. (2003). Response to student writing: Implications for second language students.Mahwah, NJ: Erlbaum.
14.
FlemingV. M., & AlexanderJ. M. (2001). The benefits of peer collaboration: A replication with a delayed posttest. Contemporary Educational Psychology, 26, 588–601.
15.
GrahamS., & HarrisK. R. (1989). Components analysis of cognitive strategy instruction: Effects on learning disabled students' compositions and self-efficacy. Journal of Educational Psychology, 81(3), 353–361.
16.
HillocksG. (1986). Research on written composition: New directions for teaching.Urbana, IL: ERIC Clearinghouse on Reading and Communication Skills/National Conference on Research in English.
17.
KimL. (2004). Online technologies for teaching writing: Students react to teacher response in voice and written modalities. Research in the Teaching of English, 38(3), 304–337.
18.
LandR. E.Jr. & EvansS. (1987, February). Classroom inquiry: What our students taught us about paper marking. The English Journal, 76(2), 113–116.
19.
MatsumuraL. C.Patthey-ChavezG. G.ValdesR., & GarnierR. (2002). Teacher feedback, writing assignment quality, and third-grade students' revision in lower- and higher-achieving urban schools. The Elementary School Journal, 103(1), 3–25.
20.
MyersM. (2003). What can computers and AES contribute to a K-12 writing program? In ShermisM. D. & BursteinJ. C. (Eds.), Automated essay scoring: A cross-disciplinary perspective.Mahwah, NJ: Lawrence Erlbaum Associates.
21.
National Writing Project. (2003). Because writing matters: Improving student writing in our schools.San Francisco, CA: Jossey-Bass.
22.
PageE. B. (1966). The imminence of grading essays by computer. Phi Delta Kappan, 48, 238–243.
23.
PageE. B. (1994). Computer grading of student prose, using modern concepts and software. Journal of Experimental Education, 62, 127–142.
24.
Patthey-ChavezG. B.MatsumuraL., & ValdesR. (2004). Investigating the process approach to writing instruction in urban middle schools. Journal of Adolescent and Adult Literacy, 47(6), 462–479.
25.
PeatM., & FranklinS. (2002). Supporting student learning: The use of computer-based formative assessment modules. British Journal of Educational Technology, 33(5), 515–523.
26.
ReedW. M.BurtonJ. K., & KellyP. P. (1985). The effects of writing ability and mode of discourse on cognitive capacity engagement. Research in the Teaching of English, 19, 283–297.
27.
SambellK.SambellA., & SextonG. (1999). Student perceptions of the learning benefits of computer-assisted assessment: A case study in electronic engineering. In BrownS.RaceP., & BullJ. (Eds.), Computer-assisted assessment in higher education (pp. 179–191). London: Kogan Page Limited.
28.
ScharberC., & DexterS. (2004, March). Automated essay score predictions as a formative assessment tool. Paper presented at the fifteenth international conference of the Society for Information Technology and Teacher Education, Atlanta, GA.
29.
ShermisM. D., & BursteinJ. C. (2003). Automated essay scoring: A cross-disciplinary perspective.Mahwah, NJ: Lawrence Erlbaum Associates.
30.
ShermisM. D.BursteinJ., & LeacockC. (2006). Applications of computers in assessment and analysis of writing. In MacArthurC. A.GrahamS., & FitzgeraldJ. (Eds.), Handbook of writing research.New York: Guilford Publications.
31.
SommersJ. (1982). Responding to student writing. College Composition and Communication, 33(2), 148–156.
32.
SperlingM., & FreedmanS. W. (1987). A good girl writes like a good girl: Written responses to student writing. Written Communication, 9(9), 342–369.
33.
StoddardB., & MacArthurC. A. (1993). A peer editor strategy: Guiding learning disabled students in response and revision. Research in the Teaching of English, 27(1), 76–102.
34.
StraubR. (1996). The concept of control in teacher response: Defining the varieties of “directive” and “facilitative” commentary. College Composition and Communication, 47(2), 223–251.
35.
ValentiS.NeriF., & CucchiarelliA. (2003). An overview of current research on automated essay grading. Journal of Information Technology Education, 2, 319–330.
36.
WallaceD. L., & HayesJ. R. (1992). Redefining revision for freshmen. In HayesJ. R.YoungR. E. (Eds.), Reading empirical research studies: The rhetoric of research (pp. 352–369). Hillsdale, NJ: Lawrence Erlbaum.
WilliamsR., & DreherH. (2005). Formative assessment visual feedback in computer graded essays. Journal of Issues in Informing Science and Information Technology, 2, 23–32.
39.
YagelskiR. (1995). The role of classroom context in the revision strategies of student writers. Research in the Teaching of English, 29, 216–338.
40.
YangY.BuckendahlC.JuszkiewiczP., & BholaD. (2002). A review of strategies for validating computer-automated scoring. Applied Measurement in Education, 15(4), 391–412.
