This study employed an automated grader to evaluate essays, both holistically and with the rating of traits (content, organization, style, mechanics, and creativity) for Webbased student essays serving as placement tests at a large Midwestern university. The authors report the results of two combined experiments, based on random selection from 1,193 essays. In the first experiment, the essays of 807 students were used to create statistical predictions for the essay-grading software. In the second experiment, the ratings from a separate, random sample of 386 essays were used to compare the ratings of six human judges against those generated by the computer. The interjudge correlation of the human raters alone was r = .71. But the interrater reliability of all six judges in combination with computer scoring reached .83. The essay-grading software was an efficient means for evaluating the essays, with a capacity for grading approximately six documents every second. Other potential feedback measures for use in writing courses are also discussed.
Get full access to this article
View all access options for this article.
References
1.
Arbuckle, J. L. , & Wothke, W. (1999). Amos 4.0 user’s guide. Chicago: SmallWaters.
2.
Bennett, R. E. , & Bejar, I. I. (1998). Validity and automated scoring: It’s not only the scoring. Educational Measurement: Issues and Practice, 17(4), 9-17.
3.
Burstein, J. , Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., Kukich, K., et al. (1998). Computer analysis of essay content for automated score prediction: A prototype automated scoring system for GMAT analytical writing assessment essays (RR-98-15). Princeton, NJ: Educational Testing Service.
4.
Hamp-Lyons, L. (1995). Rating nonnative writing: The trouble with holistic scoring. TESOL Quarterly, 29, 759-762.
5.
Harrington, S. , Shermis, M. D., & Rollins, A. (2000). The influence of processing on English placement test results. Computers and Composition, 17, 197-210.
6.
Keith, T. Z. (1998, April). Construct validity of PEG. Paper presented at the American Educational Research Association, San Diego, CA.
7.
Landauer, T. , Laham, D., & Foltz, P. (1998, April). The Goldilocks principle for vocabulary acquisition and learning: Latent semantic analysis theory and applications. Paper presented at the American Educational Research Association, San Diego, CA.
8.
McCollum, K. (1999, January 29). Computers will help grade essays on Graduate Management Admission Test. Chronicle of Higher Education, 44, p. A30-A30.
9.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-104). New York: Macmillan.
10.
Mzumara, H. R. , Shermis, M. D., & Fogel, M. (1998). Validity of the IUPUI placement test scores for course placement: 1997-1998. Indianapolis: Indiana University Purdue University Indianapolis Testing Center.
11.
Page, E. B. (1994). Computer grading of student prose, using modern concepts and software. Journal of Experimental Education, 62, 127-142.
12.
Page, E. B. , Lavoie, M. J., & Keith, T. Z. (1996, April). Computer grading of essay traits in student writing. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.
13.
Page, E. B. , & Petersen, N. S. (1995). The computer moves into essay grading: Updating the ancient test. Phi Delta Kappan, 76, 561-565.
14.
Page, E. B. , Poggio, J. P., & Keith, T. Z. (1997, March). Computer analysis of student essays: Finding trait differences in the student profile. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
15.
Perkins, K. (1983). On the use of composition scoring techniques, objective measures, and objective tests to evaluate ESL writing ability. TESOL Quarterly, 17, 651-671.
16.
Petersen, N. S. , & Page, E. B. (1997, March). New developments in Project Essay Grade: Second ETS blind test with GRE essays. Paper presented at the American Educational Research Association, Chicago.
17.
Roid, G. H. (1994). Patterns of writing skills derived from cluster analysis of direct-writing assessments. Applied Measurement in Education, 7, 159-170.
18.
Shermis, M. D. (1997, March). Recent developments in college placement testing: Assessments via the World Wide Web. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
19.
Shermis, M. D. , Mzumara, H. R., Lillig, C., & Brown, M. (1997, August). Computerized adaptive testing through the World Wide Web. Paper presented at the annual meeting of the American Psychological Association, Chicago.
20.
Shermis, M. D. , Mzumara, H. R., Olson, J., & Harrington, S. (2001). On-line grading of student essays: PEG goes on the Web at IUPUI. Assessment and Evaluation in Higher Education, 26, 247-259.
21.
Smith, W. (1993). Assessing the reliability and adequacy of using holistic scoring of essays as a college composition placement technique. In M. M. Williamson & B. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 142-205). Cresskill, NJ: Hampton.
22.
Stiggings, R. J. , & Bridgeford, N. J. (1983). An analysis of published tests of writing proficiency. Educational and Psychological Measurement, 2, 6-10, 26-26.
23.
Vantage Technologies, Inc. (1998). The efficacy of IntellimetricTM and traditional scoring for secondary-level literary analysis essays (RB-304). Yardley, PA: Author.
24.
White, E. (1995). Teaching and assessing writing (2nd ed.). San Francisco: Jossey-Bass.
