This paper summarizes an experiment designed to assess the validity of essay grading between holistic and analytic human graders and a computerized grader based on latent semantic analysis. The validity of the grade was gauged by the extent to which the student's knowledge of the topic correlated with the grader's expert knowledge. To assess knowledge, Pathfinder networks were generated by the student essay writers, the holistic and analytic graders, and the computerized grader. It was found that the computer generated grades more closely matched the definition of valid grading than did human generated grades.
Get full access to this article
View all access options for this article.
References
1.
BergerC.DershimerC. (1993). Using Technology to Measure Change in Students' Science Learning. Paper presented at the meeting of the National Association for Research in Science Teaching, Atlanta, GA.
2.
CharneyD. (1984). The validity of using holistic scoring to evaluate writing: A critical overviewResearch in the Teaching of English, 18, 65–81.
3.
CoffmanW.E. (1971). Essay examinations In ThorndikeR.L. (Ed.). Educational measurement (2nd ed., pp. 271–302). Washington, DC: American Council on Education.
4.
GoldsmithT.E.DavenportD.E. (1990). Assessing structural similarity of graphs In SchvaneveldtR.W. (Ed.). Pathfinder associative networks: Studies in knowledge organization (pp. 75–87). Norwood, NJ: Ablex Publishing Corporation.
5.
DiederichP.B.FrenchJ.W.CarltonS.T. (1961). Factors in judgements of writing ability (Research Bulletin RB 61-15). Princeton, NJ: Educational Testing Services.
6.
GoldsmithT.E.JohnsonP.J.ActonW.H. (1991). Assessing structural knowledgeJournal of Educational Psychology, 83, 88–96.
7.
GomezR.L.HadfieldO.D.HousnerL.D. (1996). Conceptual maps and simulated teaching episodes as indicators of competence in teaching elementary educationJournal of Educational Psychology, 88, 572–585.
8.
GonzalvoP.CañasJ.J.BajoM.T. (1994). Structural representations in knowledge acquisitionJournal of Educational Psychology, 86, 601–616.
9.
HuotB.A. (1990). Reliability, validity, and holistic scoring: What we know and what we need to knowCollege Composition and Communication, 41, 201–213.
10.
MadiganR.J.BrosamerJ.J. (1991). Holistic grading of written work in introductory psychology: Reliability, validity, and efficiencyTeaching of Psychology, 18, 91–94.
11.
JonassenD.H.BeissnerK.YacciM. (1993). Structural knowledge: Techniques for representing, conveying, and acquiring structural knowledge.Hillsdale, NJ: Erlbaum.
12.
KintschW. (1998). Comprehension. A paradigm for cognition.Cambridge, UK: Cambridge University Press.
13.
LandauerT.K.FoltzP.W.LahamD. (1998). An introduction to latent semantic analysisDiscourse Processes, 25, 259–284.
14.
McGaghieW.C. (1996). Comparison of knowledge structures with the Pathfinder scaling algorithm. Paper presented at the meeting of the American Educational Research Association, New York, NY. (ERIC Document Reproduction Service No. ED 401 282).
15.
SchvaneveldtR.W.DursoF.T.DearholtD.W. (1989). Network structures in proximity data In BowerG. (Ed.). The psychology of learning and motivation: Advances in research and theory (Vol. 24, pp. 249–284). New York: Academic Press.
16.
WhiteE.M. (1984). HolisticisimCollege Composition and Communication, 35, 400–409.
17.
WilsonJ.M. (1994). Network representations of knowledge about chemical equilibrium: Variations with achievementJournal of Research in Science Teaching, 31, 1133–1147.
