Does Holistic Assessment Predict Writing Performance?

Abstract

This study addressed the question, “How consistently do students perform on holistically scored writing assignments?” Instructors from 13 introductory writing classes at two colleges were asked to provide essay sets written by their students in response to the three to five most important writing assignments in their classes. In all, 796 essays were collected from 241 students. The study drew on a pool of 15 experienced judges to evaluate the essays. Each essay set was scored holistically and independently by 6 of the judges who either ranked or graded the essays in the set. All papers written by a particular student were scored by the same judges. Pairwise correlations of the scores assigned to each essay set were computed for each judge and then averaged across judges. The average of these correlations was 0.16, indicating very low consistency of holistically scored student performance from essay to essay. This result suggests that drawing conclusions from one or even a few writing samples is problematic.

Get full access to this article

View all access options for this article.

References

Barrett, T.J. (1994, November). Generalizability of writing tasks at fourth grade in the Riverside Unified School District. Paper presented at the annual meeting of the California Educational Research Association, San Diego, CA.

Breland, H. M. , Camp, R. , Jones, R. J. , Morris, M. M. , & Rock, D. A. (1987). Assessing writing skill (Research Monograph No. 11). New York: College Entrance Examination Board.

Brennan, L. (1977). Generalizability analysis: Principles and procedures (Technical Bulletin No. 26). Iowa City, IA: ACT.

Calfee, R. , & Perfumo, P. (1992). A survey of portfolio practices. Berkeley: University of California, Berkeley, Center for the Study of Writing.

Crocker, L. , & Algina, J. (1986). Introduction to classical and modern test theory. Fort Worth, TX: Harcourt Brace.

Gardner, H. (1983). Frames of mind: The theory of multiple intelligences. New York: Basic Books.

Godshalk, F. F. , Swineford, F. , & Coffman, W. (1966). The measurement of writing ability (Research Monograph No. 6). New York: College Entrance Examination Board.

Herman, J. L. , Gearhart, M. , & Baker, E. L. (1993). Assessing writing portfolios: Issues in the validity and meaning of scores. Educational Assessment, 1, 201-224.

Lehmann, R. H. (1987, April). Reliability and generalizability of ratings of compositions. Paper presented at the annual meeting of the American Educational Research Association, Washington, DC.

10.

Palmquist, M. , & Young, R. (1992). The notion of giftedness and student expectations about writing. Written Communication, 9, 137-168.

11.

Smith, W. (1992). Teaching experience and placement skill. In J. R. Hayes , R. E. Young , M. L. Matchett , M. McCaffrey , C. Cochran , & T. Hajduk (Eds.), Reading empirical research studies: The rhetoric of research (pp. 287-316). Hillsdale, NJ: Lawrence Erlbaum.

12.

Spearman, C. (1904). General intelligence, objectively determined and measured. American Journal of Psychology, 15, 201-293.

13.

Sternberg, R. J. (1985). Beyond IQ: A triarchic theory of human intelligence. New York: Cambridge University Press.

14.

Thurstone, L. L. (1938). Primary mental abilities (Psychometric Monographs No. 1). Chicago: University of Chicago Press.