This article examines the behavioral differences of essay scorers who demonstrate different levels of proficiency for a psychometric scoring task. The authors compare three proficiency groups to identify differences in (a) essay features they consider, (b) their understanding of the scoring rubric, and (c) their decision-making procedures. Results indicate scorers with different levels of proficiency do not focus on different essay features when making evaluative decisions but their understandings of the scoring criteria may vary. Proficient scores are more likely to focus on general features of an essay when making evaluative decisions and to adopt values espoused by the scoring rubric than are less proficient scorers. Also, proficient scorers make evaluations by reading the entire essay and then reviewing its content, whereas less proficient scorers may interrupt the reading process to monitor how well the essay satisfies the scoring criteria. Finally, the authors discuss implications for scorer selection and training.
Get full access to this article
View all access options for this article.
References
1.
American College Testing Program
. (1994). Local scoring guide: 10th Grade Writing Assessment. Iowa City, IA: Author.
2.
Barritt, L.
, Stock, P. L., & Clark, F. (1986). Researching practice: Evaluating assessment essays. College Composition and Communications, 37, 315-327.
3.
Breland, H. M.
(1983). The direct assessment of writing skill: A measurement review (College Board Report No. 83-6). New York: College Entrance Examination Board.
4.
Breland, H. M.
, & Jones, R. J. (1984). Perceptions of writing skills. Written Communication, 1, 101-119.
5.
Brennan, R. L.
(1992). Elements of generalizability theory (2nd ed.). Iowa City, IA: ACT.
6.
Charney, D.
(1984). The validity of using holistic scoring to evaluate writing: A critical overview. Research in the Teaching of English, 18, 65-81.
7.
Cohen, J.
(1988). Statistical power analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8.
Diederich, P.
, French, J. W., & Carlton, S. (1961). Factors in judgments of writing ability (RB61-15). Princeton, NJ: Educational Testing Service.
9.
Engelhard, G. J.
(1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
10.
Ericsson, K. A.
, & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge, MA: MIT Press.
11.
Fowles, M. E.
(1978). Basic skills assessment: Manual for scoring the writing sample. Analytic scoring/holistic scoring. Princeton, NJ: Educational Testing Service.
12.
Frederiksen, J. R.
(1992, April). Learning to “see:” Scoring video portfolios or “beyond the hunter-gatherer” in performance assessment. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
13.
Frederiksen, J. R.
, Sipusic, M., Gamoran, M., & Wolfe, E. W. (1992). Video portfolio assessment: A study for the National Board for Professional Teaching Standards. Oakland, CA: Cognitive Science Research Center, Educational Testing Service.
14.
Freedman, S. W.
, & Calfee, R. C. (1983). Holistic assessment of writing: Experimental design and cognitive theory. In P. Mosenthal, L. Tamor, & S. A. Walmsley (Eds.), Research on writing: Principles and methods (pp. 75-98). New York: Longman.
15.
Glaser, R.
, & Chi, M.T.H. (1988). Overview. In M.T.H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise (pp. xv-xxviii). Hillsdale, NJ: Lawrence Erlbaum.
16.
Holm, S.
(1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65-70.
17.
Huot, B. A.
(1993). The influence of holistic scoring procedures on reading and rating student essays. In M. M. Williamson & B. A. Huot (Eds.), Validating holistic scoring for writing assessment (pp. 206-236). Cresskill, NJ: Hampton.
18.
Kirk, R. E.
(1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). New York: Brroks/Cole.
19.
Lane, S.
, Liu, M., Ankenmann, R. D., & Stone, C. A. (1996). Generalizability and validity of mathematics performance assessment. Journal of Educational Measurement, 33, 71-92.
20.
Liebetrau, A. M.
(1983). Measures of association. Newbury Park, CA: Sage.
21.
Linacre, J. M.
(1989). Many-facet Rasch measurement. Chicago, IL: MESA.
22.
Lunz, M. E.
, Wright, B. D., & Linacre, J. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3, 331-345.
23.
Moss, P. A.
(1994). Can there be validity without reliability?Educational Researcher, 23(2), 5-12.
24.
Pula, J. J.
, & Huot, B. A. (1993). A model of background influences on holistic raters. In M. M. Williamson & B. A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 237-265). Cresskill, NJ: Hampton.
25.
Shavelson, R. J.
, Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30, 215-232.
26.
Shrout, P. E.
, & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
27.
Vaughan, C.
(1991). Holistic assessment: What goes on in the rater's mind? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 111-125). Norwood, NJ: Ablex.
28.
Voss, J. F.
& Post, T. A. (1988). On the solving of ill-structured problems. In M.T.H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise (pp. 261-285). Hillsdale, NJ: Lawrence Erlbaum.
29.
White, E. M.
(1985). Teaching and assessing writing. San Francisco, CA: Jossey Bass.
30.
Wolfe, E. W. (1995). A study of expertise in essay scoring. Unpublished doctoral dissertation, University of California, Berkeley.
31.
Wolfe, E. W.
(1997). The relationship between essay reading style and scoring proficiency in a psychometric scoring system. Assessing Writing, 4, 83-106.
32.
Wolfe, E. W.
, & Feltovich, B. (1994, April). Learning how to rate essays: A study of scorer cognition. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
33.
Wolfe, E. W.
, Gitomer, D., & Carter, R. (1998, April). The influence of changes in assessment design on the psychometric qualities of scores. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
