Parameter Estimation for Peer Grading under Incomplete Design

Abstract

Methodology developed in the paper supports administration and analysis of peer grading exercises. The technique is especially useful for the incomplete design case, wherein each grader does not grade each paper. The procedure produces simultaneous estimates of (a) bias of each grader, (b) "true" grade for each paper, and (c) relative consistency of each grader. The approach is based on a simple linear statistical model, augmented by a zerosum constraint on grader bias parameters. Ordinary least squares (OLS) estimates are computed and then refined via generalized least squares (GLS) estimation. Case weights for the GLS are based on consistency patterns of the graders, as revealed in the OLS step. The methodology is applied to artificial data to demonstrate its effectiveness. The methodology applies to any situation wherein multiple raters rate multiple ratees on an additive scale.

Get full access to this article

View all access options for this article.

References

Burke, R. J. (1969). Some preliminary data on the use of self-evaluations and peer ratings in assigning university course grades. The Journal of Educational Research, 62, 444-448.

Cox, D. R. and Hinkley, D. V. (1974). Theoretical statistics. London: Chapman & Hall.

Crocker, L. and Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt.

Eisenberg, T. (1965). Are doctoral comprehensive examinations necessary? American Psychologist, 20, 168-169.

Ellman, N. (1975). Peer evaluation and peer grading. English Journal, 64, 79-80.

Guilford, J. P. (1954). Psychometric methods. New York: McGraw-Hill.

Miner, L. E. and McDonald, S. (1981). Reliability of peer reviews. SRA Journal, 12, 21-25.

Neter, J. , Wasserman, W. , and Kutner, M. (1985). Applied linear statistical models (2nd ed.). Homewood, Illinois: Irwin.

Siders, J. A. (1983). Instructor, self, and peer review: A formative evaluation triad. College Student Journal, 17, 141-144.

10.

Siegel, L. (1982). Paired comparison evaluations of managerial effectiveness by peers and supervisors. Personnel Psychology, 35, 843-852.

11.

Werdelin, I. (1966). Teacher ratings, peer ratings, and self-ratings of behavior in school. Educational and Psychological Interactions, No. 11, 49 pages.

12.

Winer, B. J. (1962). Statistical principles in experimental design. New York: McGraw-Hill.