The Application of Generalizability Theory to the Reliability of Ratings

Abstract

Dutch

Rating procedures typically require several raters to rate individuals on several items so that the consistency of two facets (raters and items), each of which has more than two levels, need to be examined. Classical test theory is inadequate to describe reliability in this context because the presence of errors of severity, leniency and central tendency violates the assumption of parallelism. Cronbach's generalizability theory is eminently suited to the reliability estimation of ratings because it does not require this assumption. Apart from yielding intraclass correlations as generalizability coefficients, it provides for the separate estimation of the variability of raters and items and presents explicit guidelines for deciding whether generalizabililty should be raised by increasing raters or items. These theoretical implications are demonstrated in terms of a numerical example published in the South African literature.

Get full access to this article

View all access options for this article.

References

Allal

(1986). Extensions of generalizability theory through the principle of symmetry. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, USA.

Cronbach

L.J.

Gleser

G.C.

Nanda

Rajaratnam

(1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.

Cronbach

L.J.

Rajaratnam

Gleser

G.C.

(1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137–163.

Ebel

R.L.

(1951). Estimation of the reliability of ratings. Psychometrika, 6, 407–421.

Guilford

J.P.

(1954). Psychometric methods. New York: Wiley.

Hoyt

(1941). Test reliability obtained by analysis of variance. Psychometrika, 6, 153–160.

Huysamen

G.K.

(1989). Psychological and educational test theory. Bloemfontein: Author.

Jackson

R.W.B.

Ferguson

G.A.

(1941). Studies on the reliability of tests. Toronto: University of Toronto.

Johnson

(1986). Generalizability analysis of British performance data: Implications for general assessment practice. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, USA.

10.

Jones

L.V.

Appelbaum

M.I.

(1989). Psychometric methods. Annual Review of Psychology, 40, 23–43.

11.

Keppel

(1982). Design and analysis: A researcher's handbook (3rd ed.). Englewood-Cliffs, New Jersey: Prentice-Hall.

12.

Kerlinger

F.N.

(1986). Foundations of behavioral research (3rd ed.). New York: Holt, Rinehart & Winston.

13.

Kirk

R.E.

(1968). Experimental design: Procedures for the behavioral sciences. Belmont, California: Brooks/Cole.

14.

Murphy

K.R.

Balzer

W.K.

(1989). Rater errors and rating accuracy. Journal of Applied Psychology, 74, 619–624.

15.

Rowley

(1986). Application of generalizability theory to observational studies: Limitations. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, USA.

16.

Saal

F.E.

Downey

R.G.

Lahey

M.A.

(1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413–428.

17.

Serfontein

(1972). Objektivering van personeelevaluasie in die SAW (Making personnel evaluation in the SADF objective), South African Psychologist, 2(1), 30–47.

18.

Shavelson

R.L.

Webb

N.M.

(1981). Generalizability theory: 1973–1980. British Journal of Mathematical and Statistical Psychology, 34, 133–166.

19.

Van Staden

Visser

J.D.

(in press). Analysis of themes and statistical techniques: A review of the past decade of the South African Journal of Psychology. South African Journal of Psychology.

20.

Webb

N.M.

(1987). A generalizability study of job performance measurements in the Navy. Paper presented at the Annual Meeting of the American Educational Research Association, Washington, D.C., USA.

21.

Wittman

(1986). The synthesis of Cattell's BDRM, Cronbach et al's generalizability theory and Brunswik's Lens Model: A framework for improving construct and predictive validity. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, USA.

22.

Woodward

J.A.

Joe

G.W.

(1973). Maximizing the coefficient of generalizability in multifacet decision studies. Psychometrika, 38, 173–181.