Using Generalizability Theory in Early Childhood Special Education

Abstract

Using a set of hypothetical data, four approaches to the estimation of interrater reliability are illustrated and compared: correlation, comparison of means, percentage of agreement, and generalizability (G) theory techniques. For these data—composed of ratings for 10 subjects on 6 items by 2 raters—the reliability estimates varied widely. The highest estimate was .90 (the simple correlation between the two raters' total scores), and the lowest was a percentage of agreement between the two raters of 26.67% (when “agreement" meant exact match). The G theory techniques allow the researcher to estimate the amount of variance attributable to multiple sources of error within one study. Although more complex than the other methods illustrated here, the G theory procedures yield very beneficial information in many reliability-estimation situations in early childhood special education.

Get full access to this article

View all access options for this article.

References

Brennan, R.L. (1983). Elements of generalizability theory. Iowa City, IA: American College Testing Program.

Brennan, R.L. , & Kane, M. T. (1979). Generalizability theory: A review of basic concepts, issues, and procedures. In R.E. Traub (Ed.), New directions in testing and measurement. San Francisco: Jossey-Bass.

Cronbach, L.J. , Gleser, G.C. , Nanda, H. , & Rajaratnam, N. (1972). The dependability of behavioral measurements. New York: John Wiley.

Fewell, R.R. (1986). The measurement of family functioning. In L. Bickman & D.L. Weatherford (Eds.), Evaluating early intervention programs for severely handicapped children and their families (pp. 263–307). Austin, TX: Pro-Ed.

Glass, G.V. , & Hopkins, K.D. (1984). Statistical methods in education and psychology (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall.

Goodwin, L.D. , & Prescott, P.A. (1981). Issues and approaches to estimating interrater reliability in nursing research. Research in Nursing and Health, 4, 323–337.

Goodwin, L.D. , & Sandall, S.R. (1988). Interrater reliability of parent-infant interaction scales. Diagnostique, 13, 106–119.

Goodwin, W.L. , & Driscoll, L.A. (1980). Handbook for measurement and evaluation in early childhood education. San Francisco: Jossey-Bass.

Hopkins, K.D. , Stanley, J.C. , & Hopkins, B.R. (1990). Educational and psychological measurement and evaluation (7th ed.). Englewood Cliffs, NJ: Prentice-Hall.

10.

Hoyt, C. (1941). Test reliability estimated by analysis of variance. Psychometrika, 6, 153–160.

11.

Johnson, O.G. (1976). Tests and measurements in child development: Handbook II. San Francisco: Jossey-Bass.

12.

Kane, M. T. , & Brennan, R.L. (1977). The generalizability of class means. Review of Educational Research, 47, 267–292.

13.

Kirk, R.E. (1982). Experimental design: Procedures for the behavioral sciences (2nd ed.). Monterey, CA: Brooks/Cole.

14.

Lindquist, E.F. (1953). Design and analysis of experiments in psychology and education. Boston: Houghton Mifflin.

15.

Root, L.S. (1987). Faculty evaluation: Reliability of peer assessments of research, teaching, and service. Research in Higher Education, 26, 71–84.

16.

Shavelson, R.J. , & Webb, N.M. (1981). Generalizability theory: 1973-1980. British Journal of Mathematical and Statistical Psychology, 34, 133–166.

17.

Shavelson, R.J. , Webb, N.M. , & Burstein, L. (1985). Measurement of teaching. In M.C. Wittrock (Ed.), Handbook for research on teaching (3rd ed.) (pp. 50–91). New York: Macmillan.

18.

Stack, J.G. (1984). Interrater reliabilities of the adaptive behavior scale with environmental effects controlled. American Journal of Mental Deficiency, 88, 396–400.

19.

van der Kamp, L.J. T. (1976). Generalizability and educational measurement. In D.N.M. DeGruijter & L.J. T. van der Kamp (Eds.), Advances in psychological and educational measurement (pp. 173–184). New York: John Wiley.

20.

Walker, D.K. (1973). Socioemotional measures for preschool and kindergarten children. San Francisco: Jossey-Bass.

21.

Webb, N.M. , Rowley, G.L. , & Shavelson, R.J. (1988). Using generalizability theory in counseling and development. Measurement and Evaluation in Counseling and Development, 21, 81–90.

22.

Winer, B.J. (1971). Statistical principles in experimental design (2nd ed.). New York: McGraw-Hill.

23.

Zimiles, H. (1986). Rethinking the role of research: New issues and lingering doubts in an era of expanding preschool education. Early Childhood Research Quarterly, 1, 189–206.