Although student evaluations of instruction have been widely used in decisions that affect the welfare of faculty members for over a decade, questions about the dependability of such data are still apropos. This paper analyzes the generalizability of student evaluations of college instruction and applies generalizability theory data collected in an all too common design. The results suggest that the naive use of student evaluations as a measure of teacher effectiveness may lead to decisions that are based on data that are not very dependable.
Get full access to this article
View all access options for this article.
References
1.
CrawfordD. L.BradshawH. L.Perception of characteristics of effective university teachers: A scaling analysis, Educational and Psychological Measurement, 1968, 28, 1079–1085.
2.
CronbachL. J.GleserG. C.NandaH.RajaratnamN.The dependability of behavioral measurements: Theory of generability for scores and profiles. New York: Wiley & Sons, 1972.
3.
HaggardE. E.Intraclass correlation and the analysis of variance. New York: The Dryden Press, Inc., 1958.
4.
HildebrandM. (and others). Evaluating university teaching. Berkeley: Center for Research and Development in Higher Education, 1971.
5.
HillB. M.Inferences about variance components in the one-way model. Journal of the American Statistical Association, 1965, 60, 806–825.
6.
HoytC.Test reliability estimated by analysis of variance. Psychometrika, 1941, 6, 153–160.
7.
KaneM. T.BrennanR. L.The generalizability of class means. Review of Educational Research, 1977, 47, 267–292.
8.
KirkR. E.Experimental design: Procedures for the behavioral sciences. Belmont: Brooks/Cole Publishing Co., 1968.
9.
KruderG. F.RichardsonM. W.The theory of estimation of test reliability. Psychometrika, 1937, 2, 151–160.
10.
LiC. C.Introduction to experimental statistics. New York: McGraw-Hill Book Company, 1964.
11.
LindmanH. R.Analysis of variance in complex experimental designs. San Francisco: W. H. Freeman and Company, 1974.
12.
LindquistE. F.Design and analysis of experiments in psychology and education. Boston: Houghtin-Mifflin, 1953.
13.
LordF. M.Estimating test reliability. Educational and Psychological Measurement, 1955, 15, 324–336.
14.
LordF. M.NovickM.Statistical theories for mental test scores. Reading, Mass.: Addison-Wesley Publishing Company, 1968.
15.
MillerR. L.Developing programs for faculty evaluation: A source book for higher education. San Francisco: Josey-Bass Inc., Publishers, 1974.
16.
PohlmannJ. T.A description of teaching effectiveness as measured by student ratings. Journal of Educational Measurement, 1975, 12, 49–53.
17.
RosenshineB.The stability of teacher effects upon student achievement. Review of Educational Research, 1970, 40, 647–662.
18.
SockloffA. L.PapacostasA. C.Uniformity of faculty attitude toward effective teaching in lecture/discussion courses. Journal of Educational Measurement, 1975, 12, 281–293.
19.
ShapiroP. P.Approaches to evaluation of student teachers. Unpublished Paper delivered to Canadian Teachers' Federation, Conference on Evaluation of Student Teachers. Toronto, January 26–28, 1977.
WilsonR. C.DienstE. R.WatsonN. L.Characteristics of effective college teachers as perceived by their colleagues. Journal of Educational Measurement, 1973, 10, 31–37.
22.
WinerB. J.Statistical principles in experimental design. Second Edition, New York: McGraw-Hill Book Company, 1971.
