Evidence of the uncertainty attached to school and individual scores is required to avoid over interpretation of results. Generalizability analysis provides in the standard error a suitable indicator of uncertainty. Assessments depart from traditional measurements in ways that require extensions and re interpretatons of generalizability analysis. The authors find, for example, that in many analyses the variance component interpreted as Pupil x Task interaction actually arises in part from instability in pupil performance. It is necessary to recognize in the school-level analysis that pupils are nested in classes and schools. Whether to treat the population of a school's pupils as infinite or as limited to the student body assessed also requires careful consideration.
Get full access to this article
View all access options for this article.
References
1.
American Educational Research Association, American Psychological Association, and National Council on Measurement in Education . (1985). Standanrds for educational and psychological testing. Washington, DC: Author.
2.
Brennan, R. L. (1984). Some statistical issues in generalizability theory (ACT Technical Bulletin No. 46). Iowa City, IA: ACT.
3.
Brennan, R. L. (1992a). Elements of generalizability theory (2nd ed.). Iowa City, IA: ACT.
4.
Brennan, R. L. (1992b). Generalizability theory. Educational Measurement: Issues and Practice, 11(4), 27-34.
5.
Brennan, R. L. (1996a). Conditional standard errors of measurement in generalizability theory (ITP Occasional Paper No. 40). Iowa City: Iowa Testing Programs, University of Iowa.
6.
Brennan, R. L. (1996b). Generalizability of performance assessments. In G. W. Phillips (Ed.), Technical issues in performance assessments (pp. 19-58). Washington, DC: National Center for Education Statistics.
7.
Brennan, R. L. , Harris, D. J., & Hanson, B. A. (1987). 7he bootstrap and other procedures for examining the variability of estimated variance components in testing contexts (ACT Research Report Series No. 87-7). Iowa City, IA: ACT.
8.
Cronbach, L. J. (1995). A valedictory: Reflections on 60 years in educational testing. Board Bulletin. Board on Testing and Assessment, National Research Council, Washington, DC. (Available through National Technical Information Service)
9.
Cronbach, L. J. , Gleser, G. C., Nanda, H., & Rajaratnan, N. (1972). The dependability of behavioral measurements. New York: John Wlley.
10.
Crone, L. J. , Long, M. H., Frankin, B. J., & Holbrook, A. M. (1994). Composite versus component scores: Consistency of school effectiveness classification. Applied Measurement in Education, 7, 303-321. CRONBACH et al.
11.
CTB/McGraw-Hill. (1994). California Learning Assessment data. [Unpublished analysis.]
12.
Gao, X. , Shavelson, R., & Baxter, G. P. (1994). Generalizability of large-scale performance assessments in science: Promises and problems. Applied Measurement in Education, 7, 323-334.
13.
Kolen, M. J. , Hanson, B. A., & Brennan, R. L. (1992). Conditional standard errors of measurement. Journal of Educational Measurement, 29, 285-307.
14.
Koretz, D. , Stecher, B., Klein, S., & McCaffrey, D. (1994). The Vennont Portfolio Assessment Program: Findings and implications. Educational Measurement: Issues and Practice, 13(3), 5-16.
15.
Linn, R. L. , Burton, E., DeStefano, L., & Hanson, M. (1996). Generalizability of New Standards Project 1993 pilot study tasks in mathematics. Applied Measurement in Education, 9, 201-214.
16.
Livingston, S. , & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179-198.
17.
Mislevy, R. J. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439-483.
18.
Searle, S. R. , Casella, G., & McCulloch, C. E. (1992). Variance components. New York: John Wiley.
19.
Select Committee . (1994). Sampling and statistical procedures used in the California Learning Assessment System. Sacramento: California State Department of Education. (Reprinted in Cronbach, 1995)
20.
Shavelson, R. J. (1991). Generalizability theory and military performance measurements: I. Individual performance. In A. K. Wlgdor & B. F. Green, Jr. (Eds.), Performance assessment in the workplace: Vol. 2. Technical issues (pp. 207-257). Washington, DC: National Academy Press.
21.
Shavelson, R. J. , & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
