Relevant aspects of generalizability theory relating to performance assessments (PAs) are reviewed. The role of various facets in assessing the generalizability of PAs is examined, and some popular estimates of reliabilityfor PAs are considered from the perspective of generalizability theory.
Get full access to this article
View all access options for this article.
References
1.
ACT, Inc. (1994). EXPLORE technical manual. Iowa City IA: Author.
2.
Bennett, R. E. (1999). Using new technologies to improve assessment. Educational Measurement: Issues and Practice, 18 (3), 5–12.
3.
Bennett, R. E. , Morley, M., & Quardt, D. (2000). Three response types for broadening the conception of mathematical problem solving in computerized tests. Applied Psychological Measurement, 24, 294–309.
4.
Brennan, R. L. (1992a). Elements of generalizability theory(rev. ed.). Iowa City IA: American College Testing.
5.
Brennan, R. L. (1992b). Generalizability theory. Educational Measurement: Issues and Practice, 11 (4), 27–34.
6.
Brennan, R. L. (1996). Generalizability of performance assessments. In G. W. Phillips (Ed.), Technical issues in performance assessments. Washington DC: National Center for Education Statistics.
7.
Brennan, R. L. (1997). A perspective on the history of generalizability theory. Educational Measurement: Issues and Practice, 16 (4), 14–20.
8.
Brennan, R. L. (1998). Raw-score conditional standard errors of measurement in generalizability theory. Applied Psychological Measurement, 22, 301–331.
9.
Brennan, R. L. (2000). (Mis)conceptions about generalizability theory. Educational Measurement: Issues and Practice, 19 (1), 5–10.
10.
Brennan, R. L. (in press). Generalizability theory. New York: Springer-Verlag.
11.
Brennan, R. L. , Gao, X., & Colton, D. A. (1995). Generalizability analyses ofWork Keys Listening and Writing tests. Educational and Psychological Measurement, 55, 157–176.
12.
Brennan, R. L. , & Johnson, E. G. (1995). Generalizability of performance assessments. Educational Measurement: Issues and Practice, 14 (4), 9–12.
13.
Clauser, B. E. , Swanson, D. B., & Clyman, S. G. (1999). A comparison of the generalizability of scores produced by expert raters and automated scoring systems. Applied Measurement in Education, 12, 281–299.
14.
Cronbach, L. J. , Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizabilityfor scores and profiles. New York: Wiley.
15.
Cronbach, L. J. , Linn, R. L., Brennan, R. L., & Haertel, E. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373–399.
16.
Dunbar, S. B. , Koretz, D. M.,& Hoover, H. D. (1991). Quality control in the development and use of performance assessments. Applied Measurement in Education, 4, 289–303.
17.
Feldt, L. S. , & Brennan, R. (1989). Reliability. In R.L. Linn (Ed.), Educational measurement(3rd ed., pp. 105–146). New York: Macmillan.
18.
Gao, X. , Shavelson, R. J., & Baxter, G. P. (1994). Generalizability of large-scale performance assessments in science: Promises and problems. Applied Measurement in Education, 7, 323–343.
19.
Gulliksen, H. (1950). Theory of mental tests. New York: Wiley.
20.
Hoover, H. D. , & Bray, G. B. (1995, April). The research and development phase: Can a performance assessment be cost-effective? Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco.
21.
Kane, M. T. (1982). A sampling model for validity. Applied Psychological Measurement, 6, 125–160.
22.
Kane, M. T. , Crooks, T.,& Cohen, A. (1999). Validating measures of performance. Educational Measurement: Issues and Practice, 18 (2), 5–17.
23.
Koretz, D. , Klein, S., McCaffrey, D., & Stecher, B. (1994). The Vermont portfolio assessment program: Findings and implications. Educational Measurement: Issues and Practice, 13 (3), 5–16.
24.
Lane, S. , Liu, M., Ankenmann, R. D., & Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33, 71–92.
25.
Linn, R. L. (1993). Educational assessment: Expanded expectations and challenges. Educational Evaluation and Policy Analyses, 15, 1–16.
26.
Linn, R. L. , & Burton, E. (1994). Performance-based assessment: Implications of task specificity. Educational Measurement: Issues and Practice, 13 (1), 5–15.
27.
Ruiz-Primo, M. A. , Baxter, G. P., & Shavelson, R. J. (1993). On the stability of performance assessments. Journal of Educational Measurement, 30, 41–53.
28.
Shavelson, R. J. , Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30, 215–232.
29.
Shavelson, R. J. , Gao, X., & Baxter, G. (1996). On the content validity of performance assessments: Centrality of domain specification. In M. Birenbaum & F. J. R. C. Dochy (Eds.), Alternatives in assessment of achievements, learning processes and prior knowledge(pp. 131–141). Boston: Kluwer.
30.
Shavelson, R. J. , & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park CA: Sage.
31.
Starch, D. , & Elliot, E. C. (1912). Reliability of grading high school work in English. School Review, 20, 442–457.
32.
Starch, D. , & Elliot, E. C. (1913). Reliability of grading high school work in mathematics. School Review, 21, 254–259.
33.
van der Vleuten, C. P. M. , & Swanson, D. B. (1990). Assessment of clinical skills with standardized patients: The state of the art. Teaching and Learning in Medicine, 2, 58–76.
34.
Wainer, H. (1993). Measurement problems. Journal of Educational Measurement, 30, 1–21.
35.
Webb, N. M. , Schlackman, J., & Sugrue, B. (2000). The dependability and interchangeability of assessment methods in science. Applied Measurement in Education, 13, 277–301.
36.
Welch, C. (1991, April). Estimating the reliability of a direct measure of writing through generalizability theory. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago.
