This study examined the percentage of task-sampling variability in performance assessment via a meta-analysis. In total, 50 studies containing 130 independent data sets were analyzed. Overall results indicate that the percentage of variance for (a) differential difficulty of task was roughly 12% and (b) examinee’s differential performance of the same task was approximately 26%. Based on moderator analysis, research design and subject area were significant predictors of variance component for tasks (σ2t) and variance component for person—task interaction (σ2pt). Mathematics and listening were associated with relatively high proportion of task-sampling variability whereas foreign language was least affected by this variability. To reduce task variation, some techniques are recommended: (a) incorporating more facets when possible, (b) using a crossed rather than a nested design, and (c) incorporating occasion as a facet.
Adams, D.C., Gurevitch, J., & Rosenberg, M.S. (1997). Resampling tests for meta-analysis of ecological data. Ecology, 78, 1277-1283.
2.
*Bachman, L.F., Lynch, B.K., & Mason, M. ( 1993, August 2-4). Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Paper presented at the 15th Annual Language Testing Research Colloquium , Cambridge, UK. (ERIC Document Reproduction Service No. 368154)
3.
*Baker, E.L., Abedi, J., Linn, R.L., & Niemi, D. ( 1995). Dimensionality and generalizability of domain-independent performance assessments. Journal of Educational Research, 89, 197-205.
4.
*Barrett, T.J. ( 1994, November 17-18). Generalizability of writing tasks at fourth grade in the Riverside Unified School District. Paper presented at the 73rd Annual Meeting of the California Educational Research Association, San Diego, CA. (ERIC Document Reproduction Service No. 377498)
5.
*Baxter, G.P., Shavelson, R.J., Herman, S.J., Brown, K.A., & Valadez, J.R. ( 1993). Mathematics performance assessment: Technical quality and diverse student impact. Journal of Research in Mathematics Education, 24, 190-216.
6.
*Bennett, R.E., & Rock, D.L. ( 1995). Generalizability, validity, and examinee perceptions of a computer-delivered Formulating Hypotheses Test. Journal of Educational Measurement, 32, 19-36.
7.
*Boodoo, G.M., & Garlinghouse, P. (1983). Use of the essay examination to investigate the writing skills of undergraduate education majors. Educational and Psychological Measurement, 43, 1005-1014.
8.
*Boulet, J.R., Rebbecchi, T.A., Denton, E.C., McKinley, D.W., & Whelan, G.P. ( 2004). Assessing the written communication skills of medical school graduates. Advances in Health Sciences Education, 9, 47-60.
9.
*Brennan, R.L., Gao, X., & Colton, D. ( 1995). Generalizability analysis of work keys listening and writing. Educational and Psychological Measurement, 55, 157-176.
10.
Brennan, R.L., & Johnson, E.G. ( 1995). Generalizability of performance assessments. Educational Measurement: Issues and Practices, 14(4), 9-12.
11.
*Bunch, M.B., & Littlefair, W. (1988, June). Total score reliability in large-scale writing assessment. Paper presented at the Conference of the Education Commission of the States/ Colorado Department of Education Assessment, Boulder, CO. (ERIC Document Reproduction Service No. 310149)
12.
*Chiu, C.W.T., & Wolfe, E.W. ( 1997, March 24-28). Generalizability theory: A new approach to analyze non-crossed performance assessment data. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago. (ERIC Document Reproduction Service No. 408326)
13.
*Clauser, B.E., Harik, P., & Clyman, S.G. ( 2000). The generalizability of scores for a performance assessment scored with a computer-automated scoring system. Journal of Educational Measurement, 37, 245-261.
14.
*Clauser, B.E., Harik, P., & Margolis, M.J. (2006). A multivariate analysis of assessment of physicians’ clinical skills. Journal of Educational Measurement, 43, 173-191.
15.
*Clauser, B.E., Margolis, M.J., & Swanson, D.B. ( 2002). An examination of the contribution of computer-based case simulations to the USMLE Step 3 examination. Academic Medicine, 77, S80-S82.
16.
*Clauser, B.E., Margolis, M.J., Swanson, D.B., & Boulet, J. ( 2003). A multivariate generalizability analysis of data from a performance assessment of physicians’ clinical skills. Paper presented at 2003 Annual Meeting of National Council on Measurement in Education, Chicago.
17.
Conger, A.J. ( 1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88, 322-328.
18.
Crehan, K.D. ( 1997, October). A discussion of analytic scoring for writing performance assessment. Paper presented at the Annual Meeting of the Arizona Educational Research Association, Phoenix, AZ. (ERIC Document Reproduction Service No. 414336)
19.
Cronbach, L.J., Linn, R.L., Brennan, R.L., & Haertel, E.H. ( 1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373-399.
20.
*Gao, X., & Brennan, R.L. ( 2001). Variability of estimated variance components and related statistics in a performance assessment. Applied Measurement in Education, 14, 191-203.
21.
*Gao, X., Shavelson, R.J., & Baxter, G.P. ( 1994). Generalizability of large-scale performance assessments in science: Promises and problems. Applied Measurement in Education, 7, 323-342.
22.
Hoyt, W.T., & Kerns, M.-D. ( 1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403-424.
23.
Hunter, J.E., & Schmidt, F.L. ( 1990). Methods of meta-analysis: correcting error and bias in research findings. Newbury Park, CA: Sage.
24.
*Kim, S.C. ( 2000, April 24-28). Investigating the generalizability of scores from different rating systems in performance assessment. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA. (ERIC Document Reproduction Service No. 449209)
25.
*Kim, Y. ( 1993). The use of a 2-parameter graded IRT model and generalizability theory in the development and validation of a mathematics performance test. Unpublished doctoral dissertation, University of Pittsburgh , PA.
26.
Klein, S.P., Stecher, B.M., Shavelson R.J., McCaffrey, D., Ormseth, T., Bell, R.M., et al. (1998). Analytic versus holistic scoring of science performance tasks. Applied Measurement in Education, 11, 121-137.
27.
*Koretz, D., Stecher, B., Klein, S., McCaffrey, D., & Deibert, E. ( 1992). Can portfolios assess student performance and influence instruction? The 1991-92 Vermont experience. Los Angeles : National Center for Research on Evaluation, Standards, and Student Testing. (ERIC Document Reproduction Service No. 365699)
28.
*Kozaki, Y. ( 2004). Using GENOVA and FACETS to set multiple standards on performance assessment for certification in medical translation from Japanese into English. Language Testing, 21, 1-27.
29.
*Kraiger, K., & Teachout, M.S. (1991). Application of generalizability theory to the Air Force job performance measurement project: A summary of research results (AFHRL-TR-90-92). Brooks Air Force Base, TX: Air Force Systems Command.
30.
*Kreiter, C.D., Gordon, J.A., Elloit, S.T., & Ferguson, K.J. (1999). A prelude to modeling the expert: A generalizability study of expert ratings of performance on computerized clinical simulations. Advances in Health Sciences Education , 4, 261-270.
31.
*Lane, S., Liu, M., Ankenmann, R.D., & Stone, C.A. ( 1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33, 71-92.
32.
*Lane, S., Stone, C.A., Ankenmann, R.D., & Liu, M. ( 1994). Reliability and validity of a mathematics performance assessment. International Journal of Education Research, 21, 247-262.
33.
*Lee, Y.W., Golub-Smith, M., Payton, C., & Carey, J. ( 2001, April 10-14). The score reliability of the test of spoken English (TSE) from the generalizability theory perspective: Validating the current procedure. Paper presented at the Annual Meeting of the American Educational Research Association, Seattle, WA. (ERIC Document Reproduction Service No. 458241)
34.
*Lee, Y., Kantor, R., & Mollaun, P. ( 2002, April 9-13). Score reliability as an essential prerequisite for validating new writing and speaking tasks for TOEFL. Paper presented at the Annual Meeting of Teachers of English to Speakers of Other Languages, Salt Lake City, UT. (ERIC Document Reproduction Service No. 464964)
35.
*Lehmann, R.H. ( 1990). Reliability and generalizability of ratings of compositions . Studies in Educational Evaluation, 16, 501-512.
36.
*Linn, R.L. ( 1994). Generalizability of New Standards Project, 1993 pilot study tasks in mathematics. Technical issues in procedures to link state results to a common national standard. Project 2.4 quantitative models to monitor the status and progress of learning and performance and their antecedents . Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing. (ERIC Document Reproduction Service No. ED379356)
37.
Linn, R.L., & Burton, E. ( 1994). Performance-based assessment: Implications of task specificity . Educational Measurement: Issues and Practices, 13(1), 5-8.
38.
*Lunz, M.E., & Schumacker, R.E. (1997). Scoring and analysis of performance examinations: A comparison of methods and interpretations. Journal of Outcome Measurement, 1, 219-238.
39.
*Lynch, B.K., & McNamara, T.F. (1998). Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing, 15, 158-180.
40.
*Mayberry, P.W. ( 1986). From hands-on measurement to job performance: The issue of generalizability (Research Memorandum CRM 86-214). Alexandria, VA: Center for Naval Analyses.
41.
*Mayberry, P.W., & Wright, W.H. ( 1992). Reliability of mechanical maintenance performance measures (Research Memorandum CRM 91-216). Alexandria, VA: Center for Naval Analyses.
42.
*McBee, M.M., & Barnes, L.L.B. (1998). The generalizability of a performance assessment measuring achievement in eighth grade mathematics . Applied Measurement in Education, 11, 179-194.
43.
*Miller, M.D. ( 2002). Generalizability of performance-based assessments. Technical guidelines for performance assessment. Washington, DC: Council of Chief State School Officers. (ERIC Document Reproduction Service No. 471081)
44.
Nichols, P., & Kuehl, B.J. ( 1999). Prophesying the reliability of cognitively complex assessments . Applied Measurement in Education, 12, 73-94.
45.
*Nuβbaum, A. ( 1984). Multivariate generalizability theory in educational measurement: An empirical study. Applied Psychological Measurement, 8, 219-230.
46.
Orwin, R.G. ( 1994). Evaluating coding decisions. In H. Cooper & L. V. Hedges (Eds.), Handbook of research synthesis (pp. 139-162). New York: Russell Sage Foundation.
47.
Parkes, J. ( 2001). The role of transfer in the variability of performance assessment scores. Educational Assessment, 7, 143-164.
48.
*Parkes, J., Zimmaro, D.M., Zappe, S.M., & Suen, H.K. ( 2000). Reducing task-related variance in performance assessment using concept maps. Educational Research and Evaluation, 6, 357-378.
49.
Rosenberg, M.S., Adams, D.C., & Gurevitch, J. ( 1997). MetaWin: Statistical software for meta-analysis with resampling tests. Sunderland, MA: Sinauer.
50.
Rosenquist, A., Shavelson, R.J., & Ruiz-Primo, M.A. (2000). On the "exchangeability" of hands-on and computer-simulated science performance assessment. Los Angeles: Graduate School of Education & Information Studies, University of California. (ERIC Document Reproduction Service No. 454293)
51.
Ruiz-Primo, M.A., & Shavelson, R.J. (1996). Rhetoric and reality in science performance assessments: An update. Journal of Research in Science Teaching, 35, 1045-1063.
52.
*Ruiz-Primo, M.A., & Shavelson, R.J. (1997). Concept-Map based assessment: On possible sources of sampling variability. Los Angeles : Center for Research on Evaluation, Standards, and Student Testing. (ERIC Document Reproduction Service No. 422403)
53.
Searle, S.R. ( 1971). Linear models. New York: Wiley.
54.
*Shavelson, R.J., Baxter, G.P., & Gao, X. ( 1993). Sampling variability of performance assessment. Journal of Educational Measurement, 30, 215-232.
55.
*Shavelson, R.J., Mayberry, P.W., Li, W., & Webb, N.M. ( 1990). Generalizability of job performance measurements: Marine Corps rifleman. Military Psychology, 2, 129-144.
56.
*Shavelson, R.J., Ruzi-Primo, M.A., & Wiley, E.W. ( 1999). Note on sources of sampling variability in science performance assessments. Journal of Educational Measurement, 36, 61-71.
57.
*Smith, E.V., Jr., & Kulikowich, J.M. (2004). An application of generalizability theory and many-facet Rasch measurement using a complex problem-solving skills assessment. Educational and Psychological Measurement, 64, 617-639.
58.
*Solano-Flores, G., Shavelson, R.J., Ruiz-Primo, M.A., Schultz, S.E., & Wiley, E.W. ( 1997). On the development and scoring of classification and observation science performance assessment (CSE Tech. Rep. 458). Los Angeles: Center for the Study of Evaluation, University of California.
59.
*Sudweeks, R.R., Glissmeyer, C.B., Morrison, T.G., Wilcox, B.R., & Tanner, M.W. ( 2004). Establishing reliable procedures for rating ELL students’ reading comprehension using oral retellings. Reading Research and Instruction, 43, 65-86.
60.
*Suzuki, K., & Harnisch, D.L. (1996, April 8-12). An investigation on the generalizability of performance-based assessment in mathematics. Paper presented at the Annual Meeting of the American Educational Research Association, New York. (ERIC Document Reproduction Service No. 435626)
61.
*Twing, J.S., & Baker, K.K. ( 1993). Generalizability of performance assessment: Lessons from direct measures of writing. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Atlanta, GA.
62.
*Webb, M.W., Schlackman, J., & Sugrue, B. ( 2000). The dependability and interchangeability of assessment methods in science. Applied Measurement in Education, 13, 277-301.
63.
*Webb, N.M., Shavelson, R.J., Kim, K.-S., & Chen, Z. ( 1989). Reliability (generalizability) of job performance measurements: Navy machinist mates. Military Psychology, 1, 91-110.
64.
*van Weeren, J., & Theunissen, T.J.J.M. (1987). Testing pronunciation: An application of generalizability theory. Language Learning , 37, 109-122.
65.
*Wolfe, E.W., & Gitomer, D.H. ( 2000). The influence of changes in assessment design on the psychometric quality of scores (Research Memorandum RM-00-1). Princeton, NJ: Educational Testing Service.
66.
*Yin, Y., & Shavelson, R.J. (2004). Application of generalizability theory to concept-map assessment research. Los Angeles : Center for the Study of Evaluation/ National Center for Research on Evaluation, Standards, and Student Testing . (ERIC Document Reproduction Service No. 483407)
67.
*Yoon, B., & Young, M.J. ( 2000). Validating standards-referenced science assessments. Los Angeles: Center for the Study of Evaluation/ National Center for Research on Evaluation, Standards, and Student Testing . (ERIC Document Reproduction Service No. 450138)
