In generalizability analyses, unstable, and potentially invalid, variance component estimates may result from using only a limited portion of available data. However, missing observations are common in operational performance assessment settings because of the nature of the assessment design. This article describes a procedure for overcoming the computational and technological limitations in analyzing data with missing observations by extracting data from a sparsely .lled data matrix into analyzable smaller subsets of data. This subdividing method is accomplished by creating data sets that exhibit structural designs that are common in generalizability analyses, namely, the crossed, MBIB, and nested designs. The validity of this subdividing method is examined using a Monte Carlo simulation. The method is demonstrated on an operational data set.
Get full access to this article
View all access options for this article.
References
1.
Babb, J. S. (1986). Pooling maximum likelihood estimates of variance components obtained from subsets of unbalanced data. Unpublished master’s thesis, Cornell University.
2.
Bell, J. F. (1985). Generalizability theory: The software problem. Journal of Educational Statistics, 10(1), 19-29.
3.
Brennan, R. L. (1992a). Elements of generalizability theory. Iowa City, IA: American College Testing.
4.
Brennan, R. L. (1992b). An NCME instructional module on generalizability theory. Educational Measurement: Issues and Practice, 11(4), 27-34.
5.
Brennan, R. L. , Gao, X., & Colton, D. A. (1995). Generalizability analyses of work keys listening and writing tests. Educational and Psychological Measurement, 55(2), 157-176.
6.
Brennan, R. L. , Harris, D. J., & Hanson, B. A. (1987). The bootstrap and other procedures for examining the variability of estimated variance components in testing contexts. Paper presented at the annual meeting of the National Council on Measurement in Education, Washington, DC.
7.
Brennan, R. L. , Jarjoura, D., & Deaton, E. L. (1980). Some issues concerning the estimation and interpretation of variance components in generalizability theory. ACT Technical Bulletin (Report No. 36). Iowa City, IA: American College Testing.
8.
CAAP (2001). Documentation of the Collegiate Assessment of Academic Proficiency [Online]. Available: http://www.act.org/caap/index.html
9.
Chiu, C. W. T. (1999). Scoring performance assessments based on judgements: Utilizing meta-analysis to estimate variance components in generalizability theory for unbalanced situations. Unpublished doctoral dissertation, Michigan State University.
10.
Chiu, C. W. T. (2001a). Generalizability theory: Scoring large-scale assessments based on judgements. Boston, MA: Kluwer Academic.
11.
Chiu, C. W. T. (2001b). An SPSS module: Generalizability analysis for balanced and unbalanced designs [Software]. Law School Admission Council (LSAC), Newtown, PA.
12.
Crick, J. E. , & Brennan, R. L. (1982). GENOVA: A generalized analysis of variance system (FORTRAN IV computer program and manual). Dorchester, MA: Computer Facilities, University of Massachusetts at Boston.
13.
Cronbach, L. J. , Bradburn, N. M., & Horvitz, D. G. (1994). Sampling and statistical procedures used in the California Learning Assessment System. Report of the Select Committee, Palo Alto, CA.
14.
Cronbach, L. J. , Gleser, G. C., & Rajaratnam, N. (1963). Theory of generalizability. A liberalization of reliability theory. British Journal of Mathematical and Statistical Psychology, 16, 137-173.
15.
Donoghue, J. R. , & Hombo, C. M. (2000). A comparison of different model assumptions bout rater effects. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
16.
Engelhard, G., Jr. (1997). Constructing rater and task banks for performance assessments. Journal of Outcome Measurement, 1(1),19-93.
17.
Fleiss, J. L. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5, 105-112.
18.
Harwell, M. , Stone, C. A., Hsu, T. C., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychology Measurement, 20(2), 101-125.
19.
Henderson, C. R. (1953). Estimation of variance and covariance components. Biometrics, 9, 227-252.
20.
Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it?Psychological Methods, 5(1), 64-86.
21.
Koretz, D. , Stecher, B., & McCaffrey, D. (1994). The Vermont Portfolio Assessment Program: Findings and implications. Educational Measurement: Issues and Practice, 13, 5-6.
22.
Lane, S. , Liu, M., Ankenmann, R. D., & Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33(1), 71-92.
23.
Longford, N. T. (1995). Models for uncertainty in educational testing. New York: Springer-Verlag.
24.
Luecht, R. M. , & Smith, P. L. (1989). The effects of bootstrapping strategies on the estimation of variance components. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
25.
Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18, 5-11.
26.
Myford, C. M. , Marr, D. B., & Linacre, J. M. (1995). Reader calibration and its potential role in equating for the Test of Written English (Report prepared by the Center for Performance Assessment, MS # 95-02). Princeton, NJ: Educational Testing Service.
27.
Nitko, A. J. (1996). Educational assessment of students (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.
28.
Othman, A. R. (1995). Examining task sampling variability in science performance assessments. Unpublished doctoral dissertation, University of California, Santa Barbara.
29.
Patz, R. (1996). Markov chain Monte Carlo methods for item response theory models. Unpublished doctoral dissertation, Carnegie Mellon University, Pittsburgh, PA.
30.
Saal, F. E. , Downey, R. G., & Lahey, M. A. (1980). Rating the rating: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
31.
SAS Institute, Inc. (1985). SAS user’s guide: Statistics (Version 5). Cary, NC: Author.
32.
Schroeder, M. L. (1986). Inferential procedures for multifaceted coefficients of generalizability. Unpublished doctoral dissertation, University of British Columbia.
33.
Searle, S. R. (1987). Linear models for unbalanced data. New York: John Wiley.
34.
Searle, S. R. , Casella, G., & McCulloch, C. E. (1992). Variance components. New York: John Wiley.
35.
Shavelson, R. J. , Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30(3), 215-232.
36.
Smith, P. L. (1982). A confidence interval approach for variance component estimates in the context of generalizability theory. Educational and Psychological Measurement, 42, 459-466.
Wainer, H. (1993). Measurement problems. Journal of Educational Measurementf, 30(1), 1-21.
39.
Wolfe, E. W. , Chiu, C. W. T., & Myford, C. M. (2000). Detecting rater effects with a multi-faceted Rasch rating scale model. In M. Wilson & G. Engelhard (Eds.), Objective measurement: Theory into practice (Vol. 5, pp. 147-164). Princeton, NJ: Educational Testing Service.
