This article develops a procedure for measuring how much is gained by students in a pretest and posttest situation against a target score on the posttest. The author defines a productivity index, Mj, for teacher j as the ratio of estimated gains to an estimated standard that is the distance between an estimate of the pretest and target score. Using language, mathematics, and reading scores on the SAT 9 for 1999 and 2000 from 75 public elementary classrooms (Grades 3-6 in 2000), the author employs a Bayesian implementation of a multivariate mixed model for repeated test scores from individual students. The analysis points to statistically significant gains on the whole for Grades 3, 4, and 6. The strength of the approach lies in a straightforward estimation of the productivity index and a procedure for representing its uncertainty in the form of a productivity profile. This approach also facilitates a Bayesian effect size analysis free from frequentist appeals to noncentral t or F distributions.
Get full access to this article
View all access options for this article.
References
1.
Allison, P. 1990. “Change Scores as Dependent Variables in Regression Analysis.” Pp. 93-114 in Sociological Methodology, edited by C. Clogg. San Francisco: Jossey-Bass .
2.
Berk, R. A. 1988. “Fifty Reasons Why Student Achievement Gain Does Not Mean Teacher Effectiveness.” Journal of Personnel Evaluation in Education1: 345-363 .
3.
Bock, R. D. 1975. Multivariate Statistical Methods in Behavioral Research. New York: McGraw-Hill .
4.
Box, G. E. P. 1976. “Science and Statistics.” Journal of the American Statistical Association71: 791-799 .
5.
Brooks, S. P. and A. Gelman. 1998. “Alternative Methods for Monitoring Convergence of Iterative Simulations.” Journal of Computational and Graphical Statistics7: 434-455 .
6.
Brooks, S. P. and G. O. Roberts. 1998. “Assessing Convergence of Markov Chain Monte Carlo Algorithms.” Statistics and Computing8: 319-335 .
7.
Bryk, A. S. and S. W. Raudenbush. 1992. Hierarchical Linear Models: Applications and Data Analysis Methods. Newbury Park, CA: Sage .
8.
Bryk, A. S., Y. M. Thum, J. Q. Easton, and S. Luppescu. 1998. “Assessing School Academic Productivity: The Case of Chicago School Reform.” Social Psychology of Education2: 103-142 .
9.
Bryk, A. S. and H. I. Weisberg. 1976. “Value-Added Analysis: A Dynamic Approach to the Estimation of Treatment Effects.” Journal of Educational Statistics1: 127-155 .
10.
Burket, G. R. 1984. “Response to Hoover.” Educational Measurement: Issues and Practices3: 15-16 .
11.
California Department of Education (CDE). 1999. Public Schools Accountability Act (PSAA) [Legislation]. California Department of Education .
12.
Campbell, D. T. and D. A. Kenny. 1999. A Primer on Regression Artifacts. New York: Guilford .
13.
Carlin, B. P. 1996. “Hierarchical Longitudinal Modelling.” Pp. 303-319 in Markov Chain Monte Carlo in Practice, edited by W. R. Gilks, S. Richardson, and D. J. Spiegelhalter. London: Chapman & Hall .
14.
Carlin, B. P. and T. A. Louis. 1996. Bayes and Empirical Bayes Methods for Data Analysis. London: Chapman & Hall .
15.
Casella, G. and E. I. George. 1992. “Explaining the Gibbs Sampler.” Journal of the American Statistical Association46: 167-174 .
16.
Coleman, J. S. and N. L. Karweit. 1972. Information Systems and Performance Measures in Schools. Englewood Cliffs, NJ: Educational Technology Publications .
17.
Collins, L. M. 1996. “Is Reliability Obsolete? A Commentary on ‘Are Simple Gain Scores Obsolete?’ ” Applied Psychological Measurement20: 289-292 .
18.
Cowles, M. K. and B. P. Carlin. 1996. “Markov Chain Monte Carlo Diagnostics: A Comparative Review.” Journal of the American Statistical Association91: 883-904 .
19.
Cronbach, L. J. and L. Furby. 1970. “How Should We Measure ‘Change’—or Should We?” Psychological Bulletin74: 68-80 .
20.
Draper, D. 1995. “Inference and Hierarchical Modeling in the Social Sciences.” Journal of Educational and Behavioral Statistics20: 115-147 .
21.
Eissenberg, T. E. and L. M. Rudner. 1988. “Explaining Test Results to Parents.” Practical Assessment, Research & Evaluation1: 1 .
22.
Fitz-Gibbon, C. T. 1997. “The Value-Added National Project: Final Report. Feasibility Studies for a National System of Value-Added Indicators.” Report, SCAA, London .
23.
Fleishman, A. I. 1980. “Confidence Intervals for Correlation Ratios.” Educational and Psychological Measurement40: 659-670 .
24.
Gelfand, A. E. 1996. “Model Determination Using Sampling-Based Methods.” Pp. 145-161 in Markov Chain Monte Carlo in Practice, edited by W. R. Gilks, S. Richardson, and D. J. Spiegelhalter. London: Chapman & Hall .
25.
Gelman, A., J. C. Carlin, H. Stern, and D. B. Rubin. 1995. Bayesian Data Analysis. London: Chapman & Hall .
26.
Gilks, W. R., S. Richardson, and D. J. Spiegelhalter, eds. 1996. Markov Chain Monte Carlo in Practice. London: Chapman & Hall .
27.
Glass, G. and K. Hopkins. 1984. Statistical Methods in Education and Psychology. 2d ed. Boston: Allyn & Bacon .
28.
Glass, G. V. 1990. “Using Student Test Scores to Evaluate Teachers.” Pp. 229-240 in The New Handbook of Teacher Evaluation, edited by L. Millman and L. Darling-Hammond. Newbury Park, CA: Sage .
29.
Goldstein, H. 1995. Multilevel Statistical Models. London: Edward Arnold .
30.
Goldstein, H. and D. J. Spiegelhalter. 1996. “League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance.” Journal of the Royal Statistical Society, Series A159: 384-443 .
31.
Good, T. L., B. J. Biddle, and J. E. Brody. 1975. Teachers Make a Difference. Lanham, MD: University Press of America .
32.
Gray, J., D. Jesson, H. Goldstein, K. Hedger, and J. Rabash. 1995. “A Multi-Level Analysis of School Improvement: Changes in School’s Performance Over Time.” School Effectiveness and School Improvement6: 97-114 .
33.
Haertel, E. 1986. “The Valid Use of Student Achievement Measures for Teacher Evaluation.” Educational Evaluation and Policy Analysis8: 45-60 .
34.
Harker, R. and R. Nash. 1996. “Academic Outcomes and School Effectiveness: Type ‘A’ and Type ‘B’ Effects.” New Zealand Journal of Educational Studies32: 143-170 .
35.
Harris, C. W. 1963. Problems in the Measurement of Change. Madison: University of Wisconsin Press .
36.
Hills, J. R. 1984. “Interpreting NCE Scores.” Educational Measurement: Issues and Practices3: 25-26, 31 .
37.
Hoeting, J. A., D. Madigan, A. E. Raftery, and C. T. Volinsky. 1999. “Bayesian Model Averaging: A Tutorial.” Statistical Science14: 382-417 .
38.
Hoover, H. D. 1984a. “The Most Appropriate Scores for Measuring Educational Development in Elementary Schools: GE’s.” Educational Measurement: Issues and Practices3: 8-14 .
39.
Hoover, H. D. 1984b. “Rejoinder to Burket.” Educational Measurement: Issues and Practices3: 16-18 .
40.
Jaeger, R. M. 1993. Statistics: A Spectator Sport. 2d ed. Newbury Park, CA: Sage .
41.
Kalaian, H. and S. W. Raudenbush. 1996. “A Multivariate Mixed Linear Model for Meta-Analysis.” Psychological Methods3: 227-235 .
42.
Kaplan, D. and P. R. Elliott. 1997. “A Model-Based Approach to Validating Education Indicators Using Multilevel Structural Equation Modeling.” Journal of Educational and Behavioral Statistics22: 323-348 .
43.
Kass, R. E. and R. E. Raftery. 1995. “Bayes Factors.” Journal of the American Statistical Association90: 773-795 .
44.
Laird, N. M. and J. H. Ware. 1982. “Random-Effects Models for Longitudinal Data.” Biometrika65: 581-590 .
45.
Lindley, D. V. and A. F. M. Smith. 1972. “Bayesian Estimates for the Linear Model.” Journal of the Royal Statistical Society, Series B34: 1-41 .
46.
Linn, R. L. 2000. “Assessments and Accountability.” Educational Researcher29: 4-16 .
47.
Linn, R. L. 2001. “The Design and Evaluation of Educational Assessment and Accountability Systems.” CSE Technical Report No. 539, University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
48.
Linn, R. L. and J. A. Slinde. 1977. “The Determination of the Significance of Change Between Pre- and Post-Periods.” Review of Educational Research47: 121-150 .
49.
Littell, R. C., G. A. Milliken, W. W. Stroup, and R. D. Wolfinger. 1996. SAS System for Mixed Models. Cary, NC: SAS Institute .
50.
Maassen, G. H. 2000. “Kelley’s Formula as a Basis for the Assessment of Reliable Change.” Psychometrika65: 187-197 .
51.
Maris, E. 1998. “Covariance Adjustment Versus Gain Score—Revisited.” Psychological Methods3: 309-327 .
52.
Mellenbergh, G. J. 1999. “A Note on Simple Gain Score Precision.” Applied Psychological Measurement23: 87-89 .
53.
Meyer, R. H. 1996. “Value-Added Indicators of School Performance.” Pp. 197-223 in Improving America’s Schools: The Role of Incentives, edited by E. A. Hanushek and D. W. Jorgenson. Washington, DC: National Academic Press .
54.
Millman, J. 1981. “Student Achievement as a Measure of Teacher Competence.” Pp. 146-166 in Handbook of Teacher Evaluation, edited by J. Millman. Beverly Hills, CA: Sage .
55.
No Child Left Behind Act (2001), Pub. L. No. 107-110, 115 Stat. 1425.
56.
O’Hagan, A. 1994. Bayesian Inference. Vol. 2B. London: Arnold .
57.
O’Hagan, A., J. W. Stevens, and J. Montmartin. 2000. “Inference for the Cost-Effectiveness Acceptability Curve and the Cost-Effectiveness Ratio.” Pharmocoeconomics17: 339-349 .
58.
O’Hagan, A., J. W. Stevens, and J. Montmartin. 2001. “Bayesian Cost-Effectiveness Analysis From Clinical Trial Data.” Statistics in Medicine20: 733-753 .
59.
Raudenbush, S. W., R. T. Brennan, and R. C. Barnett. 1995. “A Multivariate Hierarchical Model for Studying Psychological Change Within Married Couples.” Journal of Family Psychology9: 161-176 .
60.
Raudenbush, S. W., R. P. Fotiu, and Y. F. Cheong. 1999. “Synthesizing Results From the Trial State Assessment.” Journal of Educational and Behavioral Statistics24: 413-438 .
61.
Raudenbush, S. W. and X.-F. Liu. 2000. “Statistical Power and Optimal Design for Multisite Randomized Trials.” Psychological Methods5: 199-213 .
62.
Raykov, T. 1999. “Are Simple Gain Scores Obsolete? On an Approach to the Study of Correlates and Predictors of Ability Growth.” Applied Psychological Measurement23: 120-126 .
63.
Rogosa, D. R. 1995. “Myths and Methods: Myths About Longitudinal Research Plus Supple-mental Questions.” Pp. 3-66 in The Analysis of Change, edited by J. M. Gottman. Mahwah, NJ: Lawrence Erlbaum .
64.
Rogosa, D. R., D. Brandt, and M. Zimowski. 1982. “A Growth Curve Approach to the Measurement of Change.” Psychological Bulletin90: 726-748 .
65.
Sanders, W. L. and S. P. Horn. 1994. “The Tennessee Value-Added Assessment System (TVAAS): Mixed-Model Methodology in Educational Assessment.” Journal of Personnel Evaluation8: 299-311 .
66.
Sanders, W. L. and J. C. Rivers. 1996. “Cummulative and Residual Effects of Teachers on Future Student Academic Achievement.” Research progress report, University of Tennessee, Value-Added Research and Assessment Center, Knoxville, TN .
67.
Saunders, L. 1999. “A Brief History of Educational ‘Value Added’: How Did We Get to Where We Are?” School Effectiveness and School Improvement10: 233-256 .
68.
Schacter, J. 2001. Teacher Advancement Program Performance-Based Accountability. Santa Monica, CA: Milken Family Foundation .
69.
Schulz, E. M. and W. A. Nicewander. 1997. “Grade Equivalent and IRT Representations of Growth.” Journal of Educational Measurement34: 315-332 .
70.
Schuster, C. and A. von Eye. 1998. “Determining the Meaning of Parameters in Multi-level Models for Longitudinal Data.” International Journal of Behavioral Development22: 475-491 .
71.
Searle, S. R. 1971. Linear Models. New York: John Wiley .
72.
Seltzer, M., K. C. Choi, and Y. M. Thum. 2001. “Examining Relationships Between Where Students Start and How Rapidly They Progress: Implications for Conducting Analyses That Help Illuminate the Distribution of Achievement Within Schools.” CSE Technical Report No. 560, University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
73.
Seltzer, M., K. Frank, and A. S. Bryk. 1994. “The Metric Matters: The Sensitivity of Conclusions About Growth in Student Achievement to Choice of Metric.” Educational Evaluation and Policy Analysis16: 41-49 .
74.
Shinkfield, A. J. and D. L. Stufflebeam. 1995. Teacher Evaluation: Guide to Effective Practice. Kalamazoo, MI: Center for Research on Educational Accountability and Teacher Evaluation .
75.
Slavin, R. E. and N. A. Madden. 1991. “Modifying Chapter 1 Program Improvement Guidelines to Reward Appropriate Practices.” Educational Evaluation and Policy Analysis13: 369-379 .
76.
Smith, A. F. M. and A. E. Gelman. 1992. “Bayesian Statistics Without Tears: A Sampling-Resampling Perspective.” Journal of the American Statistical Association46: 84-88 .
77.
Snijders, T. and R. Bosker. 1999. Multilevel Analysis. Thousand Oaks, CA: Sage .
78.
Spiegelhalter, D. J., N. G. Best, B. P. Carlin, and A. van der Linde. 2002. “Bayesian Measures of Model Complexity and Fit (With Discussion).” Journal of the Royal Statistical Society, Series B64: 583-640 .
79.
Spiegelhalter, D. J., A. Thomas, and N. G. Best. 1999. “WinBUGS: Bayesian Inference Using Gibbs Sampling, Version 1.3” [Computer program]. Cambridge, UK: MRC Biostatistics Unit .
80.
Standard Achievement Test Series Technical Data Report. 1997. 9th ed. San Antonio, TX: Harcourt-Brace Educational Measurement .
81.
Steiger, J. H. and R. T. Fouladi. 1997. “Non-Centrality Interval Estimation and the Evaluation of Statistical Models.” Pp. 221-257 in What If There Were No Significance Tests? edited by L. L. Harlow, S. A. Mulaik, and J. H. Steiger. Mahwah, NJ: Lawrence Erlbaum .
82.
Stone, J. E. 1999. “Value-Added Assessment: An Accountability Revolution.” Pp. 239-249 in Better Teachers, Better Schools, edited by M. Kanstoroom and C. E. Finn Jr. Washington, DC: Thomas B. Fordham Foundation .
83.
Thomas, S. and H. Goldstein. 1995. “Questionable Value.” Education185: 17 .
84.
Thompson, B. 2002. “What Future Quantitative Social Science Research Could Look Like: Confidence Intervals for Effect Sizes.” Educational Researcher31: 25-32 .
85.
Thum, Y. M. 1994. “Analysis of Individual Variation: A Multivariate Hierarchical Linear Model for Behavioral Data.” Doctoral dissertation, University of Chicago.
86.
Thum, Y. M. 1997. “Hierarchical Linear Models for Multivariate Outcomes.” Journal of Educational and Behavioral Statistics22: 77-108 .
87.
Thum, Y. M. 2002. “Measuring Student and School Progress with the California API.” CSE Technical Report No. 578, University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
88.
Thum, Y. M. 2003. “No Child Left Behind: Methodological Challenges and Recommendations for Measuring Adequate Yearly Progress.” CSE Technical Report No. 590, University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
89.
Tymms, P. 1990. “Can an Indicator System Improve the Effectiveness of Science and Mathematics Education? The Case of the UK.” Evaluation and Research in Education4: 61-73 .
90.
Webster, W. J. and R. L. Mendro. 1997. “The Dallas Value-Added Accountability System.” Pp. 81-99 Grading Teachers, Grading Schools: Is Student Achievement a Valid Evaluation Measure? edited by J. Millman. Thousand Oaks, CA: Corwin Press .
91.
Willett, J. B. 1988. “Questions and Answers in the Measurement of Change.” Review of Research in Education15: 345-422 .
92.
Williams, R. H. and D. W. Zimmerman. 1996. “Are Gain Scores Obsolete?” Applied Psychological Measurement20: 59-69 .
93.
Willms, D. 1992. Monitoring School Performance: A Guide for Educators. Washington, DC: Falmer .
94.
Willms, D. J. and S. W. Raudenbush. 1989. “A Longitudinal Hierarchical Linear Models for Estimating School Effects and Their Stability.” Journal of Educational Measurement26: 209-232 .
95.
Yen, W. M. 1986. “The Choice of Scale for Educational Measurement: An IRT Perspective.” Journal of Educational Measurement23: 299-325 .
