Purpose: The article reports on a study of school leader decision making that examined variation in the validity of teacher evaluation ratings in a school district that has implemented a standards-based teacher evaluation system. Research Methods: Applying mixed methods, the study used teacher evaluation ratings and value-added student achievement data to identify 23 school leaders with “more” and “less” valid results. These leaders were interviewed to learn about their attitudes on teacher evaluation, their decision-making strategies, and school contexts. Results from interviews with a subset of eight school leaders with 2 years of consistent validity scores (n = 4 more valid and n = 4 less valid) were analyzed. Findings: Substantial variation was found in the relationship of evaluators' ratings of teachers and value-added measures of the average achievement of the teachers' students. The results did not yield a simple explanation for the differences in validity of evaluators' ratings. Instead, evaluators' decisions were found to be a complex and idiosyncratic function of motivation, skill, and context. Conclusion: The results suggested why overall validity of ratings was lower than expected and highlighted challenges in research on school leader decision making and cautions for using such decisions for high stakes purposes. Recommendations are provided for improving evaluation accuracy and validity, with considerations for performance evaluation policy and future research on school leader decision making and teacher evaluation.
Bernardin, H.J., & Cardy, R.L. (1982). Appraisal accuracy: The ability and motivation to remember the past. Public Personnel Management, 11(4), 352-357.
2.
Bretz, R.D., Milkovich, G.T., & Read, W. (1992). The current state of performance appraisal research and practice: Concerns, directions, and implications. Journal of Management, 18(2), 321-352.
3.
Brophy, J. (1986). Teacher influences on student achievement. American Psychologist, 41(1), 1069-1077.
4.
Bryk, A., & Schneider, B. (2002). Trust in schools: A core resource for improvement . New York: Russell Sage.
5.
Cleveland, J.N., Murphy, K.R., & Williams, R.E. (1989). Multiple uses of performance appraisal: Prevalence and correlates. Journal of Applied Psychology , 74(1), 130-135.
6.
Danielson, C. (1996). Enhancing professional practice: A framework for teaching. Alexandria, VA: Association for Supervision and Curriculum Development.
7.
Danielson, C., & McGreal, T. (2000). Teacher evaluation to enhance professional practice . Alexandria, VA: Association for Supervision and Curriculum Development.
8.
DeCotiis, T., & Petit, A. (1978). The performance appraisal process: A model and some testable propositions. Academy of Management Review, 3, 635-646.
9.
DeNisi, A.S., Cafferty, T.P., & Meglino, B.M. (1984). A cognitive view of the performance appraisal process: A model and research propositions. Organizational Behavior and Human Performance, 33, 360-396.
10.
Ellett, C.D. (1997). Classroom-based assessments of teaching and learning . In J. Stronge, Evaluating teaching: A guide to current thinking and best practice (pp. 107-28). Newbury Park, CA: Sage.
11.
Ellett, C.D., Annunziata, J., & Schiavone, S. (2002). Web-based support for teacher evaluation and professional growth: The Professional Assessment and Comprehensive Evaluation System (PACES) . Journal of Personnel Evaluation in Education, 16(1), 63-74.
12.
Ericsson, K.A., & Simon, H.A. (1993). Protocol analysis: Verbal reports as data (Revised ed.). Cambridge, MA: The MIT Press .
13.
Freeberg, N.E. (1969). Relevance of rater-ratee acquaintance in the validity and reliability of ratings. Journal of Applied Psychology , 53, 518-524.
14.
Grey, R.J., & Kipness, D. (1976). Untangling the performance appraisal dilemma: The influence of perceived organizational context on evaluative processes . Journal of Applied Psychology, 61, 329-335.
15.
Halverson, R., Kelley, C., & Kimball, S. (2004). Implementing teacher evaluation systems: How principals make sense of complex artifacts to shape local instructional practice. In W. Hay & C. Miskel (Ed.), Educational administration, policy, and reform: Research and measurement. a volume in research and theory in educational administration (Vol. 3, pp. 153-188.). Greenwich, CT: George F. Johnson.
16.
Hedge, J.W., & Kavanagh, M.J. (1988). Improving the accuracy of performance evaluations: Comparison of three methods of performance appraiser training. Journal of Applied Psychology, 73(1), 68-73.
17.
Heneman, H.G., III, Milanowski, A., Kimball, S.M., & Odden, A. (2006). Standards-based teacher evaluation as a foundation for knowledge- and skill-based pay (CPRE Policy Brief, RB-45). Philadelphia: Consortium for Policy Research in Education, University of Pennsylvania.
18.
Honig, M.I. (2006). Complexity and policy implementation: Challenges and opportunities for the field. In M. I. Honig (Ed.), New directions in education policy implementation: Confronting complexity (pp. 1-24). Albany, NY: State University of New York Press.
19.
Ivancevich, J.M. (1983). Contrast effects in performance evaluation and reward practices. Academy of Management Journal, 26(3), 465-476.
20.
Jacob, B. & Lefgren, L. (2006). When principals rate teachers. Education Next, 6(2), 58-64.
21.
Judge, T.A., & Ferris, G.R. (1993). Social context of performance evaluation decisions . Academy of Management Journal, 38(1), 80-105.
22.
Kimball, S.M. (2001). Innovations in teacher evaluation: Case studies of two school districts with teacher evaluation systems based on the Framework for Teaching. Ann Arbor, MI: UMI Dissertation Publishing.
23.
Kimball, S.M. (2002). Analysis of feedback, enabling conditions and fairness perceptions of teachers in three school districts with new standards-based evaluation systems. Journal of Personnel Evaluation in Education , 16(4), 241-268.
24.
Kimball, S.M., White, B., Milanowski, A.T., & Borman, G. (2004). Examining the relationship between teacher evaluation and student assessment results in Washoe County. Peabody Journal of Education, 79(4), 54-78.
25.
Klein, S.J. (1998). Standards for teacher tests. Journal of Personnel Evaluation in Education, 12(2), 123-138.
26.
Kluger, A.N., & DeNisi, A. (1996). The effects of feedback intervention on performance: A historical review, meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284.
Louis, K.M., & Marks, H.M. (1998). Does professional community affect the classroom? Teachers' work and student experiences in restructuring schools. American Journal of Education, 106, 532-575.
29.
Martin, S.L., & Klimoski, R.J. (1990). Use of verbal protocols to trace cognitions associated with self- and supervisor evaluations of performance . Organizational Behavior & Human Decision Processes, 46(1), 135-154.
30.
Medley, D.M., & Coker, H. (1987). The accuracy of principals' judgments of teacher performance. Journal of Educational Research, 80(4), 242-247.
31.
Milanowski, A.T. (2004). The relationship between teacher performance evaluation scores and student achievement: Evidence from Cincinnati. Peabody Journal of Education, 79(4), 33-53.
32.
Milanowski, A.T., Kimball, S.M., & Odden, A.R. (2005). Teacher accountability measures and links to learning . In R. Rubenstein et al. (Eds.), Measuring school performance & efficiency: Implications for practice and research. Yearbook of the American Education Finance Association (pp. 133-188). Larchmont, NY: Eye on Education.
33.
Mischel, W. (1977). The interaction of person and situation. In D. Magnusson & N. S. Endler (Eds.), Personality at the crossroads: Current issues in interactional psychology (pp. 333-352). Hillsdale, NJ: Lawrence Erlbaum.
Napier, N.K., & Latham, G.P. (1986). Outcome expectancies of people who conduct performance appraisals. Personnel Psychology, 39, 827-837.
36.
Nelson, B.S., & Sassi, A. (2005). The effective principal: Instructional leadership for high-quality learning. New York: Teachers College Press.
37.
Peterson, K.D. (2000). Teacher evaluation: A comprehensive guide to new directions and practice (2nd ed.). Thousand Oaks, CA: Corwin Press.
38.
Shulman, L.S. (1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57(1), 1-22.
39.
Smith, D.E. (1986). Training programs for performance appraisal: A review. Academy of Management Review, 11(1), 22-40.
40.
Smither, J.W., Barry, S.R., & Reilly, R.R. (1989). An investigation of the validity of expert true score estimates in appraisal research. Journal of Applied Psychology, 74(1), 143-151.
41.
Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage.
42.
Stodolsky, S.S. (1990). Classroom observation. In J. Millman & L. Darling-Hammond (Eds.), The new handbook of teacher evaluation: Assessing elementary and secondary school teachers (pp. 175-190). Newbury Park, CA: Sage.
43.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Applied Social Research Methods Series (Vol. 46., pp. 297-319). Thousand Oaks, CA: Sage.
44.
Tziner, A., Murphy, K.R., & Cleveland, J.N. (2001). Relationship between attitudes toward organizations and performance appraisal systems and rating behavior . International Journal of Selection and Assessment, 9(3), 226-239.
45.
U.S. Department of Education. (2006). Teacher incentive fund, notice inviting applications for New Awards for Fiscal Year (FY) 2006. Retrieved July 17, 2006, from http://www.ed.gov/legislation/FedRegister/announcements/2006-2/050106e.html
46.
Woehr, D.J., & Huffcutt, A.I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205.
