Classroom observations remain the predominant data source used in teacher evaluations, but little is known about how rater characteristics may affect teachers’ scores. For special educators, whose instructional practice requires specialized knowledge and skills, school administrators (i.e., the raters) without experience in special education teaching may not be able to provide reliable and unbiased scores. This study included three school administrators who viewed and scored the classroom teaching of 19 special educators in California and Idaho; individuals with special education teaching experience also scored a subset of the instructional lessons. Generalizability study analyses revealed that the school administrators were not consistently reliable scorers of the special education teachers’ instruction. The school administrators were also more lenient on average in their ratings than peer teacher raters. Findings from this preliminary investigation suggest that rater background may matter, and future studies should explore how specific rater characteristics such as background and experience affect special educators’ performance ratings.
AllenJ. P.PiantaR. C.GregoryA.MikamiA. Y.LunJ. (2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333, 1034–1037. doi:10.1126/science.1207998
2.
BakerE. L.BartonP. E.Darling-HammondL.HaertelE.LaddH. L.LinnR. L.. . . ShepardL. A. (2010). Problems with the use of student test scores to evaluate teachers. Washington, DC: The Economic Policy Institute.
3.
BellC. A.GitomerD. H.McCaffreyD. F.HamreB. K.PiantaR. C.QiY. (2012). An argument approach to observation protocol validity. Educational Assessment, 17, 62–97. doi:10.1080/10627197.2012.715014
4.
BrennanR. L. (1992). Generalizability theory. Educational Measurement: Issues and Practice, 11(4), 27–34.
5.
BrennanR. L. (2000). Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement, 24, 339–353. doi:10.1177/01466210022031796
6.
BrownellM. T.JonesN. D. (2015). Teacher evaluation in special education: Approaches, supporting research, and challenges confronted. Journal of Special Education Leadership, 28(2), 63–73.
7.
BuzickH. M.LaitusisC. C. (2010). Using growth for accountability: Measurement challenges for students with disabilities and recommendations for research. Educational Researcher, 39, 537–544. doi:10.3102/0013189X10383560
8.
CardinetJ.JohnsonS.PiniG. (2010). Applying generalizability theory using EduG. New York, NY: Taylor & Francis.
9.
ChiuC. W.WolfeE. W. (2002). A method for analyzing sparse data matrices in the generalizability theory framework. Applied Psychological Measurement, 26, 321–338. doi:10.1177/0146621602026003006
10.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
11.
Council for Exceptional Children. (2013). The Council for Exceptional Children’s position on special education teacher evaluation. Teaching Exceptional Children, 45(3), 73–76. doi:10.1177/004005991304500308
12.
EllingtonJ. K.WilsonM. A. (2017). The performance appraisal milieu: A multilevel analysis of context effects in performance ratings. Journal of Business and Psychology, 32, 87–100. doi:10.1007/s10869-016-9437-x
13.
ErlichO.ShavelsonR. J. (1976). The application of generalizability theory to the study of teaching (Beginning Teacher Evaluation Study). San Francisco, CA: Far West Laboratory.
14.
Every Student Succeeds Act, Pub. L. 114-95, 20 U.S.C. §§ 6301 et seq. (2015).
15.
GoldringE.GrissomJ. A.RubinM.NeumerskiC. M.CannataM.DrakeT.. . . SchuermannP. (2015). Make room value added: Principals’ human capital decisions and the emergence of observation data. Educational Researcher, 44, 96–104. doi:10.3102/0013189X15575031
16.
GrahamM.MilanowskiA.MillerJ. (2012). Measuring and promoting inter-rater agreement of teacher and principal performance ratings. Washington, DC: Center for Educator Compensation Reform.
17.
GrissomJ. A.LoebS.MasterB. (2014). Effective instructional time use for school leaders: Longitudinal evidence from observations of principals. Educational Researcher, 42, 433–444. doi:10.3102/0013189X13510020
18.
HarariM. B.RudolphC. W.LaginessA. J. (2015). Does rater personality matter? A meta-analysis of rater Big Five–performance rating relationships. Journal of Occupational and Organizational Psychology, 88, 387–414. doi:10.1111/joop.12086
19.
HarrisD. N.IngleW. K.RutledgeS. A. (2014). How teacher evaluation methods matter for accountability: A comparative analysis of teacher effectiveness ratings by principals and teacher value-added measures. American Educational Research Journal, 51, 73–112. doi:10.3102/0002831213517130
20.
HillH. C.CharalambousC. Y.KraftM. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41, 56–64. doi:10.3102/0013189X12437203
21.
HoA. D.KaneT. J. (2013). The reliability of classroom observations by school personnel. Seattle, WA: Bill and Melinda Gates Foundation.
22.
HoldheideL. R.GoeL.CroftA.ReschlyD. J. (2010, July). Challenges in evaluating special education teachers and English language learner specialists [Research and Policy Brief]. Washington, DC: National Comprehensive Center for Teacher Quality.
JawaharI. M.WilliamsC. R. (1997). Where all the children are above average: The performance appraisal purpose effect. Personnel Psychology, 50, 905–925. doi:10.1111/j.1744-6570.1997.tb01487.x
25.
JohnsonE.SemmelrothC. L. (2012). Examining interrater agreement analyses of a pilot special education observation tool. The Journal of Special Education Apprenticeship, 1(2). Retrieved from http://josea.info/index.php?page=vol1no2
26.
JonesN. D.BrownellM. T. (2013). Examining the use of classroom observations in the evaluation of special education teachers. Assessment for Effective Intervention, 39, 112–124. doi:10.1177/1534508413514103
27.
LevyP. E.WilliamsJ. R. (2004). The social context of performance appraisal: A review and framework for the future. Journal of Management, 30, 881–905. doi:10.1016/j.jm.2004.06.005
28.
LittleO. (2009). Teacher evaluation systems: The window for opportunity and reform. Washington, DC: National Education Association.
29.
PraetoriusA. K.PauliC.ReusserK.RakoczyK.KliemeE. (2014). One lesson is all you need? Stability of instructional quality across lessons. Learning and Instruction, 31, 2–12. doi:10.1016/j.learninstruc.2013.12.002
30.
SchoormanD. F. (1988). Escalation bias in performance appraisals: An unintended consequence of supervisor participation in hiring decisions. Journal of Applied Psychology, 73, 58–62. doi:10.1037//0021-9010.73.1.58
31.
SemmelrothC. L.JohnsonE. (2013). Measuring rater reliability on a special education observation tool. Assessment for Effective Intervention, 39, 131–145. doi:10.1177/1534 508413511488
32.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. Newbury Park, CA: SAGE.
33.
SledgeA.PazeyB. L. (2013). Measuring teacher effectiveness through meaningful evaluation: Can reform models apply to general education and special education teachers?Teacher Education and Special Education, 36, 231–246. doi:10.1177/0888406413489839
34.
SteinbrecherT. D.FixR.MahalS. A.SernaL.McKeownD. (2015). All you need is patience and flexibility: Administrators’ perspectives on special educator knowledge and skills. Journal of Special Education Leadership, 28, 89–102.
35.
TaylorE. S.TylerJ. H. (2011, March). The effect of evaluation on performance: Evidence from longitudinal student achievement data of mid-career teachers (Working Paper No. 16877). Cambridge, MA: National Bureau of Economic Research. Retrieved from http://www.nber.org/papers/w16877
WeisbergD.SextonS.MulhernJ.KneelingD. (2009). The widget effect: Our national failure to acknowledge and act on teacher effectiveness. New York, NY: The New Teacher Project.
