Teacher rating scales (TRS) often play a part in service eligibility decisions for gifted services. Although schools regularly use TRS to identify gifted students either as part of an informal nomination process or through behavioral rating scales, there is little research documenting the between-teacher variance in teacher ratings and the consequences of such rater dependence. To evaluate the possible benefits or disadvantages of using TRS as part of a gifted identification process, we examined the student-, teacher-, and school-level variance in TRS, controlling for student ability and achievement to determine the unique information, consistency, and potential bias in TRS. Between 10% and 25% of a students’ TRS score can be attributed to the teacher doing the rating, and between-teacher standard deviations represent an effect size of one-third to one-half standard deviation. Our results suggest that TRS are not easily comparable across teachers, making it impossible to set a cut score for admission into a program (or for further screening) that functions equitably across teachers.
DaiD. Y.ChenF. (2013). Three paradigms of gifted education: In search of conceptual clarity in research and practice. Gifted Child Quarterly, 57(3), 151–168. https://doi.org/10.1177/0016986213490020
3.
DowneyD. B.PribeshS. (2004). When race matters: Teachers’ evaluations of students’ classroom behavior. Sociology of Education, 77(4), 267–282. https://doi.org/10.1177/003804070407700401
4.
DowneyD. B.von HippelP. T.BrohB. A. (2004). Are schools the great equalizer? Cognitive inequality during the summer months and the school year. American Sociological Review, 69(5), 613–635. http://www.jstor.org/stable/3593031
5.
EklundK.KilgusS.von der EmbseN.BeardmoreM.TannerN. (2017). Use of universal screening scores to predict distal academic and behavioral outcomes: A multilevel approach. Psychological Assessment, 29(5), 486–499. https://doi.org/10.1037/pas0000355
6.
FordD. Y.KingR. A. (2014, Fall). Desegregating gifted education for under-represented Black and Hispanic students: Equity promotes equality. Teaching for High Potential, 1, 13–16.
GentryM.WhitingG.GrayA. M. (2022). Systemic inequities in identification and representation of Black youth with gifts and talents: Access, equity, and missingness in urban and other school locales. Urban Education, Advance online publication. https://doi.org/10.1177/00420859221095000
10.
GrissomJ. A.ReddingC. (2016). Discretion and disproportionality: Explaining the underrepresentation of high-achieving student of color in gifted programs. AERA Open, 2(1), 1–25. https://doi.org/10.1177/2332858415622175
11.
GrissomJ. A.ReddingC.BleibergJ. F. (2019). Money over merit? Socioeconomic gaps in receipt of gifted services. Harvard Education Review, 89(3), 337–369. https://doi.org/10.17763/1943-5045-89.3.337
12.
HamiltonR.McCoachD. B.TutwilerM. S.SiegleD.GubbinsE. J.CallahanC. M.BrodersenA. V.MunR. U. (2018). Disentangling the roles of institutional and individual poverty in the identification of gifted students. Gifted Child Quarterly, 62(1), 6–24. https://doi.org/10.1177/0016986217738053
13.
HarlenW. (2005). Trusting teachers’ judgment: Research evidence of the reliability and validity of teachers’ assessment used for summative purposes. Research Papers in Education, 20(3), 245–270. https://doi.org/10.1080/02671520500193744
14.
HarradineC. C.ColemanM. R. B.WinnD.-M. C. (2014). Recognizing academic potential in students of color: Findings of U-STARS∼PLUS. Gifted Child Quarterly, 58(1), 24–34. https://doi.org/10.1177/0016986213506040
15.
HertzogN. B. (2009). The arbitrary nature of giftedness. In ShavininaL. V. (Ed.), International handbook on giftedness (pp. 205–214). Springer Science.
HospJ. L.ReschlyD. J. (2003). Referral rates for intervention or assessment: A meta-analysis of racial differences. Journal of Special Education, 37(2), 67–80. https://doi.org/10.1177/00224669030370020201
18.
LongD.McCoachD. B.SiegleD.CallahanC. M.GubbinsE. J. (2023). Inequality at the starting line: Under-representation in gifted identification and disparities in early achievement. AERA Open, 9(1), 1–25. https://doi.org/10.1177/23328584231171535
19.
MartínezJ. F.StecherB.BorkoH. (2009). Classroom assessment practices, teacher judgments, and student achievement in mathematics: Evidence from the ECLS. Educational Assessment, 14(2), 78–102. https://doi.org/10.1080/10627190903039429
20.
MashburnA. J.HamreB. K.DownerJ. T.PiantaR. C. (2006). Teacher and classroom characteristics associated with teachers’ ratings of prekindergartners’ relationships and behaviors. Journal of Psychoeducational Assessment, 24(4), 367–380. https://doi.org/10.1177/0734282906290594
21.
MatthewsM. S. (2018). Utilizing non-test assessments in identifying gifted and talented learners. In CallahanC. M.Hertberg-DavisH. L. (Eds.), Fundamentals of gifted education: Considering multiple perspectives (pp. 135–145). Routledge.
22.
McBeeM. T.PetersS. J.MillerE. M. (2016). The impact of the nomination stage on gifted program identification: A comprehensive psychometric analysis. Gifted Child Quarterly, 60(4), 258–278. https://doi.org/10.1177/0016986216656256
23.
McCoachD. B.CintronD. W. (2022). Quantitative toolkit series: Vol. 9. Introduction to modern modeling methods. Sage.
24.
McCoachD. B.NewtonS. D.GambinoA. J. (2022). Evaluating the fit and adequacy of multilevel models. In O’ConnellA. A.McCoachD. B.BellB. A. (Eds.), Multilevel modeling methods with introductory and advanced applications (pp. 51–94). Information Age Press.
25.
Meissel, K., Meyer, F., Yao, E. S., & Rubie-Davies, C. M. (2017). Subjectivity of teacher judgments: Exploring student characteristics that influence teacher judgments of student ability. Teaching and Teacher Education, 65, 48–60. https://doi.org/10.1016/j.tate.2017.02.021
26.
MillerE. M. (2009). The effect of training in gifted education on elementary classroom teachers’ theory-based reasoning about the concept of giftedness. Journal for the Education of the Gifted, 33(1), 65–105. https://doi.org/10.1177/016235320903300104
27.
MorganP. L.FarkasG.HillemeierM. M.MattisonR.MaczugaS.LiH.CookM. (2015). Minorities are disproportionately underrepresented in special education: Longitudinal evidence across five disability conditions. Educational Researcher, 44(5), 278–292. https://doi.org/10.3102/0013189X15591157
PereiraN. (2021). Finding talent among elementary English learners: A validity study of the HOPE teacher rating scale. Gifted Child Quarterly, 65(2), 153–166. https://doi.org/10.1177/0016986220985942
30.
PetersS. J.GentryM. (2010). Multigroup construct validity evidence of the HOPE scale: Instrumentation to identify low-income elementary students for gifted programs. Gifted Child Quarterly, 54(4), 298–313. https://doi.org/10.1177/0016986210378332
31.
PetersS. J.GentryM. (2012). Additional validity evidence and across-group equivalency of the HOPE Teacher Rating Scale. Gifted Child Quarterly, 57(2), 85–100. https://doi.org/10.1177/0016986212469253
32.
PetersS. J.GentryM.WhitingG. W.McBeeM. T. (2019). Who gets served in gifted education? Demographic proportionality and a call for action. Gifted Child Quarterly, 63(4), 273–287. https://doi.org/10.1177/0016986219833738
33.
PfeifferS. I. (2015). Essentials of Gifted Assessment. Wiley.
PfeifferS. I.JarosewichT. (2007). The gifted rating scales-school form: An analysis of the standardization sample based on age, gender, race, and diagnostic efficiency. Gifted Child Quarterly, 51(1), 39–50. https://doi.org/10.1177/0016986206296658
36.
Rambo-HernandezK. E.BrigandiC.AminS.SpillaneN. (2023). Who gets identified? The consequences of variability in teacher ratings and combination rules for determining eligibility for gifted services for young children. Journal for the Education of the Gifted, 46(2), 111–139. https://doi.org/10.1177/01623532231162612
37.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models. Applications and data analysis methods (2nd ed.). Sage Publications.
38.
RenzulliJ. S.SmithL. H.WhiteA. J.CallahanC. M.HartmanR. K.WestbergK. L.GavinM. K.ReisS. M.SiegleD.Sytsma ReedR. E. (2010). Scales for Rating the Behavioral Characteristics of Superior Students: Technical and administration manual (3rd ed.). Prufrock.
39.
RightsJ. D. (2022). Aberrant distortion of variance components in multilevel models under conflation of level-specific effects. Psychological Methods, Advance online publication. https://doi.org/10.1037/met0000514.supp
40.
RyserG. R.McConnellK. (2004). Scales for Identifying Gifted Students. Prufrock Press.
41.
SiegleD.McCoachD. B.GubbinsE. J.LongD.HamiltonR. (2018, January 9-10). Gifted identification gap: When just as good is not good enough. Annual Principal Investigators Meeting of the Institute for Education Science.
42.
SnijdersT. A. B.BoskerR. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd Edition). Sage.
43.
SplettJ. W.Smith-MillmanM.RabornA.BrannK. L.FlaspohlerP. D.MarasM. A. (2018). Student, teacher, and classroom predictors of between-teacher variance of students’ teacher-rated behavior. School Psychology Quarterly, 33(3), 460–468. https://doi.org/10.1037/spq0000241
44.
StarckJ. G.RiddleT.SinclairS.WarikooN. (2020). Teachers are people too: Examining the racial bias of teachers compared to other American adults. Educational Researcher, 49(4), 273–284. https://doi.org/10.3102/0013189X20912758
45.
VanTassel-BaskaJ. L. (2008). Epilogue: What do we know about identifying and assessing the learning of gifted students? In VanTassel-BaskaJ. L. (Ed.), Alternative assessment with gifted and talented students (pp. 309–319). Prufrock Press.
46.
VolpeR. J.DiPernaJ. C.HintzeJ. M.ShapiroE. S. (2005). Observing students in classroom settings: A review of seven coding schemes. School Psychology Review, 34(4), 454–474. https://doi.org/10.1080/02796015.2005.12088009
47.
WindS. A. (2020). Monotonicity as a nonparametric approach to evaluating rater fit in performance assessments. Measurement: Interdisciplinary Research and Perspectives, 18(3), 124–141. https://doi.org/10.1080/15366367.2020.1711679
48.
YaremychH. E.PreacherK. J.HedekerD. (2022). Centering categorical predictors in multilevel models: Best practices and interpretation. Psychological Methods, 28(3), 613–630. https://doi.org/10.1037/met0000434.supp
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
