This study assesses the reliability of two observation rubrics, one in math and the other in science, and documents how the rubric data were used to inform a teacher education program. Classroom observations are typically considered essential for assessing teaching practice, yet many popular observation frameworks, while comprehensive in aim, do not appropriately capture key features of teaching valued by teacher education programs. Many of these tools do not attend to issues of equity, humanizing pedagogy, and thus, social justice. We report on the development of two observation rubrics—secondary math and science—that embody the aims and values of our teacher education program, specifically, equity and humanizing pedagogy, and the results of our examination of the reliability of ratings of teaching practice generated using these rubrics. We discuss the various sources of measurement error and the implications for further developing and using the observation rubric in our program.
AbediJ. (2005). Issues and consequences for English language learners. In HermanJ. L.HaertelE. H. (Eds.), The 104th yearbook of the national society for the study of education, part 2: Uses and misuses of data for educational accountability and improvement (pp. 175-198). Malden, MA: Blackwell.
2.
AllenJ. P.PiantaR. C.GregoryA.MikamiA. Y.LunJ. (2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333(6045), 1034-1037.
3.
AnyonJ. (2006). Social class and the hidden curriculum of work. Childhood socialization, 162, 369.
4.
ArbaughF.BrownC. A. (2005). Analyzing mathematical tasks: A catalyst for change?Journal of Mathematics Teacher Education, 8(6), 499-536.
5.
BartoloméL. (1994). Beyond the methods fetish: Toward a humanizing pedagogy. Harvard Educational Review, 64(2), 173-195.
6.
BergerR. (2003). An ethic of excellence: Building a culture of craftsmanship with students. Portsmouth, NH: Heinemann.
7.
BodzinA. M.BeererK. M. (2003). Promoting inquiry-based science instruction: The validation of the Science Teacher Inquiry Rubric (STIR). Journal of Elementary Science Education, 15(2), 39-49.
8.
BostonM.BosticJ.LesseigK.ShermanM. (2015). A comparison of mathematics classroom observation protocols. Mathematics Teacher Educator, 3(2), 154-175.
9.
BrennanR. L. (2001). An essay on the history and future of reliability from the perspective of replications. Journal of Educational Measurement, 38, 295-317.
10.
BrownB. (2006). “It isn’t no slang that can be said about this stuff”: Language, identity, and appropriating science discourse. Journal of Research in Science Teaching, 43(1), 96-126.
11.
CarrasquilloA.KucerS.AbramsR. (2004). Beyond the beginnings: Literacy interventions for upper elementary English language learners. Clevedon, UK: Multilingual Matters.
12.
ChiuC. W. T.WolfeE. W. (2002). A method for analyzing sparse data matrices in the generalizability theory framework. Applied Psychological Measurement, 26(3), 321-338.
13.
CicchettiD. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284-290.
14.
Cochran-SmithM.ShakmanK.JongC.TerrellD. G.BarnattJ.McQuillanP. (2009). Good and just teaching: The case for social justice in teacher education. American Journal of Education, 115(3), 347-377.
15.
Council of Chief State School Officers. (2010). Common core state standards for English language arts & literacy in history/social studies, science, and technical subjects. Washington, DC: Author.
16.
CumminsJ. (1999). BICS and CALP: Clarifying the distinction (Opinion Paper ED438551). Washington, DC: Education Resource Information Center.
Darling-HammondL. (2013). Getting teacher evaluation right: What really matters for effectiveness and improvement. New York, NY: Teachers College Press.
19.
ErduranS.SimonS.OsborneJ. (2004). TAPping into argumentation: Developments in the application of Toulmin’s argument pattern for studying science discourse. Science Education, 88(6), 915-933.
20.
ForzaniF. M. (2014). Understanding “core practices” and “practice-based” teacher education: Learning from the past. Journal of Teacher Education, 65(4), 357-368.
21.
FrankeM. L.KazemiE.BatteyD. (2007). Mathematics teaching and classroom practice. In LesterF. K. (Ed.), Second handbook of research on mathematics teaching and learning (Vol. 1, pp. 225-256). Charlotte, NC: National Council of Teachers of Mathematics.
22.
FreireP. (2000). Pedagogy of the oppressed: With an introduction by Donaldo Macedo. New York, NY: Continuum.
23.
GirouxH. A. (1988). Teachers as intellectuals: Toward a critical pedagogy of learning. Westport, CT: Bergin & Garvey.
24.
GonzálezN.MollL. C.AmantiC. (Eds.). (2013). Funds of knowledge: Theorizing practices in households, communities, and classrooms. New York, NY: Routledge.
25.
HiebertJ.GrouwsD. A. (2007). The effects of classroom mathematics teaching on students’ learning. In LesterF. K. (Ed.), Second handbook of research on mathematics teaching and learning (pp. 371-404). Greenwich, CT: Information Age.
26.
HillH. C.CharalambousC. Y.KraftM. A. (2012). When rater reliability is not enough teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56-64.
27.
HoA. D.KaneT. J. (2013). The reliability of classroom observations by school personnel (Research paper, MET Project). Seattle, WA: Bill & Melinda Gates Foundation.
28.
HoferB. K.PintrichP. R. (Eds.). (2004). Personal epistemology: The psychology of beliefs about knowledge and knowing. Hove, UK: Psychology Press.
29.
KaneT. J.StaigerD. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains (Research paper, MET Project). Seattle, WA: Bill & Melinda Gates Foundation.
30.
KerstingN. (2008). Using video clips of mathematics classroom instruction as item prompts to measure teachers’ knowledge of teaching mathematics. Educational and Psychological Measurement, 68(5), 845-861.
31.
KilpatrickJ.SwaffordJ.FindellB. (Eds.). (2001). Adding it up: Helping children learn mathematics. Washington, DC: National Academy Press.
32.
KnightT. (2001). Longitudinal development of educational theory: Democracy and the classroom. Journal of Education Policy, 16(3), 249-263.
33.
Knorr-CetinaK. (1999). Epistemic cultures: How the sciences make knowledge. Cambridge, MA: Harvard University Press.
34.
MaggioniL.ParkinsonM. M. (2008). The role of teacher epistemic cognition, epistemic beliefs, and calibration in instruction. Educational Psychology Review, 20(4), 445-461.
MarshallJ. C.SmartJ.HortonR. M. (2010). The design and validation of EQUIP: An instrument to assess inquiry-based instruction. International Journal of Science and Mathematics Education, 8(2), 299-321.
37.
MarshallJ. C.SmartJ.LotterC.SirbuC. (2011). Comparative analysis of two inquiry observational protocols: Striving to better understand the quality of teacher-facilitated inquiry-based instruction. School Science and Mathematics, 111(6), 306-315.
McDonaldM.KazemiE.KavanaghS. S. (2013). Core practices and pedagogies of teacher education: A call for a common language and collective activity. Journal of Teacher Education, 64(5), 378-386.
40.
National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: Author.
41.
Next Generation Science Standards Lead States. (2013). Next Generation Science Standards: For states, by states (Vol. 1: The Standards). Washington, DC: The National Academies Press.
42.
NietoS. (2002). Language, culture, and teaching critical perspectives for a new century. Mahwah, NJ: Lawrence Erlbaum.
43.
PiantaR. C.HamreB. K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38(2), 109-119.
44.
PiburnM.SawadaD.TurleyJ.FalconerK.BenfordR.BloomI.JudsonE. (2000). Reformed Teaching Observation Protocol (RTOP) reference manual. Tempe: Arizona Collaborative for Excellence in the Preparation of Teachers.
45.
QuartzKH. (2009). The careers of urban teachers: A synthesis of findings from UCLA’s longitudinal study of urban educators. In BayerM.BrinkkjaerU.PlauborgH.RollsS., (Eds.), Teachers’ Career Trajectories and Work Lives, (pp.143-158). New York, NY: Springer
46.
RogoffB. (1994). Developing understanding of the idea of communities of learners. Mind, Culture and Activity, 1(4), 209-229.
47.
SteinM. K.KaufmanJ. H.ShermanM.HillenA. F. (2011). Algebra: A challenge at the crossroads of policy and practice. Review of Educational Research, 81(4), 453-492.
48.
TomlinsonC. A. (2001). How to differentiate instruction in mixed- ability classrooms (2nd ed., Professional Development). Alexandria, VA: Association for Supervision & Curriculum Development.
49.
VygotskyL. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.
50.
WalkingtonC.AroraP.IhornS.GordonJ.WalkerM.AbrahamL.MarderM. (2012). Development of the UTeach observation protocol: A classroom observation instrument to evaluate mathematics and science teachers from the UTeach preparation program (Unpublished paper). Southern Methodist University.
51.
WareJ. E.GandekB. (1998). Methods for testing data quality, scaling assumptions, and reliability: The IQOLA project approach. Journal of Clinical Epidemiology, 51(11), 945-952.
52.
WebbN. M.ShavelsonR. J.HaertelE. H. (2006). Reliability coefficients and generalizability theory. Handbook of Statistics, 26, 81-124.
53.
WeinsteinC.NovodvorskyI. (2015). Middle and secondary classroom management: Lessons from research and practice. New York, NY: McGraw-Hill.
54.
WellsG. (2009). The meaning makers: Learning to talk and talking to learn (Vol. 15). Bristol, UK: Multilingual Matters.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
