The Effect of Observation Length and Presentation Order on the Reliability and Validity of an Observational Measure of Teaching Quality

Abstract

Observational methods are increasingly being used in classrooms to evaluate the quality of teaching. Operational procedures for observing teachers are somewhat arbitrary in existing measures and vary across different instruments. To study the effect of different observation procedures on score reliability and validity, we conducted an experimental study that manipulated the length of observation and order of presentation of 40-minute videotaped lessons from secondary grade classrooms. Results indicate that two 20-minute observation segments presented in random order produce the most desirable effect on score reliability and validity. This suggests that 20-minute occasions may be sufficient time for a rater to observe true characteristics of teaching quality assessed by the measure used in the study, and randomizing the order in which segments were rated may reduce construct irrelevant variance arising from carry over effects and rater drift.

Keywords

observational measures generalizability theory reliability validity

Get full access to this article

View all access options for this article.

References

Allen

Pianta

Gregory

Mikami

Lun

(2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333, 1034-1037.

Bell

C. A.

Gitomer

D. H.

McCaffrey

D. F.

Hamre

B. K.

Pianta

R. C.

(2012). An argument approach to observation protocol validity. Educational Assessment, 17, 62-87. doi:10.1080/10627197.2012.715014

Bill and Melinda Gates Foundation. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle, WA: Author. Retrieved from http://www.metproject.org/downloads/MET_Gathering_Feedback_Practioner_Brief.pdf

Brennan

R. L.

(1992). Generalizability theory: An NCME instructional module. Educational Measurement: Issues and Practice, 11, 27-34.

Casabianca

Lockwood

J. R.

(2013, March). Rater drift and time trends in teaching observations. Paper presented at the annual meeting of the Society for Research on Educational Effectiveness, Washington, DC.

Congdon

P. J.

McQueen

(2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37, 163-178.

Cronbach

L. J.

Gleser

G. C.

Nanda

Rajaratnam

(1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.

Danielson

(2011). The framework for teaching evaluation instrument. Princeton, NJ: Danielson Group.

Ebbinghaus

(1913). On memory: A contribution to experimental psychology. New York, NY: Teacher’s College Press.

10.

Erlich

Borich

(1979). Occurrence and generalizability of scores on a classroom interaction instrument. Journal of Educational Measurement, 16, 11-18.

11.

Fritz

Chen

(2013, February). Federal grants, rise of charter schools expand teacher evaluations. Retrieved from http://www.pbs.org/newshour/rundown/2013/02/teacher-evaluations.html

12.

Grossman

Loeb

Cohen

Hammerness

Wyckoff

Boyd

Lankford

(2010). Measure for measure: The relationship between measures of instructional practice in middle school English language arts and teachers’ value-added scores (National Bureau of Economic Research Working Paper No. 16015). Cambridge, MA: National Bureau of Economic Research.

13.

Hamre

Pianta

Downer

Hamagami

Mashburn

Jones

. . . Brackett

(2013). Teaching through interactions: Testing a developmental framework of teacher effectiveness in over 4,000 classrooms. Elementary School Journal, 113(4), 461-487.

14.

Heitin

(2011, October 26). Study: States’ teacher-evaluation policies are A-changin’. Education Week. Retrieved from http://blogs.edweek.org/edweek/teacherbeat/2011/10/study_states_teacher_evaluatio.html

15.

Hill

H. C.

Blunk

Charalambous

Lewis

Phelps

G. C.

Sleep

Ball

D. L.

(2008). Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition and Instruction, 26, 430-511.

16.

Hill

H. C.

Charalambous

C. Y.

Kraft

M. A.

(2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41, 56-64.

17.

Hintze

J. M.

(2005). Psychometrics of direct observation. School Psychology Review, 34, 507-519.

18.

A. D.

Kane

T. J.

(2013). The reliability of teaching observations by school personnel. Seattle, WA: Bill and Melinda Gates Foundation.

19.

Kane

M. T.

(1982). A sampling model for validity. Applied Psychological Measurement, 6, 125-160.

20.

Kane

M. T.

(2011). The errors of our ways. Journal of Educational Measurement, 48, 12-30.

21.

Leventhal

Turcotte

S. J.

Abrami

P. C.

Perry

R. P.

(1983). Primacy/recency effects in student ratings of instruction: A reinterpretation of gain-loss effects. Journal of Educational Psychology, 75, 692-704.

22.

Malmberg

Hagger

Burn

Mutton

Colls

(2010). Observed classroom quality during teacher education and two years of professional practice. Journal of Educational Psychology, 102, 916-932.

23.

Marcoulides

(1989a). The application of generalizability analysis to observational studies. Quality & Quantity, 23, 115-127. doi:10.1007/BF00151898

24.

Marcoulides

G. A.

(1989b). Performance appraisal: Issues of validity. Performance Improvement Quarterly, 2, 3-12.

25.

Mashburn

A. J.

Downer

J. D.

Hamre

B. K.

Justice

L. M.

Pianta

R. C.

(2010). Teaching consultation and children’s language and literacy development during pre-kindergarten. Applied Developmental Science, 14, 179-198.

26.

Mashburn

A. J.

Downer

J. T.

Rivers

S. E.

Brackett

M. A.

Martinez

(2013). Improving the power of an efficacy study of a social and emotional learning program: Application of generalizability theory to the measurement of classroom-level outcomes. Prevention Science. Advance online publication. doi:10.1007/s11121-012-0357-3

27.

Mashburn

A. J.

Pianta

R. C.

Hamre

B. K.

Downer

J. T.

Barbarin

Bryant

. . . Howes

(2008). Measures of pre-k quality and children’s development of academic, language and social skills. Child Development, 79, 732-749. doi:10.1111/j.1467-8624. 2008.01154.x

28.

McCaffrey

Yuan

Savitsky

Lockwood

J. R.

Edelen

M.O.

(2013, February). Using latent hierarchical estimation to uncover multivariate structure in teaching observation protocol data. Paper presented at the annual meeting of the Society for Research on Educational Effectiveness, Washington, DC.

29.

Meyer

J. P.

Henry

A. E.

Mashburn

A. J.

(2011). Occasions and the reliability of teaching observations: Alternative conceptualizations and methods of analysis. Educational Assessment Journal, 16, 227-243. doi:10.1080/10627197.2011.638884

30.

Mikami

Gregory

Allen

J. P.

Pianta

R. C.

Lun

(2011). Effects of a teacher professional development intervention on peer relationships in secondary classrooms. School Psychology Review, 40, 367-385.

31.

Pianta

R. C.

Hamre

Hayes

Mintz

LaParo

K. M.

(2008). Classroom Assessment Scoring System–Secondary (CLASS-S). Charlottesville, VA: University of Virginia.

32.

Savitsky

T. T.

McCaffrey

D. F.

(2013, February). Bayesian hierarchical multivariate formulation with factor analysis for nested ordinal data. Paper presented at the annual meeting of the Society for Research on Educational Effectiveness, Washington, DC.

33.

U.S. Department of Education. (2009). Race to the top executive summary. Washington, DC: Author. Retrieved from http://www2.ed.gov/programs/racetothetop/executive-summary.pdf