Using Audio Recordings to Reliably and Efficiently Observe Teacher Behavior Related to Explicit Instruction

Abstract

Rubric-based observations of pre- and inservice teachers are common practice in schools. Popular observation tools often result in minimal variation in ratings between teachers, require extensive training and time demands for raters, and provide minimal feedback for professional development. Alternatively, direct observation methods are evidenced to effectively measure instructional behaviors. Applying direct observation to audio recordings would produce quantitative scores and provide valuable feedback to teachers about their instruction. As such, the purpose of the present pilot study was to examine the reliability and efficiency of using audio recordings to measure practices related to explicit instruction. Fleiss’s kappa was modeled to determine the reliability of multiple raters. Regression and correlation examined the strength and direction of the relationship between the full length of a teacher’s lesson and the first 20 min of the lesson. Results indicate that using audio recordings is reliable with kappas ranging from .45 to .80. Based on regression analyses, the first 20 min of a teacher’s lesson is predictive of the rates of behaviors observed in a full lesson. Correlations suggest large, positive relationships between rates of behaviors in the first 20 min and the full lesson. Recommendations for future studies of audio-recorded observations and progress monitoring teacher behavior are discussed.

Keywords

observation audio direct observation explicit instruction preservice

Get full access to this article

View all access options for this article.

References

Adamson

R. M.

Wachsmuth

S. T.

(2014). A review of direct observation research within the past decade in the field of emotional and behavioral disorders. Behavioral Disorders, 39(4), 181–189. https://doi.org/10.1177/019874291303900403

Altman

(1991). Practical statistics for medical research. Chapman and Hall.

Archer

Hughes

(2011). Explicit Instruction. Guilford.

Baker

Lesaux

Jayanthi

Dimino

Proctor

C. P.

Morris

. . . Newman-Gonchar

(2014). Teaching academic content and literacy to English learners in elementary and middle school (IES Practice Guide: NCEE 2014-4012). What Works Clearinghouse. https://ies.ed.gov/ncee/wwc/practiceguide/19

Bell

C. A.

Gitomer

D. H.

McCaffrey

D. F.

Hamre

B. K.

Pianta

R. C.

(2012). An argument approach to observation protocol validity. Educational Assessment, 17(2–3), 62–87. https://doi.org/10.1080/10627197.2012.715014

Berk

R. A.

(1979). Generalizability of behavioral observations: A clarification of interobserver agreement and interobserver reliability. American Journal of Mental Deficiency, 83(5), 460–472.

Brownell

M. T.

Billingsley

B. S.

McLeskey

Sindelar

P. T.

(2012). Teacher quality and effectiveness in an era of accountability: Challenges and solutions in special education? In Crockett

J. B.

Billingsley

Boscardin

M. L.

(Eds.), Handbook of leadership and administration for special education (pp. 269–289). Routledge.

Cameron

A. C.

Trivedi

P. K.

(2010). Microeconometrics using Stata. Stata Press.

Cohen

(1988). Statistical power analysis for the behavioral sciences. Lawrence Erlbaum.

10.

Cohen

Goldhaber

(2016). Building a more complete understanding of teacher evaluation using classroom observations. Educational Researcher, 45(6), 378–387. https://doi.org/10.3102/0013189X16659442

11.

Crowe

C. C.

Rivers

S. E.

Bertoli

M. C.

(2017). Mind the gap: Accountability, observation and special education. Assessment in Education: Principles, Policy & Practice, 24(1), 21–43. https://doi.org/10.1080/0969594X.2015.1114913

12.

Culbertson

(2012). Putting the value in teacher evaluation. Phi Delta Kappan, 94(3), 14–18. https://doi.org/10.1177/003172171209400304

13.

Curby

T. W.

Rimm-Kaufman

S. E.

Ponitz

C. C.

(2009). Teacher-child interactions and children’s achievement trajectories across kindergarten and first grade. Journal of Educational Psychology, 101, 912–925. https://doi.org/10.1037/a0016647

14.

Danielson

(2013). The framework for teaching evaluation instrument. The Danielson Group. https://danielsongroup.org

15.

Darling-Hammond

Jaquith

Hamilton

(2012). Creating a comprehensive system for evaluating and supporting effective teaching. Stanford Center for Opportunity Policy in Education. https://edpolicy.stanford.edu/sites/default/files/publications/creating-comprehensive-system-evaluating-and-supporting-effective-teaching.pdf

16.

Doabler

C. T.

Clarke

Kosty

Turtura

J. E.

Sutherland

Maddox

S. A.

Smolkowski

(2021). Using direct observation to document “practice-based evidence” of evidence-based mathematics instruction. Journal of Learning Disabilities, 54(1), 20–35. https://doi.org/10.1177/0022219420911375

17.

Downer

J. T.

López

M. L.

Grimm

K. J.

Hamagami

Pianta

R. C.

Howes

(2012). Observations of teacher-child interactions in classrooms serving Latinos and dual language learners: Applicability of the Classroom Assessment Scoring System in diverse settings. Early Childhood Research Quarterly, 27, 21–32. https://doi.org/10.1016/j.ecresq.2011.07.005

18.

Fleiss

J. L.

(1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619

19.

Floress

M. T.

Jenkins

L. N.

Reinke

W. M.

McKown

(2018). General education teachers’ natural rates of praise: A preliminary investigation. Behavioral Disorders, 43(4), 411–422. https://doi.org/10.1177/0198742917709472

20.

Gage

N. A.

Lewis

T. J.

Adamson

R. M.

(2010). An examination of 35 years of behavioral disorders: What, how, and who has been published. Behavioral Disorders, 35(4), 280–293. https://doi.org/10.1177/019874291003500402

21.

Hanushek

E. A.

Rivkin

S. G.

(2010). Generalizations about using value-added measures of teacher quality. American Economic Review, 100(2), 267–271. https://doi.org/10.1257/aer.100.2.267

22.

Harrop

Daniels

(1986). Methods of time sampling: A reappraisal of momentary time sampling and partial interval recording. Journal of Applied Behavior Analysis, 19, 73–77. https://doi.org/10.1901/jaba.1986.19-73

23.

Hawkins

S. M.

Heflin

L. J.

(2011). Increasing secondary teachers’ behavior-specific praise using a video self-modeling and visual performance feedback intervention. Journal of Positive Behavior Interventions, 13(2), 97–108. https://doi.org/10.1177/1098300709358110

24.

Hill

H. C.

Charalambous

C. Y.

Kraft

M. A.

(2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56–64. https://doi.org/10.3102/0013189X12437203

25.

Howes

Burchinal

Pianta

Brant

Early

Clifford

Barbarin

(2008). Ready to learn? Children’s pre-academic achievement in pre-kindergarten programs. Early Childhood Research Quarterly, 23, 27–50. https://doi.org/10.1016/j.ecresq.2007.05

26.

Hughes

C. A.

Morris

J. R.

Therrien

W. J.

Benson

S. K.

(2017). Explicit instruction: Historical and contemporary contexts. Learning Disabilities Research & Practice, 32(3), 140–148. https://doi.org/10.1111/ldrp.12142

27.

Jacob

McGovern

(2015). The mirage: Confronting the hard truth about our quest for teacher development. The New Teacher Project.

28.

Jones

N. D.

Brownell

M. T.

(2014). Examining the use of classroom observations in the evaluation of special education teachers. Assessment for Effective Intervention, 39(2), 112–124. https://doi.org/10.1177/1534508413514103

29.

Kane

T. J.

Staiger

D. O.

(2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains [Measures of Effective Teaching (MET) Project]. Bill & Melinda Gates Foundation.

30.

Kane

T. J.

Taylor

E. S.

Tyler

J. H.

Wooten

A. L.

(2011). Identifying effective classroom practices using student achievement data. Journal of Human Resources, 46(3), 587–613. https://doi.org/10.3368/jhr.46.3.587

31.

Keller

C. L.

Brady

M. P.

Taylor

R. L.

(2005). Using self-evaluation to improve student teacher interns’ use of specific praise. Education and Training in Developmental Disabilities, 40(4), 368–376.

32.

Kraft

M. A.

Gilmour

A. F.

(2017). Revisiting the widget effect: Teacher evaluation reforms and the distribution of teacher effectiveness. Educational Researcher, 46(5), 234–249. https://doi.org/10.3102/0013189X17718797

33.

Kyriakides

Creemers

B. P.

(2008). Using a multidimensional approach to measure the impact of classroom-level factors upon student achievement: A study testing the validity of the dynamic model. School Effectiveness and School Improvement, 19(2), 183–205. https://doi.org/10.1080/09243450802047873

34.

Landis

J. R.

Koch

G. G.

(1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33, 363–374. https://doi.org/10.2307/2529786

35.

La Paro

K. M.

Pianta

R. C.

Stuhlman

. (2004). The Classroom Assessment Scoring System: Findings from the prekindergarten year. The Elementary School Journal, 104, 409–426. https://doi.org/10.1086/499760

36.

Lewis

T. J.

Scott

T. M.

Wehby

J. H.

Wills

H. P.

(2014). Direct observation of teacher and student behavior in school settings: Trends, issues and future directions. Behavioral Disorders, 39(4), 190–200. https://doi.org/10.1177/019874291303900404

37.

Lindström

E. R.

Chow

J. C.

Zimmerman

K. N.

Zhao

Settanni

Ellison

(2021). A systematic review and meta-analysis of the relation between engagement and achievement in early childhood research. Topics in Early Childhood Special Education, 41(3), 221–235. https://doi.org/10.1177/02711214211032720

38.

Lylo

B. J.

Lee

D. L.

(2013). Effects of delayed audio-based self-monitoring on teacher completion of learning trials. Journal of Behavioral Education, 22(2), 120–138. https://doi.org/10.1007/s10864-012-9166-9

39.

MacSuga-Gage

A. S.

Simonsen

(2015). Examining the effects of teacher-directed opportunities to respond on student outcomes: A systematic review of the literature. Education and Treatment of Children, 38, 211–239. https://doi.org/10.1353/etc.2015.0009

40.

Mashburn

A. J.

Meyer

J. P.

Allen

J. P.

Pianta

R. C.

(2014). The effect of observation length and presentation order on the reliability and validity of an observational measure of teaching quality. Educational and Psychological Measurement, 74(3), 400–422. https://doi.org/10.1177/0013164413515882

41.

Mashburn

A. J.

Pianta

Hamre

B. K.

Downer

J. T.

Barbarin

Bryant

. . . Howes

(2008). Measures of classroom quality in pre-kindergarten and children’s development of academic, language and social skills. Child Development, 79, 732–749. https://doi.org/10.1111/j.1467-8624.2008.01154.x

42.

McKenna

J. W.

Shin

Ciullo

(2015). Evaluating reading and mathematics instruction for students with learning disabilities: A synthesis of observation research. Learning Disability Quarterly, 38(4), 195–207. https://doi.org/10.1177/0731948714564576

43.

McLeskey

Barringer

M.-D.

Billingsley

Brownell

Jackson

Kennedy

Lewis

Maheady

Rodriguez

Scheeler

M. C.

Winn

Ziegler

(2017). High-leverage practices in special education. CEC & CEEDAR Center.

44.

Morris-Mathews

Stark

K. R.

Jones

N. D.

Brownell

M. T.

Bell

C. A.

(2021). Danielson’s framework for teaching: Convergence and divergence with conceptions of effectiveness in special education. Journal of Learning Disabilities, 54(1), 66–78. https://doi.org/10.1177/0022219420941804

45.

The New Teacher Project. (2013). Fixing classroom observations: How common core will change the way we look at teaching. https://tntp.org/assets/documents/TNTP_FixingClassroomObservations_2013.pdf

46.

Palardy

G. J.

Rumberger

R. W.

(2008). Teacher effectiveness in first grade: The importance of background qualifications, attitudes, and instructional practices for student learning. Educational Evaluation and Policy Analysis, 30(2), 111–140. https://doi.org/10.3102/0162373708317680

47.

Pianta

R. C.

Howes

Burchinal

Bryant

Clifford

Early

Barbarin

(2005). Features of pre-kindergarten programs, classrooms, and teachers: Do they predict observed classroom quality and child-teacher interactions? Applied Developmental Science, 9, 144–159. https://doi.org/10.1207/s1532480xads0903_2

48.

Pianta

R. C.

La Paro

K. M.

Hamre

B. K.

(2008). Classroom Assessment Scoring System™: Manual K-3. Paul H. Brookes.

49.

Pua

D. J.

Peyton

D. J.

Brownell

M. T.

Contesse

V. A.

Jones

N. D.

(2021). Preservice observation in special education: A validation study. Journal of Learning Disabilities, 54(1), 6–19. https://doi.org/10.1177/0022219420920382

50.

Pustejovsky

J. E.

Swan

D. M.

(2015). Four methods for analyzing partial interval recording data, with application to single-case research. Multivariate Behavioral Research, 50(3), 365–380. https://doi.org/10.1080/00273171.2015.1014879

51.

Reinke

W. M.

Lewis-Palmer

Merrell

(2008). The classroom check-up: A classwide teacher consultation model for increasing praise and decreasing disruptive behavior. School Psychology Review, 37(3), 315–332. https://doi.org/10.1080/02796015.2008.12087879

52.

Smith

Snow

Serry

Hammond

(2021). The role of background knowledge in reading comprehension: A critical review. Reading Psychology, 42(3), 214–240. https://doi.org/10.1080/02702711.2021.1888348

53.

Sutherland

K. S.

Wehby

J. H.

(2001). Exploring the relationship between increased opportunities to respond to academic requests and the academic and behavioral outcomes of students with EBD: A review. Remedial and Special Education, 22(2), 113–121. https://doi.org/10.1177/074193250102200205

54.

Sutherland

K. S.

Wehby

J. H.

Yoder

P. J.

(2002). Examination of the relationship between teacher praise and opportunities for students with EBD to respond to academic requests. Journal of Emotional and Behavioral Disorders, 10(1), 5–13. https://doi.org/10.1177/106342660201000102

55.

Swanson

E. A.

(2008). Observing reading instruction for students with learning disabilities: A synthesis. Learning Disability Quarterly, 31(3), 115–133. https://doi.org/10.2307/254746

56.

Tapp

Wehby

Ellis

(1995). A multiple option observation system for experimental studies: MOOSES. Behavior Research Methods, Instruments, & Computers, 27(1), 25–31. https://doi.org/10.3758/BF03203616

57.

Taylor

E. S.

Tyler

J. H.

(2011). The effect of evaluation on performance: Evidence from longitudinal student achievement data of mid-career teachers (No. w16877). National Bureau of Economic Research.

58.

Weisberg

Sexton

Mulhern

Keeling

Schunck

Palcisco

Morgan

(2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. The New Teacher Project.

59.

Whitcomb

(2014). Review of fixing classroom observations. National Education Policy Center. https://nepc.info/sites/default/files/ttr-teval-tntp.pdf

60.

Yoder

P. J.

Ledford

J. R.

Harbison

A. L.

Tapp

J. T.

(2018). Partial-interval estimation of count: Uncorrected and Poisson-corrected error levels. Journal of Early Intervention, 40(1), 39–51. https://doi.org/10.1177/1053815117748407

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB