Initial Development of the Child Responsiveness Scale for Early Childhood Settings

Abstract

This article presents initial psychometric data for scores on the Child Responsiveness Scale (CRS), an observational measure to assess child responsiveness to teacher delivery of practices designed to promote social-emotional competencies. In Study 1, 633 live observations conducted with 54 teachers and 91 children at risk for emotional and behavioral disorders in early childhood classrooms were rated independently by two coders. Interrater reliability, ICC(2,2), for the CRS item scores, averaged .67 (SD = .06; range .55 to .72). The CRS scale scores demonstrated sensitivity to variation across teachers, children, and coders. In Study 2, 634 recorded observations with 52 teachers and 52 children in early childhood classrooms were rated independently by two coders. Interrater reliability, ICC(2,2), for the item scores, averaged .62 (SD = .08; range .53 to .74). An exploratory factor analysis and a confirmatory factor analysis conducted on separate samples indicated that the items were best represented as two correlated factors, one for positive and one for negative items. Analyses provided preliminary evidence for CRS scale score validity. Future research directions are discussed regarding using the CRS to evaluate child responsiveness’s role in promoting child outcomes in early childhood classrooms.

Keywords

teacher–child interactions early childhood measure development treatment integrity child responsiveness

Get full access to this article

View all access options for this article.

References

Acar

I. H.

Rudasill

K. M.

Molfese

Torquati

Prokasky

(2015). Temperament and preschool children’s peer interactions. Early Education and Development, 26(4), 479–495. https://doi.org/10.1080/10409289.2015.1000718

Asparouhov

(2005). Sampling weights in latent variable modeling. Structural Equation Modeling, 12(3), 411–434. https://doi.org/10.1207/s15328007sem1203_4

Barber

J. P.

Foltz

Crits-Christoph

Chittams

(2004). Therapists’ adherence and competence and treatment discrimination in the NIDA collaborative cocaine treatment study. Journal of Clinical Psychology, 60(1), 29–41. https://doi.org/10.1002/(ISSN)1097-4679

Bartlett

M. S.

(1951). The effect of standardization on a Chi-square approximation in factor analysis. Biometrika, 38, 337–344. https://doi.org/10.2307/2332580

Carbonneau

K. J.

Van Orman

D. S.

Lemberger-Truelove

M. E.

Atencio

D. J.

(2019). Leveraging the power of observations: Locating the sources of error in the Individualized Classroom Assessment Scoring System. Early Education and Development, 31(1), 1–16. https://doi.org/10.1080/10409289.2019.1617572

Cavell

T. A.

Elledge

L. C.

Malcolm

K. T.

Faith

M. A.

Hughes

J. N.

(2009). Relationship quality and the mentoring of aggressive, high-risk children. Journal of Clinical Child & Adolescent Psychology, 38(2), 185–198. https://doi.org/10.1080/15374410802698420

Chu

B. C.

Kendall

P. C.

(2004). Positive association of child involvement and treatment outcome within a manual-based cognitive-behavioral treatment for children with anxiety. Journal of Consulting and Clinical Psychology, 72(5), 821–829. https://doi.org/10.1037/0022-006X.72.5.821

Cicchetti

D. V.

(1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284

Collyer

Eisler

Woolgar

(2020). Systematic literature review and meta-analysis of the relationship between adherence, competence and outcome in psychotherapy for children and adolescents. European Child & Adolescent Psychiatry, 29(4), 417–431. https://doi.org/10.1007/s00787-018-1265-2

10.

Conroy

M. A.

Sutherland

K. S.

Algina

Ladwig

Werch

Martinez

Jessee

Gyure

(2019). Outcomes of the BEST in CLASS intervention on teachers’ use of effective practices, self-efficacy, and classroom quality. School Psychology Review, 48(1), 31–45. https://doi.org/10.17105/SPR-2018-0003.V48-1

11.

Dane

A. V.

Schneider

B. H.

(1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45. https://doi.org/10.1016/s0272-7358(97)00043-3

12.

Doss

B. D.

(2004). Changing the way we study change in psychotherapy. Clinical Psychology: Science and Practice, 11, 368–386. https://doi.org/10.1093/clipsy.bph094

13.

Downer

J. T.

Booren

L. M.

Lima

O. K.

Luckner

A. E.

Pianta

R. C.

(2010). The Individualized Classroom Assessment Scoring System (inCLASS): Preliminary reliability and validity of a system for observing preschoolers’ competence in classroom interactions. Early Childhood Research Quarterly, 25(1), 1–16. https://doi.org/10.1016/j.ecresq.2009.08.004

14.

Elvins

Green

(2008). The conceptualization and measurement of therapeutic alliance: An empirical review. Clinical Psychology Review, 28(7), 1167–1187. https://doi.org/10.1016/j.cpr.2008.04.002

15.

Feil

E. G.

Frey

A. J.

Walker

H. M.

Seeley

J. R.

Golly

Small

J. W.

(2014). The efficacy of a home-school intervention for preschoolers with challenging behaviors: A randomized controlled trial of Preschool First Step to Success. Journal of Early Intervention, 36(3), 151–170. https://doi.org/10.1177/1053815114566090

16.

Feil

E. G.

Severson

H. H.

Walker

H. M.

(1998). Screening for emotional and behavioral delays: The Early Screening Project. Journal of Early Intervention, 21(3), 252–266. https://doi.org/10.1177/105381519802100306

17.

Fixsen

D. L.

Naoom

S. F.

Blase

K. A.

Friedman

R. M.

Wallace

(2005). Implementation research: A synthesis of the literature. The National Implementation Research Network.

18.

Fleiss

J. L.

(1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5(1), 105–112. https://doi.org/10.1177/014662168100500115

19.

Hogue

Dauber

Henderson

C. E.

Liddle

H. A.

(2014). Reliability of therapist self-report on treatment targets and focus in family-based intervention. Administration and Policy in Mental Health Services Research, 41(5), 697–705. https://doi.org/10.1007/s10488-013-0520-6

20.

Hogue

Liddle

H. A.

Rowe

(1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy: Theory, Research, Practice, Training, 33(2), 332–345. https://doi.org/10.1037/0033-3204.33.2.332

21.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185. https://doi.org/10.1007/BF02289447

22.

L.-T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

23.

Humphrey

Hennessey

Lendrum

Wigelsworth

Turner

Panayiotou

Joyce

Pert

Stephens

Squires

Woods

Harrison

Calam

(2018). The PATHS curriculum for promoting social and emotional well-being among children aged 7–9 years: A cluster RCT. Public Health Research, 6(10), 1–116. https://doi.org/10.3310/phr06100

24.

Kaiser

H. F.

Rice

(1974). Little Jiffy, Mark IV. Educational and Psychological Measurement, 34(1), 111–117. https://doi.org/10.1177/001316447403400115

25.

Karver

M. S.

Handelsman

J. B.

Fields

Bickman

(2005). A theoretical model of common process factors in youth and family therapy. Administration and Policy in Mental Health Services Research, 7, 35–51. https://doi.org/10.1007/s11020-005-1964-4

26.

Koo

T. K.

M. Y.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012.

27.

Margolin

Oliver

Gordis

O’Hearn

Medina

Ghosh

Morland

(1998). The nuts and bolts of behavioral observation of marital and family interaction. Clinical Child & Family Psychology Review, 1(4), 195–213. https://doi.org/10.1023/a:1022608117322

28.

McLeod

B. D.

(2011). The relation of the alliance with outcomes in youth psychotherapy: A meta-analysis. Clinical Psychology Review, 31, 603–616. https://doi.org/10.1016/j.cpr.2011.02.001

29.

McLeod

B. D.

Islam

N. Y.

Wheat

(2013). Designing, conducting, and evaluating therapy process research. In Comer

Kendall

(Eds.), The Oxford handbook of research strategies for clinical psychology (pp. 142–164). Oxford University Press.

30.

McLeod

B. D.

Porter

Hogue

Becker-Haimes Jensen-Doss

(2023). What is the status of multi-informant treatment fidelity research? Journal of Clinical Child and Adolescent Psychology, 52(1), 74–94. https://doi/10.1080/15374416.2022.2151713

31.

McLeod

B. D.

Smith

M. M.

Southam-Gerow

M. A.

Weisz

J. R.

Kendall

P. C.

(2015). Measuring differentiation for implementation research: The Therapy Process Observational Coding System for Child Psychotherapy Revised Strategies scale. Psychological Assessment, 27(1), 314–325. https://doi.org/10.1037/pas0000037

32.

McLeod

B. D.

Southam-Gerow

M. A.

Bair

C. E.

Rodriguez

Smith

M. M.

(2013). Making a case for treatment integrity as a psychological treatment quality indicator. Clinical Psychology: Science and Practice, 20(1), 14–32. https://doi.org/10.1111/cpsp.12020

33.

McLeod

B. D.

Southam-Gerow

M. A.

Rodriguez

Quinoy

Arnold

Kendall

P. C.

Weisz

J. R.

(2018). Development and initial psychometrics for a therapist competence instrument for CBT for youth anxiety. Journal of Clinical Child and Adolescent Psychology, 47(1), 47–60. https://doi.org/10.1080/15374416.2016.1253018

34.

McLeod

B. D.

Southam-Gerow

M. A.

Weisz

J. R.

(2009). Conceptual and methodological issues in treatment integrity measurement. School Psychology Review, 38(4), 541–546.

35.

McLeod

B. D.

Sutherland

K. S.

(2015). Scoring manual for the Child Responsiveness Scale. Unpublished scoring manual prepared at Virginia Commonwealth University, Richmond, VA.

36.

McLeod

B. D.

Sutherland

K. S.

Broda

Granger

K. L.

Martinez

R. G.

Conroy

M. A.

Snyder

P. A.

Southam-Gerow

M. A.

(2022). Development and initial psychometrics of a generic treatment integrity measure designed to assess practice elements of evidence-based interventions for early childhood settings. Prevention Science, 23, 488–501. https://doi.org/10.1007/s11121-021-01304-w

37.

McLeod

B. D.

Weisz

J. R.

(2005). The Therapy Process Observational Coding System Alliance Scale: Measure characteristics and prediction of outcome in usual clinical practice. Journal of Consulting and Clinical Psychology, 73(2), 323–333. https://doi.org/10.1037/0022-006X.73.2.323

38.

McNeish

Wolf

M. G.

(2023). Dynamic fit index cutoffs for one-factor models. Behavioral Research, 55, 1157–1174.

39.

Morris

Fitzpatrick

M. R.

Renaud

(2016). A pan-theoretical conceptualization of client involvement in psychotherapy. Psychotherapy Research, 26(1), 70–84. https://doi.org/10.1080/10503307.2014.935521

40.

Muthén

L. K.

Muthén

B. O.

(2017). Mplus user’s guide (8th ed.).

41.

Myers

S. M.

Pianta

R. C.

(2008). Developmental commentary: Individual and contextual influences on student-teacher relationships and children’s early problem behaviors. Journal of Clinical Child and Adolescent Psychology, 37(3), 600–608. https://doi.org/10.1080/15374410802148160

42.

Newborg

(2005). Battelle Developmental Inventory (2nd ed.). Riverside.

43.

Nobre

J. S.

da Motta Singer

(2007). Residual analysis for linear mixed models. Biometrical Journal, 49(6), 863–875. https://doi.org/10.1002/bimj.200610341

44.

Perepletchikova

Kazdin

A. E.

(2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science & Practice, 12(4), 365–383. https://doi.org/10.1093/clipsy.bpi045

45.

Raver

C. C.

Knitzer

(2002). Ready to enter: What research tells policymakers about strategies to promote social and emotional school readiness among three- and four-year-old children (Promoting the Emotional Well-being of Children and Families Policy Paper No. 3). National Center for Children in Poverty.

46.

R Core Team. (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

47.

Revelle

(2023). psych: Procedures for psychological, psychometric, and personality research (R package version 2.3). Northwestern University. https://CRAN.R-project.org/package=psych

48.

Roorda

D. L.

Koomen

H. M. Y.

Spilt

J. L.

Oort

F. J.

(2011). The influence of affective teacher–student relationships on students’ school engagement and achievement: A meta-analytic approach. Review of Educational Research, 81(4), 493–529. https://doi.org/10.3102/0034654311421793

49.

Rosenthal

Rosnow

R. L.

(1984). Essentials of behavioral research: Methods and data analysis. McGraw-Hill.

50.

Sanetti

L. M. H.

Charbonneau

Knight

Cochrane

W. S.

Kulcyk

M. C. M.

Kraus

K. E.

(2020). Treatment fidelity reporting in intervention outcome studies in the school psychology literature from 2009 to 2016. Psychology in the Schools, 57(6), 901–922. https://doi.org/10.1002/pits.22364

51.

Sanetti

L. M. H.

Kratochwill

T. R.

(2007). Treatment integrity in behavioral consultation: Measurement, promotion, and outcomes. International Journal of Behavioral Consultation and Therapy, 4(1), 95–113. https://doi.org/10.1037/h0100835

52.

Sanetti

L. M. H.

Kratochwill

T. R.

(2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445–459.

53.

Satorra

Bentler

P. M.

(2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66(4), 507–514. https://doi.org/10.2139/ssrn.199064

54.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420

55.

Southam-Gerow

M. A.

McLeod

B. D.

Arnold

C. A.

Rodriguez

Cox

J. R.

Reise

S. P.

Bonifay

W. E.

Weisz

J. R.

Kendall

P. C.

(2016). Initial development of a treatment adherence measure for cognitive-behavioral therapy for child anxiety. Psychological Assessment, 28(1), 70–80. https://doi.org/10.1037/pas0000141

56.

Sutherland

K. S.

Conroy

M. A.

Algina

Ladwig

Jessee

Gyure

(2018). Reducing child problem behaviors and improving teacher-child interactions and relationships: A randomized controlled trial of BEST in CLASS. Early Childhood Research Quarterly, 42(1), 31–43. https://doi.org/10.1016/j.ecresq.2017.08.001

57.

Sutherland

K. S.

Conroy

M. A.

McLeod

B. D.

Algina

(2018). Teacher competence of delivery of BEST in CLASS as a mediator of treatment effects. School Mental Health, 10, 214–225. https://doi.org/10.1007/s12310-017-9224-5

58.

Sutherland

K. S.

Granger

Conroy

McLeod

B. D.

Vallarta

Rosas

(2023). Examining the role of student responsiveness in treatment effects of a tier 2 program targeting reductions in problem behavior. Prevention Science, 24, 974–984.

59.

Sutherland

K. S.

McLeod

B. D.

(2015). Scoring manual for the treatment integrity measure for early childhood settings: The adherence and competence scale. Unpublished scoring manual prepared at Virginia Commonwealth University, Richmond, VA.

60.

Sutherland

K. S.

McLeod

B. D.

(2022). Advancing the science of integrity measurement in school mental health research. School Mental Health, 14(1), 1–6. https://doi.org/10.1007/s12310-021-09468-0

61.

Sutherland

K. S.

McLeod

B. D.

Conroy

(2022). Developing treatment integrity measures for teacher-delivered interventions: Progress, recommendations, and future directions. School Mental Health, 14(1), 7–19.

62.

Sutherland

K. S.

McLeod

B. D.

Conroy

M. A.

Cox

J. R.

(2013). Measuring implementation of evidence-based programs targeting young children at risk for emotional/behavioral disorders: Conceptual issues and recommendations. Journal of Early Intervention, 35(2), 129–149. https://doi.org/10.1177/10538151135150

63.

Sutherland

K. S.

McLeod

B. D.

Conroy

Abrams

Smith

M. M.

(2014). Preliminary psychometric properties of the BEST in CLASS Adherence and Competence Scale. Journal of Emotional and Behavioral Disorders, 22(4), 249–259. https://doi.org/10.1177/1063426613497258

64.

Thompson

Felce

Symons

F. J

(Eds.). (2000). Behavioral observation: Technology and applications in developmental disabilities. Brookes Publishing.

65.

Walker

H. M.

Kavanagh

Stiller

Golly

Severson

H. H.

Feil

E. G.

(1998). First step to success: An early intervention approach for preventing school antisocial behavior. Journal of Emotional & Behavioral Disorders, 6(2), 66–81. https://doi.org/10.1177/10634266980060020

66.

Webb

C. A.

DeRubeis

R. J.

Barber

J. P.

(2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211. https://doi.org/10.1037/a0018912

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.19 MB