Development of Brief Rating Scales for Progress Monitoring Internalizing Behavior

Abstract

Prevalence estimates suggest that up to 20% of students in schools experience significant internalizing behaviors that impact behavioral, social, or academic functioning. School-based interventions have great potential to promote student mental health; however, validated and feasible brief assessments are needed to progress monitor students’ responses to these supports. The purpose of the current study was twofold: to (a) develop and validate teacher-completed brief rating scales for progress monitoring internalizing concerns in elementary-age students and (b) determine the reliability of the resultant measures. First, item content was generated and subjected to evaluation by two panels of school-based consumers and researchers. Within the second phase of development, exploratory and confirmatory factor analyses were used to reduce the initial number of items and ensure that the items were indicators of one latent factor. Teachers in Grades K–3 (N = 307) each completed ratings for one randomly selected student in their classroom. Results of factor analysis for each scale indicated one-factor solutions for the 4-item Anxious/Depressed (ω = .88) and 4-item Socially Withdrawn (ω = .87) scales.

Keywords

brief rating scale progress monitoring behavioral assessment internalizing problems exploratory factor analysis

Get full access to this article

View all access options for this article.

References

Achenbach

T. M.

McConaughy

S. H.

Howell

C. T.

(1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232.

Barrett

P. M.

Webster

Turner

(2000). FRIENDS prevention of anxiety and depression for children group leader’s manual. Australian Academic Press.

Bradshaw

C. P.

Buckley

J. A.

Ialongo

N. S.

(2008). School-based service utilization among urban children with early onset educational and mental health problems: The squeaky wheel phenomenon. School Psychology Quarterly, 23, 169–186.

Calear

A. L.

Christensen

(2010). Systematic review of school-based prevention and early intervention programs for depression. Journal of Adolescence, 33, 429–438.

Cartwright-Hatton

McNicol

Doubleday

(2006). Anxiety in a neglected population: Prevalence of anxiety disorders in pre-adolescent children. Clinical Psychology Review, 26, 817–833.

Christ

T. J.

Riley-Tillman

T. C.

Chafouleas

S. M.

(2009). Foundation for the development and use of Direct Behavior Rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention, 34, 201–213. https://doi.org/10.1177/1534508409340390

Comrey

A. L.

Lee

H. B.

(1992). A first course in factor analysis (2nd ed.). Lawrence Erlbaum.

Dart

E. H.

Arora

P. G.

Collins

T. A.

Doll

(2019). Progress monitoring measures for internalizing symptoms: A systematic review of the peer-reviewed literature. School Mental Health, 11, 265–275.

Dart

E. H.

Furlow

C. M.

Collins

T. A.

Brewer

Gresham

F. M.

Chenier

K. H.

(2015). Peer-mediated check-in/check-out for students at-risk for internalizing disorders. School Psychology Quarterly, 30(2), 229–243. https://doi.org/10.1037/spq0000092

10.

De Los Reyes

Cook

C. R.

Gresham

F. M.

Makol

B. A.

Wang

(2019). Informant discrepancies in assessments of psychosocial functioning in school-based services and research: Review and directions for future research. Journal of School Psychology, 74, 74–89.

11.

Duchesne

Vitaro

Larose

Tremblay

R. E.

(2008). Trajectories of anxiety during elementary-school years and the prediction of high school noncompletion. Journal of Youth and Adolescence, 37, 1134–1146.

12.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299.

13.

Farahmand

F. K.

Grant

K. E.

Polo

A. J.

Duffy

S. N.

(2011). School-based mental health and behavioral programs for low-income, urban youth: A systematic and meta-analytic review. Clinical Psychology: Science and Practice, 18, 372–390.

14.

Farrell

L. J.

Barrett

P. M.

(2003). Prevention of childhood emotional disorders: Reducing the burden of suffering associated with anxiety and depression. Child and Adolescent Mental Health, 12, 58–65.

15.

Flook

Repetti

R. L.

Ullman

J. B.

(2005). Classroom social experiences as predictors of academic performance. Developmental Psychology, 41, 319–327.

16.

Ghandour

R. M.

Sherman

L. J.

Vladutiu

C. J.

Ali

M. M.

Lynch

S. E.

Bitsko

R. H.

Blumberg

S. J.

(2019). Prevalence and treatment of depression, anxiety, and conduct problems in U.S. children. The Journal of Pediatrics, 206, 256–267. https://doi.org/10.1016/j.jpeds.2018.09.021

17.

Hair

J. F.

Jr. Black

Babin

B. J.

Anderson

R. E.

Tatham

R. L.

(2006). Multivariate data analysis (6th ed.). Pearson Prentice Hall.

18.

Harter

(1986). Processes underlying the construction, maintenance, and enhancement of self-concept in children. In Suis

Greenwald

(Eds.), Psychological perspectives on the self (pp. 137–181). Lawrence Erlbaum.

19.

Herzig-Anderson

Colognori

Fox

J. K.

Stewart

C. E.

Warner

C. M.

(2012). School-based anxiety treatments for children and adolescents. Child and Adolescent Psychiatric Clinics of North America, 21, 655–668. http://doi.org/10.1016/j.chc.2012.05.006

20.

Hodges

Wong

M. M.

(1996). Psychometric characteristics of a multidimensional measure to assess impairment: The Child and Adolescent Functional Assessment Scale (CAFAS). Journal of Child and Family Studies, 5, 445–467.

21.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. http://doi.org/10.1080/10705519909540118

22.

Jorgensen

T. D.

Pornprasertmanit

Schoemann

A. M.

Rosseel

(2021). semTools: Useful tools for structural equation modeling. R package version 0.5-4 [Computer software]. https://CRAN.R-project.org/package=semTools

23.

Kendall

P. C.

Hedtke

(2006). Coping cat workbook (2nd ed). Workbook Publishing.

24.

Kenny

D. A.

Kaniskan

McCoach

D. B.

(2015). The performance of RMSEA in models with small degrees of freedom. Sociological Methods & Research, 44(3), 486–507. https://doi.org/10.1177/0049124114543236

25.

Kilgus

S. P.

Chafouleas

S. M.

Riley-Tillman

T. C.

(2013). Development and initial validation of the Social and Academic Behavior Risk Screener for elementary grades. School Psychology Quarterly, 28(3), 210–226.

26.

Kline

R. B.

(2016). Principles and practice of structural equation modeling. Guilford Publications.

27.

Lane

K. L.

Oakes

W. P.

Harris

P. J.

Menzies

H. M.

Cox

Lambert

(2012). Initial evidence for the reliability and validity of the student risk screening scale for internalizing and externalizing behaviors at the elementary level. Behavioral Disorders, 37, 99–122.

28.

Levitt

V. H.

Merrell

K. W.

(2009). Linking assessment to intervention for internalizing problems of children and adolescents. School Psychology Forum, 3, 13–26.

29.

McCoach

D. B.

Gable

R. K.

Madura

(2013). Instrument design in the affective domain (3rd ed.). Springer.

30.

McDonald

R. P.

M.-H. R.

(2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64–82. https://doi.org/10.1037/1082-989X.7.1.64

31.

McLoone

Hudson

J. L.

Rapee

R. M.

(2006). Treating anxiety disorders in a school setting. Education and Treatment of Children, 29, 219–242.

32.

Merrell

K. W.

(2008). Helping students overcome depression and anxiety: A practical guide. Guilford Press.

33.

Merrell

K. W.

Gueldner

B. A.

(2010). Preventive Interventions for students with internalizing disorders: Effective strategies for promoting mental health in schools. In Shinn

M. R.

Walker

H. M.

(Eds.), Interventions for achievement and behavior problems in a three-tier model including RTI (pp. 729–823). National Association of School Psychologists.

34.

National Center on Intensive Intervention. (n.d.). Identifying assessments. https://intensiveintervention.org/tools-charts/identifying-assessments

35.

National Center on Student Progress Monitoring. (2006). Review of progress monitoring tools. http://www.studentprogress.org/chart/chart.asp

36.

Neil

A. L.

Christensen

(2009). Efficacy and effectiveness of school-based prevention and early intervention programs for anxiety. Clinical Psychology Review, 29, 208–215.

37.

Pett

M. A.

Lackey

N. R.

Sullivan

J. J.

(2003). Making sense of factor analysis. SAGE.

38.

Puddy

R. W.

Boles

R. E.

Dreyer

M. L.

Maikranz

Roberts

M. C.

Vernberg

E. M.

(2008). Demonstrating support for the formative and summative assessment paradigm in a school-based intensive mental health program. Journal of Child and Family Studies, 17, 253–263.

39.

RStudio Team (2020). RStudio: Integrated Development for R. RStudio, PBC, Boston, MA. http://www.rstudio.com/

40.

Rapport

M. D.

Denney

C. B.

Chung

Hustace

(2001). Internalizing behavior problems and scholastic achievement in children: Cognitive and behavioral pathways as mediators of outcome. Journal of Clinical Child Psychology, 30, 536–551.

41.

Revelle

(2021). psych: Procedures for Psychological, Psychometric, and Personality Research. R package version 2.1.3 [Computer program]. https://CRAN.R-project.org/package=psych

42.

Rhemtulla

Brosseau-Liard

P. É.

Savalei

(2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373.

43.

Rosseel

(2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

44.

Rubin

K. H.

Chen

McDougall

Bowker

McKinnon

(1995). The Waterloo Longitudinal Project: Predicting adolescent internalizing and externalizing problems from early and mid-childhood. Developmental Psychopathology, 7, 751–764.

45.

Rubin

K. H.

Coplan

R. J.

Bowker

J. C.

(2009). Social withdrawal in childhood. Annual Review of Psychology, 60, 141–171.

46.

Sanders

D. E.

Merrell

K. W.

Cobb

H. C.

(1999). Internalizing symptoms and affect of children with emotional and behavioral disorders: A comparative study with an urban African American sample. Psychology in the Schools, 36, 187–197.

47.

Schniering

C. A.

Rapee

R. M.

Lyneham

H. J.

Wuthrich

Hudson

J. L.

Wignall

(2006). The Cool Kids® Adolescent Anxiety & Depression Program Therapist Manual. Centre for Emotional Health, Macquarie University.

48.

Shi

Lee

Maydeu-Olivares

(2019). Understanding the model size effect on SEM fit indices. Educational and Psychological Measurement, 79(2), 310–334. https://doi.org/10.1177/0013164418783530

49.

Smith

S. R.

(2007). Making sense of multiple informants in child and adolescent psychopathology: A guide for clinicians. Journal of Psychoeducational Assessment, 25, 139–149.

50.

Splett

J. W.

Garzona

Gibson

Wojtalewicz

Raborn

Reinke

(2019). Teacher recognition, concern, and referral of children’s internalizing and externalizing behavior problems. School Mental Health, 11, 228–239.

51.

von der Embse

N. P.

Scott

E. C.

Kilgus

S. P.

(2015). Sensitivity to change and concurrent validity of Direct Behavior Ratings for academic anxiety. School Psychology Quarterly, 30, 244–259.

52.

Walker

H. M.

Nishioka

V. M.

Zeller

Severson

H. H.

Feil

E. G.

(2000). Causal factors and potential solutions for the persistent under- identification of students having emotional or behavioral disorders in the context of schooling. Assessment for Effective Intervention, 26, 29–39. https://doi.org/10.1177/073724770002600105

53.

Webster-Stratton

Reid

M. J.

Hammond

(2004). Treating children with early-onset conduct problems: Intervention outcomes for parent, child, and teacher training. Journal of Clinical Child & Adolescent Psychology, 33(1), 105–124.

54.

Wickham

(2016). Ggplot2: Elegant graphics for data analysis. Springer-Verlag.