Psychometric Evaluation of a New Digitally Animated Child Self-Report Assessment Instrument: The Interactive Child Distress Screener

Abstract

The present study aimed to provide the first psychometric evaluation of the newly developed, digitally animated assessment instrument: the Interactive Child Distress Screener (ICDS). The latent factor structure of the novel ICDS was first established using exploratory factor analysis (EFA) on 15 pairs of animated items using a community sample (N = 266) of child–parent dyads. EFA results support a two-factor structure representing two broad domains of internalizing and externalizing difficulties (r = .52) and comprised 12 items. The reliability of the factors was strong with ordinal alpha and omega coefficients above .84 and .87, respectively, for each of the subscales. Convergent validity for the overall sample was supported with established child and parent-reported measures of internalizing and externalizing problems; however, the ICDS factors demonstrated convergence greater in magnitude with other child-reported measures such as the Me and My School Survey. Satisfaction and utility ratings by children indicated that the digital format was highly acceptable.

Keywords

ICDS digital assessment exploratory factor analysis child self-report scale psychometrics

Get full access to this article

View all access options for this article.

References

Achenbach

T. M.

McConaughy

S. H.

Ivanova

M. Y.

Rescorla

L. A.

(2011). Manual for the ASEBA brief problem monitor (BPM) (pp. 1–33). University of Vermont.

Arseneault

Kim-Cohen

Taylor

Caspi

Moffitt

T. E.

(2005). Psychometric evaluation of 5- and 7-year-old children’s self-reports of conduct problems. Journal of Abnormal Child Psychology, 33(5), 537–550. https://doi.org/10.1007/s10802-005-6736-5

Bevans

K. B.

Ahuvia

I. L.

Hallock

T. M.

Mendonca

Roth

Forrest

C. B.

Blackwell

Kramer

Wakschlag

(2020). Investigating child self-report capacity: A systematic review and utility analysis. Quality of Life Research, 29(5), 1147–1158. https://doi.org/10.1007/s11136-019-02387-3

Bowen

N. K.

(2008). Cognitive testing and the validity of child-report data from the Elementary School Success Profile. Social Work Research, 32(1), 18–28. https://doi.org/10.1093/swr/32.1.18

Carter

A. S.

Briggs-Gowan

M. J.

Davis

N. O.

(2004). Assessment of young children’s social-emotional development and psychopathology: Recent advances and recommendations for practice. Journal of Child Psychology and Psychiatry, 45(1), 109–134. https://doi.org/10.1046/j.0021-9630.2003.00316.x

Centers for Disease Control and Prevention. (2020). Data and statistics on children’s mental health. https://www.cdc.gov/childrensmentalhealth/data.html

Chambers

C. T.

Johnston

(2002). Developmental differences in children’s use of rating scales. Journal of Pediatric Psychology, 27(1), 27–36. https://doi.org/10.1093/jpepsy/27.1.27

Comrey

A. L.

(1988). Factor-analytic methods of scale development in personality and clinical psychology. Journal of Consulting and Clinical Psychology, 56(5), 754–761. https://doi.org/10.1037/0022-006X.56.5.754

Deighton

Croudace

Fonagy

Brown

Patalay

Wolpert

(2014). Measuring mental health and wellbeing outcomes for children and adolescents to inform practice and policy: A review of child self-report measures. Child and Adolescent Psychiatry and Mental Health, 8(1), Article 14. https://doi.org/10.1186/1753-2000-8-14

10.

Deighton

Lereya

S. T.

Casey

Patalay

Humphrey

Wolpert

(2019). Prevalence of mental health problems in schools: Poverty and other risk factors among 28 000 adolescents in England. British Journal of Psychiatry, 215(3), 565–567. https://doi.org/10.1192/bjp.2019.19

11.

Deighton

Tymms

Vostanis

Belsky

Fonagy

Brown

Martin

Patalay

Wolpert

(2013). The development of a school-based measure of child mental health. Journal of Psychoeducational Assessment, 31(3), 247–257. https://doi.org/10.1177/0734282912465570

12.

Dowdy

Ritchey

Kamphaus

R. W.

(2010). School-based screening: A population-based approach to inform and monitor children’s mental health needs. School Mental Health, 2(4), 166–176. https://doi.org/10.1007/s12310-010-9036-3

13.

Droit-Volet

Coull

(2015). The developmental emergence of the mental time-line: Spatial and numerical distortion of time judgement. PLOS ONE, 10(7), 1–20. https://doi.org/10.1371/journal.pone.0130465

14.

Field

Miles

Field

(2012). Discovering statistics. Discovering statistics Using R. SAGE.

15.

Floyd

F. J.

Widaman

K. F.

(1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7(3), 286–299. https://doi.org/10.1037/1040-3590.7.3.286

16.

Gadermann

A. M.

Guhn

Zumbo

B. D.

(2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research and Evaluation, 17(3), 1–13. https://doi.org/10.7275/n560-j767

17.

Gardner

Murphy

Childs

Kelleher

Pagano

Jellinek

McInerny

T. K.

Wasserman

R. C.

Nutting

Chiappetta

Sturner

(1999). The PSC-17: A brief pediatric symptom checklist with psychosocial problem subscales. A report from PROS and ASPN. Ambulatory Child Health, 5(3), 225–236.

18.

Goodman

(2001). Psychometric properties of the strengths and difficulties questionnaire. Journal of the American Academy of Child and Adolescent Psychiatry, 40(11), 1337–1345. https://doi.org/10.1097/00004583-200111000-00015

19.

Goodman

(1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry and Allied Disciplines, 38(5), 581–586. https://doi.org/10.1111/j.1469-7610.1997.tb01545.x

20.

Goodman-Scott

Donohue

Betters-Bubon

(2019). The case for universal mental health screening in schools. Counseling Today, 62(3), 40–45.

21.

Guzman

M. P.

Jellinek

George

Hartley

Squicciarini

A. M.

Canenguez

K. M.

Kuhlthau

K. A.

Yucel

White

G. W.

Guzman

Murphy

J. M.

(2011). Mental health matters in elementary school: First-grade screening predicts fourth grade achievement test scores. European Child and Adolescent Psychiatry, 20(8), 401–411. https://doi.org/10.1007/s00787-011-0191-3

22.

Hawes

D. J.

Dadds

M. R.

(2004). Australian data and psychometric properties of the Strengths and Difficulties Questionnaire. Australian and New Zealand Journal of Psychiatry, 38(8), 644–651. https://doi.org/10.1111/j.1440-1614.2004.01427.x

23.

Hayton

J. C.

Allen

D. G.

Scarpello

. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191–205. https://doi.org/10.1177/1094428104263675

24.

Hoe

S. L.

(2008). Issues and procedures in adopting structural equation modeling technique. Journal of Applied Quantitative Methods, 3(1), 76–83.

25.

Humphrey

Wigelsworth

(2016). Making the case for universal school-based mental health screening. Emotional and Behavioural Difficulties, 21(1), 22–42. https://doi.org/10.1080/13632752.2015.1120051

26.

Ivey

(2020). Mental health screening for children and adolescents. Pediatric Nursing, 46(1), 27–31.

27.

Jardine

Glinianaia

S. V.

McConachie

Embleton

N. D.

Rankin

(2014). Self-reported quality of life of young children with conditions from early infancy: A systematic review. Pediatrics, 134(4), e1129–e1148. https://doi.org/10.1542/peds.2014-0352

28.

Jeffrey

Klomhaus

Enenbach

Lester

Krishna

(2020). Self-report rating scales to guide measurement-based care in child and adolescent psychiatry. Child and Adolescent Psychiatric Clinics of North America, 29(4), 601–629. https://doi.org/10.1016/j.chc.2020.06.002

29.

Kamphaus

M. C.

Reynolds

C. R

. (2007). Behavior assessment system for children – second edition (BASC-2): Behavioral and emotional screening system (BESS). Bloomington, MN: Pearson.

30.

Lawrence

Hafekost

Johnson

S. E.

Saw

Buckingham

W. J.

Sawyer

M. G.

Ainley

Zubrick

S. R.

(2016). Key findings from the second Australian child and adolescent survey of mental health and wellbeing. Australian & New Zealand Journal of Psychiatry, 50(9), 876–886. https://doi.org/10.1177/0004867415617836

31.

Lorenzo-Seva

Timmerman

M. E.

Kiers

H. A. L.

(2011). The Hull method for selecting the number of common factors. Multivariate Behavioral Research, 46, 340–364. https://doi.org/10.1080/00273171.2011.564527

32.

Manassis

Mendlowitz

Kreindler

Lumsden

Sharpe

Simon

M. D.

Woolridge

Monga

Adler-Nevo

(2009). Mood assessment via animated characters: A novel instrument to evaluate feelings in young children with anxiety disorders. Journal of Clinical Child and Adolescent Psychology, 38(3), 380–389. https://doi.org/10.1080/15374410902851655

33.

March

Day

Zieschank

Ireland

(2018). The interactive child distress screener: Development and preliminary feasibility testing. JMIR MHealth and UHealth, 6(4), Article e90. https://doi.org/10.2196/mhealth.9456

34.

Mardia

K. V.

(1970). Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3), 519–530. https://doi.org/10.2307/2334770

35.

McCormack

Hoerl

(2017). The development of temporal concepts: Learning to locate events in time. Timing and Time Perception, 5(3–4), 297–327. https://doi.org/10.1163/22134468-00002094

36.

McCrae

J. S.

Brown

S. M.

(2018). Systematic review of social–emotional screening instruments for young children in child welfare. Research on Social Work Practice, 28(7), 767–788. https://doi.org/10.1177/1049731516686691

37.

Merikangas

K. R.

Burstein

Swanson

S. A.

Avenevoli

Cui

Benjet

Georgiades

Swendsen

(2010). Lifetime prevalence of mental disorders in U.S. adolescents: Results from the national comorbidity survey replication–Adolescent supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 49(10), 980–989. https://doi.org/10.1016/j.jaac.2010.05.017

38.

Mihalopoulos

Vos

Pirkis

Carter

(2012). The population cost-effectiveness of interventions designed to prevent childhood depression. Pediatrics, 129(3), Article e723. https://doi.org/10.1542/peds.2011-1823

39.

Moffa

Wagle

Dowdy

Palikara

Castro

Dougherty

Furlong

M. J.

(2019). The Me and My School Questionnaire: Examining the cross-cultural validity of a children’s self-report mental health measure. International Journal of School and Educational Psychology, 9(1), 1–11. https://doi.org/10.1080/21683603.2019.1650858

40.

Muthén

L. K.

Muthén

B. O

. (2012). Mplus user’s guide (7th ed.) Muthén & Muthén.

41.

National Research Council and Institute of Medicine. (2009). Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities ( Warner

K. E.

Boat

O’Connell

M. E.

, eds.). The National Academies Press. https://doi.org/10.17226/12480

42.

Newlove-Delgado

Ford

T. J.

(2020). Screening methods and when to use them. In Taylor

Verhulst

F. C.

Wong

Yoshida

Nikapota

(Eds.), Mental health and illness of children and adolescents (pp. 17–38). Springer Nature Singapore. https://doi.org/10.1007/978-981-10-2348-4_7

43.

Ogundele

M. O.

(2018). Behavioural and emotional disorders in childhood: A brief overview for paediatricians. World Journal of Clinical Pediatrics, 7(1), 9–26. https://doi.org/10.5409/wjcp.v7.i1.9

44.

Patalay

Deighton

Fonagy

Vostanis

Wolpert

(2014). Clinical validity of the Me and My School questionnaire: A self-report mental health measure for children and adolescents. Child and Adolescent Psychiatry and Mental Health, 8(17), 1–7. https://doi.org/10.1186/1753-2000-8-17

45.

Piper

B. J.

Gray

H. M.

Raber

Birkett

M. A.

(2014). Reliability and validity of brief problem monitor, an abbreviated form of the child behavior checklist. Psychiatry and Clinical Neurosciences, 68(10), 759–767. https://doi.org/10.1111/pcn.12188

46.

Ringoot

A. P.

Jansen

P. W.

Rijlaarsdam

Jaddoe

V. W. V.

Verhulst

F. C.

Tiemeier

(2017). Self-reported problem behavior in young children with and without a DSM-disorder in the general population. European Psychiatry, 40, 110–115. https://doi.org/10.1016/j.eurpsy.2016.08.009

47.

The Royal Australian and New Zealand College of Psychiatrists. (2010). Prevention and early intervention of mental illness in infants, children and adolescents: Planning strategies for Australia and New Zealand. https://www.ranzcp.org/news-policy/policy-and-advocacy/position-statements/the-prevention-and-early-intervention-of-mental-il

48.

Schwarz

G. E.

(1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461–464. https://https://doi.org/10.1214/aos/1176344136

49.

Truman

Robinson

Evans

A. L.

Smith

Cunningham

Millward

Minnis

(2003). The Strengths and Difficulties Questionnaire: A pilot study of a new computer version of the self-report scale. European Child and Adolescent Psychiatry, 12(1), 9–14. https://doi.org/10.1007/s00787-003-0303-9

50.

Whitney

D. G.

Peterson

M. D.

(2019). US national and state-level prevalence of mental health disorders and disparities of mental health care use in children. JAMA Pediatrics, 173(4), 389–391. https://doi.org/10.1001/jamapediatrics.2018.5399

51.

Wood

B. J.

McDaniel

(2020). A preliminary investigation of universal mental health screening practices in schools. Children and Youth Services Review, 112(March), Article 104943. https://doi.org/10.1016/j.childyouth.2020.104943

52.

Zieschank

Machin

Day

Ireland

March

. (2020). Children’s perspectives on emotions informing a child-reported screening instrument. Journal of Child and Family Studies, 30, 3105–3120. https://doi.org/10.1007/s10826-021-02086-z

53.

Zieschank

Day

Ireland

M. J.

March

. (2021). Co-design and qualitative validation of animated assessment item content for a child-reported digital distress screener. Internet Interventions, 24, Article 100381. https://doi.org/10.1016/j.invent.2021.100381