Ecological momentary assessment (EMA) captures naturalistic experience in real time and holds promise to improve our understanding and treatment of youth psychopathology. While psychometric evaluation of EMA methods is crucial, particularly for use as a tool in clinical trials, research examining the reliability and validity of EMA items in youth is lacking.
Method:
This study evaluates EMA responses from 204 child and adolescent participants (M age = 12.54, 60.8% female), including 131 participants with an anxiety disorder and 73 participants with no psychiatric diagnosis. We assessed the within- and between-person variability, internal consistency, test–retest reliability, and convergent and discriminant validity of two EMA items probing anxiety symptoms; one positive affect item served as a comparison.
Results:
All psychometric properties of the anxiety items were at least satisfactory in youth with anxiety disorders. However, there was restricted variability and poor test–retest reliability in youth with no diagnosis.
Discussion:
These results might facilitate future clinical trials using EMA to investigate pediatric anxiety. Results also suggest that unique EMA items might be needed to reliably track anxiety in healthy youth. Future work should continue to examine the psychometric properties of EMA protocols before implementation in clinical trials.
ClinicalTrials.gov identifier: NCT00018057
Get full access to this article
View all access options for this article.
References
1.
BirmaherB, BrentDA, ChiappettaL, et al.Psychometric properties of the Screen for Child Anxiety Related Emotional Disorders (SCARED): A replication study. J Am Acad Child Adolesc Psychiatry, 1999; 38(10):1230–1236.
2.
BirmaherB, KhetarpalS, BrentD, et al.The screen for child anxiety related emotional disorders (SCARED): Scale construction and psychometric characteristics. J Am Acad Child Adolesc Psychiatry, 1997; 36(4):545–553.
3.
BolgerN, DavisA, RafaeliE. Diary methods: Capturing life as it is lived. Annu Rev Psychol, 2003; 54(1):579–616.
4.
BolgerN, LaurenceauJ-P. Intensive Longitudinal Methods: An Introduction to Diary and Experience Sampling Research. Guilford Press: New York, NY; 2013.
5.
BroseA, SchmiedekF, GerstorfD, et al.The measurement of within-person affect variation. Emotion, 2020; 20(4):677–699.
6.
BylsmaLM, Taylor-CliftA, RottenbergJ. Emotional reactivity to daily events in major and minor depression. J Abnorm Psychol, 2011; 120(1):155–167.
7.
CronbachLJ. Coefficient alpha and the internal structure of tests. Psychometrika, 1951; 16(3):297–334.
8.
De Los ReyesA, CookCR, GreshamFM, et al.Informant discrepancies in assessments of psychosocial functioning in school-based services and research: Review and directions for future research. J Sch Psychol, 2019; 74:74–89.
9.
De Los ReyesA, KazdinAE. Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychol Bull, 2005; 131(4):483–509.
10.
Ebner-PriemerUW, TrullTJ. Ambulatory assessment: An innovative and promising approach for clinical psychology. Eur Psychol, 2009; 14(2):109–119.
11.
ForbesEE, SteppSD, DahlRE, et al.Real-world affect and social context as predictors of treatment response in child and adolescent depression and anxiety: An ecological momentary assessment study. J Child Adolesc Psychopharmacol, 2012; 22(1):37–47.
12.
ForkmannT, SpangenbergL, RathD, et al.Assessing suicidality in real time: A psychometric evaluation of self-report items for the assessment of suicidal ideation and its proximal risk factors using ecological momentary assessments. J Abnorm Psychol, 2018; 127(8):758–769.
13.
FunkhouserCJ, KaiserAJ, AlquezaKL, et al.Depression risk factors and affect dynamics: An experience sampling study. J Psychiatr Res, 2021; 135:68–75.
14.
GeldhofGJ, PreacherKJ, ZyphurMJ. Reliability estimation in a multilevel confirmatory factor analysis framework. Psychol Methods, 2014; 19(1):72–91.
15.
GinsburgGS, KeetonCP, DrazdowskiTK, et al.The utility of clinicians ratings of anxiety using the Pediatric Anxiety Rating Scale (PARS). Child Youth Care Forum, 2011; 40:93–105.
16.
HeronKE, EverhartRS, McHaleSM, et al.Using mobile-technology-based ecological momentary assessment (EMA) methods with youth: A systematic review and recommendations. J Pediatr Psychol, 2017; 42(10):1087–1107.
17.
JahngS, WoodPK, TrullTJ. Analysis of affective instability in ecological momentary assessment: Indices using successive difference and group comparison via multilevel modeling. Psychol Methods, 2008; 13(4):354.
18.
KaufmanJ, BirmaherB, BrentD, et al.Schedule for affective disorders and schizophrenia for school-age children-present and lifetime version (K-SADS-PL): Initial reliability and validity data. J Am Acad Child Adolesc Psychiatry, 1997; 36(7):980–988.
19.
KircanskiK, ThompsonRJ, SorensonJE, et al.Rumination and worry in daily life: Examining the naturalistic validity of theoretical constructs. Clin Psychol Sci, 2015; 3(6):926–939.
20.
KooTK, LiMY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med, 2016; 15(2):155–163.
21.
LaurentJ, CatanzaroSJ, Joiner JrTE, et al.A measure of positive and negative affect for children: Scale development and preliminary validation. Psychol Assess, 1999; 11(3):326–338.
22.
LeeKM, LeeJ, ChungCY, et al.Pitfalls and important issues in testing reliability using intraclass correlation coefficients in orthopaedic research. Clin Orthop Surg, 2012; 4(2):149–155.
23.
LittleRJ. A test of missing completely at random for multivariate data with missing values. J Am Stat Assoc, 1988; 83(404):1198–1202.
24.
MischelW, PeakePK. Beyond déjà vu in the search for cross-situational consistency. Psychol Rev, 1982; 89(6):730–755.
25.
MischelW, ShodaY. A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychol Rev, 1995; 102(2):246–268.
26.
MischelW, ShodaY, Mendoza-DentonR. Situation-behavior profiles as a locus of consistency in personality. Curr Dir Psychol Sci, 2002; 11(2):50–54.
27.
Myin-GermeysI, OorschotM, CollipD, et al.Experience sampling research in psychopathology: Opening the black box of daily life. Psychol Med, 2009; 39(9):1533–1547.
28.
NaimR, SmithA, ChueA, et al.Using ecological momentary assessment to enhance irritability phenotyping in a transdiagnostic sample of youth. Dev Psychopathol, 2021; 33(5):1734–1746.
29.
NezlekJB. A practical guide to understanding reliability in studies of within-person variability. J Res Pers, 2017; 69:149–155.
30.
OppenheimerCW, SilkJS, LeeKH, et al.Suicidal ideation among anxious youth: A preliminary investigation of the role of neural processing of social rejection in interaction with real world negative social experiences. Child Psychiatry Hum Dev, 2020; 51(2):163–173.
31.
Palmier-ClausJE, Myin-GermeysI, BarkusE, et al.Experience sampling research in individuals with mental illness: Reflections and guidance. Acta Psychiatr Scand, 2011; 123(1):12–20.
32.
PinheiroJ, BatesD, DebRoyS, et al.nlme: Linear and Nonlinear Mixed Effects Models; 2018. Available from: https://CRAN.R-project.org/package?nlme [Last accessed: July16, 2023].
33.
PortneyLG, WatkinsMP. Foundations of Clinical Research: Applications to Practice. Pearson/Prentice Hall: Upper Saddle River, NJ; 2009.
34.
Research Units on Pediatric Psychopharmacology Anxiety Study Group. The pediatric anxiety rating scale (PARS): Development and psychometric properties. J Am Acad Child Adolesc Psychiatry, 2002; 41(9):1061–1069.
35.
RussellMA, GajosJM. Annual research review: Ecological momentary assessment studies in child psychology and psychiatry. J Child Psychol Psychiatry, 2020; 61(3):376–394.
36.
SeidlE, VenzJ, OllmannTM, et al.Dynamics of affect, cognition and behavior in a general population sample of adolescents and young adults with current and remitted anxiety disorders: An Ecological Momentary Assessment study. J Anxiety Disord, 2023; 93:102646.
37.
ShiffmanS, StoneAA, HuffordMR. Ecological Momentary Assessment. Annual review of clinical psychology. Annu Rev Clin Psychol, 2007; 4:1–32.
38.
SideridisG, SaddaawiA, Al-HarbiK. Internal consistency reliability in measurement: Aggregate and multilevel approaches. J Mod Appl Stat Methods, 2018; 17(1):1–30.
39.
SilkJS, ForbesEE, WhalenDJ, et al.Daily emotional dynamics in depressed youth: A cell phone ecological momentary assessment study. J Exp Child Psychol, 2011; 110(2):241–257.
40.
SmithAR, JonesEL, SubarAR, et al.The role of anxiety and gender in anticipation and avoidance of naturalistic anxiety-provoking experiences during adolescence: An ecological momentary assessment study. JCPP Adv, 2022; 2(3):e12084.
41.
SmithAR, KircanskiK, BrotmanMA, et al.Advancing clinical neuroscience through enhanced tools: Pediatric social anxiety as an example. Depress Anxiety, 2019; 36(8):701–711.
42.
SnijdersTA, BoskerRJ. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. Sage: Thousand Oaks, CA; 2011.
43.
StoneLB, MenniesRJ, WallerJM, et al.Help me feel better! Ecological momentary assessment of anxious youths' emotion regulation with parents and peers. J Abnorm Child Psychol, 2019; 47(2):313–324.
44.
TanPZ, ForbesEE, DahlRE, et al.Emotional reactivity and regulation in anxious and nonanxious youth: A cell-phone ecological momentary assessment study. J Child Psychol Psychiatry, 2012; 53(2):197–206.
45.
TrullTJ, SolhanMB, TragesserSL, et al.Affective instability: Measuring a core feature of borderline personality disorder with ecological momentary assessment. J Abnorm Psychol, 2008; 117(3):647.
46.
ViechtbauerW. Reliability of ESM Assessments of Mood and Mood Sensitivity. In: Digital Health in Ambulatory Assessment—Abstract Book of the 5th Biennial Conference of the Society for Ambulatory Assessment. (VögeleC. ed.) University of Luxembourg: Luxembourg City, 2017; p. 25.
47.
WalzLC, NautaMH, aan het RotM. Experience sampling and ecological momentary assessment for studying the daily lives of patients with anxiety disorders: A systematic review. J Anxiety Disord, 2014; 28(8):925–937.
48.
WatsonD, ClarkLA, TellegenA. Development and validation of brief measures of positive and negative affect: The PANAS scales. J Pers Soc Psychol, 1988; 54(6):1063–1070.
49.
WechslerD.Manual for the Wechsler Abbreviated Intelligence Scale (WASI). The Psychological Corporation: San Antonio, TX; 1999.
50.
WoyshvilleMJ, LackampJM, EisengartJA, et al.On the meaning and measurement of affective instability: Clues from chaos theory. Biol Psychiatry, 1999; 45(3):261–269.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
