Improvements in sleep problems and their associations with mental health symptoms: A study of children and adolescents participating in a digital mental health intervention

Abstract

Objective

A growing number of youth are utilizing digital mental health interventions (DMHIs) for treatment of mental health problems such as anxiety, depression, and ADHD. Although these mental health symptoms are closely related to sleep problems, it is unknown whether nonsleep DMHIs indirectly confer improvements in sleep. Using retrospective data, the current study assesses (1) whether youth sleep problems improve over participation in a nonsleep DMHI, and (2) whether mental health symptom severity and improvement are correlated with sleep problem severity over time.

Methods

Sleep problems and mental health symptoms were assessed every 30 days among children (ages 5–12) and adolescents (ages 13–17) participating in a pediatric digital mental health intervention (DMHI; N = 1219).

Results

Children and adolescents with elevated sleep problems (39.3%; n = 479) were older (P < .001), more predominantly female (P < .001), and more likely to have elevated anxiety (P < .001), depressive (P < .001) and inattention symptoms (P = .001), as compared to those with nonelevated sleep problems (60.7%; n = 740). From the baseline to last assessment, 77.3% (n = 269) of members with elevated sleep problems exhibited improvements, with sleep problems decreasing significantly over each month in care (P < .001). Members with improvements in anxiety, depressive, and/or ADHD symptoms had larger improvements in sleep over time compared to their peers with no improvement in their mental health symptoms (Months in care*Change type: P < .001 for all).

Conclusions

Our results provide preliminary evidence that participation in a pediatric DMHI is associated with improvements in sleep problems, even when youth are not being treated directly for sleep problems. These findings highlight a valuable secondary benefit of participating in mental health care within pediatric DMHIs and warrant further experimental research.

Keywords

Sleep problems anxiety depression attention deficit hyperactivity disorder adolescence digital mental health

Introduction

In recent years, mental and behavioral health problems have risen considerably among youth, and yet nearly 1 in 5 young people do not receive adequate behavioral health care.¹ In the midst of this crisis, the integration of behavioral health into primary care has been a boon to families and providers by providing increased access to effective and evidence-based behavioral health care.² In one such example of integrated care known as the collaborative care model, primary care providers (PCPs) partner with behavioral care managers and psychiatrists to implement measurement-based mental health care, allowing patients to access appropriate treatments and continuous follow-up care more easily via their existing PCP.³ Large-scale implementation of the collaborative care model has been facilitated by digital mental health interventions (DMHIs), or those mediated by technology such as computers and smartphones.² Growing evidence suggests that web-based therapy and coaching via collaborative care DMHIs confers significant improvements in children's and adolescents’ anxiety, depressive, and ADHD symptoms,^4–6 while also mitigating barriers related to transportation, provider location, and perceived stigma.^7,8

Mental health is closely linked to healthy sleep habits, particularly in childhood and adolescence.^9,10 As pediatric mental health problems have increased, so have rates of sleep problems and inadequate sleep, with recent studies estimating that nearly 80% of adolescents in the US report short sleep duration.¹¹ The COVID-19 pandemic further exacerbated the rates of sleep problems among children: an estimated 54% experienced sleep problems in 2021, up from approximately 25% prior to the pandemic.¹² Inadequate sleep not only hinders youths’ physical and cognitive development, but also increases risk for accidental injury and death,^13,14 making sleep in youth a growing public health concern.¹⁰

Despite the high rates of sleep problems among youth, most children and adolescents who receive mental health care do so for treatment of nonsleep problems, particularly depression, suicidality, anxiety, and conduct disorders.¹⁵ Recent studies estimate that more than 1 in 10 children ages 5 to 17 receive specialized mental health care¹⁶; moreover, 77% of youth with a diagnosed mental health problem have sleep difficulties in addition to their primary psychiatric symptom.¹⁷ Although several studies have identified close and bi-directional associations between sleep and mental health symptoms and treatment,^18–22 it is unknown whether participation in a collaborative care DMHI for treatment of common mental health problems (e.g. depression, anxiety, and ADHD) confers secondary improvements in sleep. This topic is particularly relevant considering the increasing number of youth participating in DMHIs²³ and the consequent need to evaluate both primary and secondary clinical benefits of DMHIs and collaborative care.

Therefore, the current study examines change in sleep problems and mental health symptoms among children and adolescents participating in synchronous web-based coaching and therapy services within a collaborative care DMHI for nonsleep mental health symptoms (anxiety, depression, and ADHD). Specifically, the purpose of this study was to (1) determine whether child and adolescent sleep problems improve over time while participating in a nonsleep DMHI, and (2) assess correlations between sleep problems and anxiety, depressive, and ADHD symptom severity at the baseline and improvement over time. Given extant research, we hypothesize that (1) sleep problems will improve while participating in a nonsleep DMHI, and (2) there will be a positive correlation between sleep problem severity and anxiety, depressive, and ADHD symptom severity at the baseline and over time.

Methods

Design and participants

Bend Health Inc. members ages five to 17, and their caregivers, were eligible for inclusion in the study if: (1) they enrolled in mental health care with Bend Health Inc. between January 1st 2023 and August 31st 2023 (9 months data collection), (2) had at least one synchronous session with a Bend Health Inc. coach or therapist, (3) did not participate in the sleep care program, and (4) completed the sleep assessment before the start of care with Bend Health Inc. (N = 1219). Upon enrollment in Bend Health Inc., all members provided informed consent for their data being used for research purposes such as retrospective analyses. All study procedures were approved by an independent review board (Biomedical Research Alliance of New York; Study 23-12-034-1374).

Treatment

Bend Health is a DHMI for children and adolescents (ages <18) and their caregivers. Treatment at Bend Health has been described previously (see Huffman et al., 2023 and Lawrence-Sidebottom et al., 2023). All elements of the intervention are conducted virtually using an internet-based interface. Members can enroll in Bend Health via referral from a healthcare provider, insurance, employer benefits, or they can enroll themselves (direct to consumer). During enrollment, caregivers are asked whether their child or teen exhibits any extreme conditions or symptoms, including detox for alcohol or illicit drug use, active suicidal ideation within the last two weeks or homicidal ideation with a plan in the last three months, moderate to severe intellectual disability, or neurocognitive disorder with severe memory or functioning difficulties. If the response to this question is “yes,” the caregiver is given crisis resource information and recommended to seek care elsewhere. Once enrolled, members are assigned a Behavioral Care Manager (BCM) who conducts an evaluation and monitors member care while they are enrolled in Bend. All members are assigned a coach, and some are also assigned a therapist depending on the symptoms and severity of the behavior the member needs care for and insurance coverage. Additionally, as a part of their treatment with Bend, some members are assigned a psychiatric practitioner who may prescribe medications as appropriate to the member's situation (e.g. when symptoms are severe or they have been referred for a medication trial). All members enrolled in Bend via a health system – accounting for approximately 60% of active members – participate in an evaluation with a psychiatric provider as part of their care plan. Members meet with all Bend Health providers in synchronous online video coaching and/or therapy sessions, where practitioners provide members with structured care programs (discussed in detail below). These sessions use evidence-based techniques and caregivers are included in aspects of the care programs, especially for children (ages < 13). While less involved in the care programs of adolescents (ages 13 to 17), caregivers must be in the same general area as their adolescent during sessions (i.e. for safety reasons).

The structured care programs are designed to be age-appropriate and to target a specific symptom domain, behavioral concern, or circumstance (e.g. anxiety, ADHD, depression, or sleep). The anxiety, ADHD, and depression care programs include 16 modules. BCMs assign members a care program based on mental health symptoms (e.g. based on assessment results), as well as goals and desired services. One care program is assigned at a time, and members typically complete one module per coaching or therapy session. Caregivers and members complete mental health and sleep assessments once a month to assess change in symptoms, as described in greater detail below.

Measures

During enrollment in the online platform, caregivers provide basic demographic information for their child or adolescent, including date of birth, sex at birth (male, female, or other), current gender identity (male, female, transgender, nonbinary, or other), and race/ethnicity. The race/ethnicity response options switched part-way through the study, with details in the Supplemental Materials section. After demographic information is provided, adolescent members complete a series of screener questions and assessments to identify common mental health and behavioral concerns.

Sleep assessment

At enrollment and each successive month with Bend Health Inc., caregivers of members are asked to respond to the single sleep screener question from the DSM-V cross-cutting measure²⁴ to flag whether the member may have had sleep problems in the past two weeks. The screener question is: “During the past two weeks, how much (or how often) has your child had problems sleeping—that is, trouble falling asleep, staying asleep, or waking up too early?” Response options are on a five-item Likert-type scale ranging from Not at all (0) to nearly every day (4). Members with a score of 2 (Several days; Mild) or greater are prompted to complete the short form of the PROMIS sleep assessment.^25,26 Caregivers of children (ages < 13) are requested to respond on behalf of the member, whereas adolescent members (ages 13 to 17) are requested to self-report. The short-form PROMIS sleep assessment includes eight questions about sleep difficulties over the past week (see Supplement). Responses are on a five-item Likert-type scale with options ranging from “Not at all” (1) to “Very much” (5).

Mental health symptom assessments

Also at enrollment, caregivers are given screening questionnaires for anxiety, depressive, inattention, hyperactivity, and oppositional symptoms. These screener questions are from the DSM-V cross-cutting measures for child depressive and anxiety symptoms, and child/adolescent inattention, hyperactivity, and oppositional symptoms.²⁴ For adolescent anxiety and depressive symptoms, the screening questions were derived from the first two items of the Generalized Anxiety Disorder-7 (GAD-7)²⁷ and Patient Health Questionnaire-9 (adolescent version; PHQ-9A),²⁸ respectively. As with the sleep assessment, members with flagged responses to the mental health screening questions are prompted to complete additional assessments. For anxiety and depressive symptoms in children (ages five to 12), the additional measures are the PROMIS depression and anxiety scales, which are completed by the caregiver.^29,30 For anxiety and depressive symptoms in adolescents (ages 13 to 17), the additional measures are the remaining GAD-7²⁷ items and the remainingPHQ-9A²⁸ items, respectively. For inattention and hyperactivity symptoms, caregivers of all members (regardless of age) complete the 18 SNAP-IV questions corresponding with these symptom domains.³¹ For oppositional symptoms, caregivers complete the 8 oppositional behaviors SNAP-IV questions. The original versions of the PROMIS measures, GAD-7, and SNAP-V subsets were used in this study, but the question about suicidal ideation was removed from the PHQ-9A.

Statistical analysis

Given that the PROMIS sleep assessment was only validated for children and adolescents ages five to 17, children younger than five were excluded from all analyses. Raw PROMIS sleep scores were converted to T-scores based on established criteria for parent-report (children ages five to 12) and self-report (adolescents ages 13 to 17).²⁶ The severity of sleep problems was categorized based on T-scores, with symptom severity categories and corresponding T-score ranges are as follows: none to slight (T-scores less than 55), mild (T-scores 55.0 to 59.9), moderate (T-scores 60.0 to 69.9), and severe (T-scores 70 and greater). For the linear mixed-effects models (discussed below), T-scores missing due to screening out of the full PROMIS measure were imputed based on the response to the screener question. If the screener score indicated that sleep problems were “none”, the PROMIS score was imputed as the midpoint for T-scores in the none to slight range, as based on reporter (Caregiver report: T-score = 46.9; self report: T-score = 45.8). If the screener score indicated that sleep problems were “slight”, the PROMIS score was imputed as the highest T-score in the none to slight range (T-score = 55). Raw PROMIS anxiety and depressive scores were also converted to T-scores based on established criteria.^29,30 PHQ-9A raw scores were divided by 8 and then multiplied by 9 to account for the omitted item.²⁸ Raw scores for the GAD-7 and SNAP-IV subsets were not converted for analysis. Members without a sleep screener or assessment before the start of care were excluded from all analyses (n = 9; 0.7% of eligible members).

Baseline member demographics

Members with moderate or severe sleep problems at the baseline (enrollment), based on sleep PROMIS T-scores, were included in the elevated sleep problems group. Members that screened-out of the PROMIS sleep measure or members with PROMIS sleep T-scores indicating none to slight or mild sleep problems were included in the nonelevated sleep problems group. The following outcomes were reported for each group: age at the baseline (in years), sex, gender-sex conformity, race/ethnicity, mental health condition, and elevated mental health symptom (at the baseline). Age at the baseline was compared between-groups using the Wilcox-signed rank test. Chi-squared tests were used to compare the following distributions between-groups: age group, sex (females vs. nonfemales), gender-sex conformity, race/ethnicity (white vs. non-white), no mental health diagnosis, anxiety disorder diagnosis, depressive disorder diagnosis, ADHD diagnosis, and rates of elevated mental health symptoms (anxiety, depressive, inattention, hyperactivity, and oppositional). Rates of members with coaching, therapy, and prescriber sessions were reported for each group, as were the average number of days between coaching and therapy sessions for members with these types of care (calculated first for each member with at least two sessions, then described for the group). The number of months in care was also described for each group and then all members. Details on the reporting of demographic information can be found in the Supplemental Materials.

Child and adolescent sleep over time in care

Only members with elevated sleep problems at the baseline, as well as a baseline assessment within a month of care start and a follow-up assessment after the start of care, were included in the analyses of sleep problems over time (n = 390). Improvements in sleep problems were first assessed by comparing member's baseline sleep T-scores to their sleep T-score at their last assessment while in care. A decrease in T-score from the baseline to each member's last assessment or screening-out of completing the full PROMIS sleep assessment was considered an improvement in score. Delta T-scores were also calculated for each member from the baseline to each member's last full assessment (i.e. no screened-out assessment). Rates of improvement (symptom reduction) were reported for all members, and delta T-scores were reported for members with a full assessment after the start of care. Note that some member's last full assessment may have occurred before their last assessment (e.g. if they screened-out of their last assessment). Delta T-scores were assessed against 0 using Wilcoxon-signed rank test to determine whether sleep problems changed significantly.

To test whether sleep problems changed over participation with the DMHI, a linear mixed-effects model was used to assess PROMIS sleep T-scores over months in care for members with a baseline assessment within one month of the start of care and a full assessment after the start of care (n = 390). The basic model assessed PROMIS sleep T-score and included a fixed effect of months in care and a random effect of member (de-identified ID) on the intercept. To determine whether any additional predictors should be included in the final model, the following predictors were added to the basic model (one at a time) and compared against the basic model using likelihood ratio tests (LRTs): elevated anxiety symptoms (at the baseline; yes vs. no), elevated depressive symptoms (at the baseline; yes vs. no), elevated inattention symptoms (at the baseline; yes vs. no), elevated hyperactivity symptoms (at the baseline; yes vs. no), elevated oppositional symptoms (at the baseline; yes vs. no), sex (female vs. nonfemale), gender-sex conformity (cis-gendered vs. transgender or nonconforming), and age at the baseline. If the addition of a single predictor improved model fit (statistically significant LRT), it was retained in the final model.

To test whether changes in mental health symptoms were significant predictors of sleep problems over the duration of care with the DMHI, PROMIS sleep T-scores were assessed over months in care for members with elevated mental health symptoms (at baseline). Members were grouped based on whether their mental health symptom (anxiety, depressive, inattention, hyperactivity, or oppositional) improved from the baseline to each member's last assessment, as evidenced by either (1) a decrease in respective assessment score greater than or equal to the calculated reliable change criterion for that age group (see supplemental materials for calculation methods), or (2) screening out of the last assessment score. Linear mixed-effects models of PROMIS sleep T-score, including an interaction of months in care with mental health symptom change type (improved or not improved) and a random effect of member ID on the intercept, were run for each mental health symptom type. For models with a significant interaction of months in care with change type, post-hoc tests were used to assess change over months for each change type. To confirm that the primary findings from the linear mixed-effects models are robust to imputation of screened-out values, all final models were repeated with imputed values excluded. These results are reported in the supplemental material. Throughout, standard descriptive statistics (e.g. percentages, mean and standard deviation [M ± SD], and median and interquartile range [IQR]) were used to describe the data. The alpha level was set to 0.05 for all statistical analyses.

Results

Baseline data and member demographics

Of the 1219 members in the study, 49.1% (n = 599) screened-out of completing the full PROMIS sleep assessment at the baseline. Based on PROMIS sleep assessment T-scores, 3.5% (n = 43) of members had low severity sleep problems, 8.0% (n = 98) had mild severity sleep problems, 28.2% (n = 344) had moderate severity sleep problems, and 11.1% (n = 135) had severe sleep problems. Thus, 60.7% (n = 740) members had nonelevated sleep problems and 39.3% (n = 479) had elevated sleep problems.

Compared to members with nonelevated sleep problems, members with elevated sleep problems were a median of one year older (Z = -4.89, P < .001). There were higher rates of females with elevated sleep problems than females with nonelevated sleep problems (χ²= 34.96, P < .001). The two groups did not differ in their rates of gender/sex nonconformity (χ²= 0.74 P = .39). Members with elevated sleep problems were more predominately white than members with nonelevated sleep problems (White vs. non-white; χ²= 4.88, P = .027). Compared to the nonelevated sleep problems group, the elevated sleep problems group had higher rates of reported anxiety disorder diagnosis (χ²= 10.71, P = .001) and depressive disorder diagnosis (χ²= 28.63, P < .001), and lower rates of ADHD diagnoses (all subtypes; χ²= 4.12, P = .042). Members with elevated sleep problems also had over double the rates of elevated anxiety symptoms (χ²= 136.45, P < .001) and elevated depressive symptoms (χ²= 100.09, P < .001) than members with nonelevated symptoms. They also had higher rates of elevated inattention symptoms (χ²= 10.26, P = .001), but similar rates of hyperactivity and oppositional symptoms (both P > .05) (Table 1).

Table 1.

Member characteristics of those with nonelevated sleep problems and those with elevated sleep problems. Between-groups comparisons are performed with Chi-square tests, unless otherwise noted. P-values < .05 are bolded, P-values < .10 are italicized.

Member characteristics	Nonelevated sleep problems group (60.7%; n = 740)	Elevated sleep problems group (39.3%; n = 479)	Between-group comparison
Age in years: Median (IQR)†	11 (6)	12 (6)	Z = -4.89, P < .001
Sex			χ²= 34.96, P < .001
Female	43.2% (n = 320)	60.8% (n = 291)
Male	56.1% (n = 415)	38.0% (n = 182)
Nonbinary	0.7% (n = 5)	1.3% (n = 6)
Gender-sex conformity			χ²= 0.74, P = .39
Cis-gendered	92.6% (n = 685)	91.0% (n = 436)
Transgender or nonconforming	7.4% (n = 55)	9.0% (n = 43)
Race/ethnicity
White	41.6% (n = 308)	48.2% (n = 231)	χ²= 4.88, P = .027
Other or multi-racial	40.3% (n = 298)	36.1% (n = 173)
Black/African American	7.4% (n = 55)	6.1% (n = 29)
Hispanic/Latino	5.3% (n = 39)	5.2% (n = 25)
Asian	5.4% (n = 40)	4.4% (n = 21)
Mental health condition
Anxiety disorder	25.9% (n = 192)	34.9% (n = 167)	χ²= 10.71, P = .001
Depressive disorder	2.4% (n = 18)	9.6% (n = 46)	χ²= 28.63, P < .001
ADHD	18.4% (n = 136)	13.8% (n = 66)	χ²= 4.12, P = .042
Elevated mental health symptoms
Anxiety	26.6% (n = 197)	60.3% (n = 289)	χ²= 136.45, P < .001
Depressive	21.6% (n = 160)	49.3% (n = 236)	χ²= 100.09, P < .001
Inattention	28.2% (n = 209)	37.2% (n = 178)	χ²= 10.26, P = .001
Hyperactivity	16.1% (n = 119)	19.8% (n = 95)	χ²= 2.57, P = .11
Oppositional	22.6% (n = 167)	26.5% (n = 127)	χ²= 2.26, P = .13
Care type
Coaching	98.6% (n = 730)	99.2% (n = 475)	NA
Therapy	18.4% (n = 136)	25.3% (n = 121)	χ²= 7.87, P = .005
Prescriber	54.3% (n = 402)	57.8% (n = 277)	χ²= 1.31, P = .25

†: Between-groups comparison was performed with Wilcoxon signed-rank test.

In terms of the type of care members participated in, more members with elevated sleep problems were in therapy, as compared to members with nonelevated sleep problems (χ²= 7.87, P = .005). Overall, 55.7% (n = 679) of members had an evaluation with a prescriber, and this rate did not differ between-groups (χ²= 1.31, P = .25). Members were in care between 0.03 and 7.77 months, with those with nonelevated sleep problems in care for a median of 2.57 months (IQR: 2.57) and members with elevated sleep problems in care for a median of 2.97 months (IQR: 2.63). Members in coaching had coaching sessions a median of every 14 days (IQR: 6.2). Members in therapy had therapy sessions a median of every 28 days (IQR: 14). The care programs members participated in are reported in Table 2.

Table 2.

Rates of participation in the primary care programs reported by group.

Care program	Nonelevated sleep problems group (60.7%; n = 740)	Elevated sleep problems group (39.3%; n = 479)
Anxiety	47.7% (n = 353)	54.3% (n = 260)
ADHD	32.4% (n = 240)	24.4% (n = 117)
Depression	9.6% (n = 71)	18.2% (n = 87)
Behavior	10.5% (n = 78)	8.4% (n = 40)
Executive functioning	6.8% (n = 50)	2.1% (n = 10)

Child and adolescent sleep over time in care

Members with elevated sleep problems completed between one and seven sleep assessments total (screener or full assessment): 22.3% (n = 107) completed one assessment, 31.3% (n = 150) completed two assessments, 24.6% (n = 118) completed three assessments, and 21.7% (n = 104) completed a total of four or more assessments. Only those with at least two assessments, and a baseline assessment within a month before the first event were included in the analyses of change in sleep over time. Thus, 348 members were included in the analyses of symptom change (n = 107 with one assessment excluded; n = 32 with an early baseline assessment excluded). All members had at least one full assessment after the start of care.

Change in sleep problems from the baseline to last assessment

For members with elevated sleep problems, 77.3% (n = 269) exhibited improvements in their sleep from the baseline to their last assessment, which was a median of 2.27 months (IQR: 2.21) after the start of care and a median of 2.48 months (IQR: 2.33) after the baseline assessment. Specifically, the distribution of sleep problem severity was as follows at the last assessment: 41.4% (n = 144) screened out, 1.7% (n = 6) low severity, 5.5% (n = 19) mild severity, 35.6% (n = 124) moderate severity, and 15.8% (n = 55) severe; Figure 1. From the baseline to their last full assessment (i.e. no screened-out assessments), sleep T-scores decreased from a median of 67.3 (IQR: 7) to 65.3 (IQR: 6.7), for delta T-scores of significantly less than 0 (Z = -6.35, P < .001).

Figure 1.

Sleep problem symptom severity at the baseline and the last assessment, reported only for members with elevated sleep problems.

Primary predictors of sleep problems

In the primary linear mixed-effects model of sleep scores across months in care with the DMHI, the addition of the following predictors improved model fit: elevated anxiety symptoms (P < .001), elevated depressive symptoms (P = .004), female sex (yes or no; P < .001), and gender-sex conformity (yes or no; P = .003). The addition of the following predictors did not improve model fit: elevated inattention symptoms (P = .19), elevated hyperactivity symptoms (P = .19), elevated oppositional symptoms (P = .22), and age (at the baseline; P = .38). Thus, the final model had fixed effects of months in care, elevated anxiety symptoms, elevated depressive symptoms, female sex, and gender-sex conformity, with a random effect of member on the intercept. Sleep T-score decreased significantly over months in care (F_1,725 = 201.89, P < .001; coefficient: −1.91); Figure 2. The main effect of anxiety symptom severity was statistically significant (F_1,343 = 15.60, P < .001), as those with elevated anxiety symptoms had more severe sleep problems (coefficient: 1.88). Females had more severe sleep problems than nonfemales (F_1,343 = 5.85, P = .016; coefficient: 1.76), as did those with gender-sex nonconformity (F_1,343 = 7.85, P = .005; coefficient: 3.13). The main effect of depressive symptom severity was not statistically significant (F_1,343 = 2.17, P = .14).

Figure 2.

Sleep problems decreased over months in care with the DMHI.

Mental health symptom improvement as a predictor of changes in sleep problems

For the analyses of mental health symptom improvement and sleep problems, the number of members in the groups for each mental symptom type are shown in Table 3. The interaction of change type with months in care was significant for all symptoms (P < .001), such that members with improved mental health symptoms had larger improvements in their sleep problems over time. Members with improved mental health symptoms had significant decreases in their sleep T-score each month for all mental health symptom types (all P < .001; estimate range: −2.51 to −2.31). Members with no improvement in their anxiety and depressive symptoms did not have significant changes in their sleep T-score each month (Anxiety: P = .20; Depressive: P = .71), whereas and members with no improvement in their inattention, hyperactivity, and oppositional symptoms had decreases in their sleep problems (All P < .05). The results from the interaction of symptom change type with months in care, and post-hoc statistical tests, are shown for all mental health symptom types in Table 4.

Table 3.

Number of members in each group (improved and not improved) and durations between the baseline and the last assessment are reported for each symptom type assessed in secondary analyses.

Symptom type	Improved		Not improved
Symptom type	Members; n (%)	First to last assessment (months); Median (IQR)	Members; n (%)	First to last assessment (months); Median (IQR)
Anxiety (n = 212)	n = 144 (67.9%)	3.50 (2.09)	n = 68 (32.1%)	3.27 (2.28)
Depressive (n = 170)	n = 121 (71.2%)	3.43 (1.77)	n = 49 (28.8%)	3.50 (1.93)
Inattention (n = 133)	n = 72 (54.2%)	3.28 (2.27)	n = 61 (65.8%)	3.43 (2.43)
Hyperactivity (n = 72)	n = 24 (33.3%)	3.28 (1.88)	n = 48 (66.7%)	3.37 (2.94)
Oppositional (n = 94)	n = 53 (56.4%)	3.27 (2.13)	n = 41 (43.6%)	3.33 (3.13)

Table 4.

Results from each linear mixed-effects model of sleep problems with an interaction of change type and months in care. For models with a significant interaction term, post-hoc analyses of T-score change per month are reported for improved symptoms and not improved symptoms. P-values > .05 are bolded.

	Months in care x Change type
	Main interaction:			Post-hoc estimate Group: Improved			Post-hoc: estimate Group: Not improved
	DF	F	P	t	P	Value	t	P	Value
Anxiety	2447	82.95	<.001	−12.88	<.001	−2.50	−1.28	.20	−0.38
Depressive	2357	47.53	<.001	−9.72	<.001	−2.33	−0.38	.71	−0.13
Inattention	2278	43.08	<.001	−8.91	<.001	−2.51	−3.60	<.001	−1.12
Hyperactivity	2150	22.42	<.001	−5.03	<.001	−2.39	−4.96	<.001	−1.61
Oppositional	2188	23.82	<.001	−6.68	<.001	−2.31	−2.51	.013	−0.99

Discussion

Principal results

The purpose of this study was to (1) determine whether child and adolescent sleep problems improve over time while participating in a nonsleep DMHI, and (2) assess correlations between sleep problems and anxiety, depressive, and ADHD symptom severity at the baseline and improvement over time. We found that 77% of children and adolescents with elevated sleep problems exhibited improvements in their sleep from before the start of care to their last assessment. Member characteristics at the baseline, especially elevated anxiety, depressive, and inattention symptoms, predicted more severe sleep problems. Further, members whose mental health symptoms improved over the course of care with the DMHI had larger improvements in their sleep than members whose mental health symptoms did not improve. To our knowledge, this is the first study to assess whether participation in a nonsleep DMHI confers secondary benefits to youth's sleep problems.

More than three in four children and adolescents with elevated sleep problems showed improvements in their sleep from before the start of care to their last assessment, after a median of 2.3 months in care. Each additional month in care was associated with larger decreases in the severity of sleep problems. Our study is novel in that all children and adolescents with sleep problems participated in care programs that did not directly target sleep, though improving sleep habits and patterns may have been a secondary goal of care, and 77% of these showed improvements in sleep. Therefore, our results suggest that mental and behavioral health care in a pediatric DMHI can confer secondary benefits on sleep, even when sleep problems are not directly targeted by the intervention. Other studies have demonstrated the effectiveness of digitally delivered sleep interventions, such as cognitive behavioral therapy for insomnia and individually-targeted sleep guidance, in mitigating child and adolescent sleep problems.^32–36 Additionally, previous research within the current DMHI and elsewhere has found that youths’ participation is associated with improvements in anxiety, depressive, and ADHD symptoms.^5,6,37,38

Moreover, our findings suggest that the close associations between sleep and comorbid mental health symptoms, which have been established in populations participating in traditional therapeutic modalities,^{18,19,39–43} extend to youth participating in a DMHI. In the current sample, children and adolescents with elevated sleep problems had higher rates of anxiety, depressive, and hyperactivity symptoms than their peers with nonelevated sleep problems. Children and adolescents with mental health problems exhibit higher rates of sleep difficulties.^18,42 Indeed, sleep disruption is considered a symptom of many mental health disorders,²⁴ though this does not necessarily mean that impairments in sleep are produced by psychiatric illness. Research has shown that poor sleep is a risk factor or precursor to many mental health challenges, including suicide risk.^44,45 Further, poor sleep may perpetuate mental health symptoms in children and adolescents.⁴² Conversely, improved sleep may maximize the benefits of other mental and behavioral interventions.⁴⁶ Our findings not only support the strong association between sleep and mental health, but also highlight the benefits of assessing and addressing sleep problems even among those seeking care for nonsleep symptoms. Given that few large-scale pediatric DMHIs have incorporated treatment for sleep problems into their care programs, our findings highlight a clear area for growth in digital mental health care for youth.

Our findings also align with a large body of research, which suggests that those with elevated sleep problems were more likely to be older and female than those with nonelevated sleep problems. Although sleep requirements in adolescence remain higher than in adulthood, adolescents exhibit a delayed circadian rhythm that underlies a strong biological preference for going to bed later and waking up later.^47,48 These biological forces are at odds with common external demands that compete with sleep, such as extracurriculars, social activities, media use before bed, and early school start times. Taken together, it is unsurprising that adolescents are chronically sleep deprived. In terms of sex, evidence suggests that females have higher sleep needs and also more commonly exhibit sleep problems (such as insomnia) in comparison to their male peers, with sex-differences most consistently detected during puberty and throughout the adult lifespan.^49,50 Although mental health care should be guided primarily by patients’ individual symptom presentation, DMHIs and practitioners may better identify and treat symptoms when they are aware of the demographic groups in which sleep problems are endemic.

Finally, we found that the magnitude of change in sleep problems varied based on whether participants exhibited improvement (symptom reduction) versus nonimprovement in other elevated mental health symptom domains. Children and adolescents whose mental health symptoms improved had larger decreases in the severity of sleep problems than children and adolescents whose mental health symptoms did not improve. This relationship was most striking for anxiety in depressive symptoms, in that members with no improvement in these symptoms did not demonstrate improvements in their sleep over months in care. Children and adolescents with elevated inattention, hyperactivity, or oppositional symptoms had improvements in their sleep problems, even those whose mental health symptoms did not improve. Recent experimental evidence suggests that adolescents with both insomnia and mental health problems exhibit concurrent improvements in sleep and mental health symptoms after behavioral intervention.³⁶ Conversely, experimentally-induced acute and chronic sleep loss, on the other hand, increases negative affect (e.g. irritability) and impairs emotional regulation in adolescents.^51,52 Taken together, our finding that mental health symptom improvement is linked to greater improvements in sleep is most likely a reflection of the bi-directional relationship between sleep and mental health. To our knowledge, this is the first study to assess sleep in conjunction with mental health symptoms in the context of a pediatric DMHI. These results further demonstrate that DMHIs are well positioned to address sleep problems in children and adolescents, given their capacity to not only target sleep problems directly, but also simultaneously treat comorbid mental health symptoms that exacerbate sleep problems.

Limitations and future directions

While the present study contributes valuable insights into associations between sleep problems and mental health symptoms in the context of a pediatric DMHI, there are a few limitations to address. First, a retrospective study design was utilized in the present study, and thus we did not include a control group (e.g. members not participating in care). Indeed, our results cannot indicate any causal relationships. Another consequence of the retrospective study design is that our results may be biased by self-selection into care, as well as self-selection to remain in care. Future studies should include a more rigorous study design with a control group and data from long-term members.

Given that study participants could “screen out” of completing the full PROMIS sleep measure, PROMIS sleep T-scores were imputed for screened out assessments based on the response to the DSM-V cross-cutting screener question. This method of imputation was selected to reduce biasing the results towards extremely low T-scores for members with slight sleep problems. To confirm that our results were robust to the absence of imputed scores, we also repeated the primary linear mixed-effects models with imputed data excluded (see supplemental material). We found that the pattern of results was largely similar between the original primary and secondary models and the confirmatory (nonimputed) models, except for the results for hyperactivity symptoms, suggesting that the associations between sleep outcomes and hyperactivity identified in the primary models may have been driven in part by imputed values. Future studies would be strengthened by the inclusion of a more thorough method of assessing subjective sleep, in which all participants must complete a comprehensive sleep assessment at every time point.

This study is also limited by the use of only a subjective measure of child and adolescent sleep. The PROMIS sleep measure has been extensively validated for use in pediatric populations similar to the participants in this study, and thus was suitable to assess subjective sleep quality. Subjective reports of child and adolescent sleep (including caregiver report) are a valuable tool for sleep screening, whereas objective sleep measures such as poly-somnography (PSG) are considered more accurate for the characterization of sleep problems in children and adolescents.⁵³ Further, while the association between poor subjective sleep and mental health symptoms is well established, some have found that this relationship is less consistent when using objective sleep measures.^43,54 For example, the effects of sleep-targeted interventions may differ based on the type of sleep measure used.⁵⁵ Finally, the subjective sleep measure used in this study cannot provide information regarding more nuanced markers of sleep quality, timing, and duration, including circadian period and phase, sleep-onset latency, wake after sleep onset, and sleep efficiency.⁵³ Future studies should seek to determine whether our findings extend to objective sleep quality, timing, and duration by including an objective measure of sleep.

The race and ethnicity response options used in this study were limited in the first portion of the study timeframe. The response options were updated (i.e. to more accurately align with US census standards) approximately two-thirds of the way through the study, so our understanding of members’ race/ethnicity was limited for members who enrolled between January and May. Given the inconsistencies associated with changing the measure part-way through, the authors did not include race and ethnicity as potential predictors in the linear mixed-effects models. The rates of white vs. non-white members were compared between groups, on the other hand, to report on the racial diversity of the participants in this study. Future studies would greatly benefit from a measure of race and ethnicity more representative of minority groups, especially considering that there is evidence of racial disparities in sleep problems throughout the lifespan.^56,57 Finally, it was out of the scope of the present study to address whether sleep outcomes may have been associated with the prescription of psychiatric medications. Members with elevated sleep problems were no more likely to see a psychiatric provider than members with nonelevated sleep problems. Nonetheless, future studies should address the role of psychiatric medication in the effectiveness of DMHIs for improving sleep outcomes.

The present study did not address whether specific behavioral intervention methods (e.g. coaching versus therapy) may be more or less beneficial to sleep outcomes than other methods. In future studies, identification of the behavioral and other therapeutic interventions that are most beneficial to sleep outcomes in the context of DMHIs would greatly enhance the quality and efficacy of DMHIs in addressing sleep problems and mental health symptoms. This study also did not address whether lifestyle factors such as technology use influence changes in sleep problems. Given that sleep is affected by myriad behavioral, circumstantial, and mental health issues, this should be addressed in future studies.

Concluding remarks

The present study suggests that participation in a nonsleep DMHI may have positive secondary benefits for child and adolescent sleep. These findings are timely, considering the growing crisis of sleep problems in youth and increasing number of youth engaged in DMHIs for mental and behavioral health care. Our findings also indicate the influence of mental health symptoms on changes in sleep problems throughout DMHI participation, highlighting the importance of measuring and treating both mental health symptoms and sleep for pediatric mental health care. Future studies should seek to pin-point specific intervention methods (e.g. behavioral strategies and participation characteristics) associated with larger positive effects on sleep outcomes.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076241249928 - Supplemental material for Improvements in sleep problems and their associations with mental health symptoms: A study of children and adolescents participating in a digital mental health intervention

Supplemental material, sj-docx-1-dhj-10.1177_20552076241249928 for Improvements in sleep problems and their associations with mental health symptoms: A study of children and adolescents participating in a digital mental health intervention by Darian Lawrence-Sidebottom, Landry Goodgame Huffman, Aislinn Beam, Amit Parikh, Rachael Guerra, Monika Roots and Jennifer Huberty in DIGITAL HEALTH

Supplemental Material

sj-docx-2-dhj-10.1177_20552076241249928 - Supplemental material for Improvements in sleep problems and their associations with mental health symptoms: A study of children and adolescents participating in a digital mental health intervention

Supplemental material, sj-docx-2-dhj-10.1177_20552076241249928 for Improvements in sleep problems and their associations with mental health symptoms: A study of children and adolescents participating in a digital mental health intervention by Darian Lawrence-Sidebottom, Landry Goodgame Huffman, Aislinn Beam, Amit Parikh, Rachael Guerra, Monika Roots and Jennifer Huberty in DIGITAL HEALTH

Footnotes

Acknowledgments

NA.

Contributorship

DLS and LGH collaborated on study conceptualization and methodology, formal analyses, writing the first draft, and data visualization, under the supervision of JH. MR was involved in funding acquisition. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Data availability

The data sets analyzed during the current study are not publicly available, as this would violate Bend Health Inc.'s privacy policy. However, aggregated and anonymized data that are not associated with individual users and does not include personal information is available from the corresponding author on reasonable request.

Declaration of conflicting interest

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article. All authors are employed or contracted with Bend Health Inc., which delivered the treatment used in this retrospective study. However, authors’ employment status and salary are not dependent upon the results of their research.

Ethical approval

This study was deemed not human subjects research by the Biomedical Research Alliance of New York.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Guarantor

JH.

ORCID iDs

Darian Lawrence-Sidebottom

Landry Goodgame Huffman

Supplemental material

Supplemental material for this article is available online.

References

2022 National Healthcare Quality and Disparities Report [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2022 [cited 2023 Feb 8]. (National Healthcare Quality & Disparities Report Chartbooks). Available from: http://www.ncbi.nlm.nih.gov/books/NBK587182/.

Raney

Bergman

Torous

, et al. Digitally driven integrated primary care and behavioral health: how technology can expand access to effective treatment. Curr Psychiatry Rep 2017; 19: 86.

Campo

Geist

Kolko

. Integration of pediatric behavioral health services in primary care: improving access and outcomes with collaborative care. Can J Psychiatry 2018; 63: 432–438.

Sharif-Sidi

Shen

Wong

, et al. Addressing depression and behavioral health needs through a digital program at scale. Healthcare 2021; 9: 100521.

Huffman

Lawrence-Sidebottom

Huberty

, et al. Using digital measurement–based care for the treatment of anxiety and depression in children and adolescents: observational retrospective analysis of bend health data. JMIR Pediatr Parent 2023; 6: e46154.

Lawrence-Sidebottom

Huffman

Huberty

, et al. Using digital measurement–based care to address symptoms of inattention, hyperactivity, and opposition in youth: retrospective analysis of bend health. JMIR Form Res 2023; 7: e46578.

Kazdin

. Addressing the treatment gap: a key challenge for extending evidence-based psychosocial interventions. Behav Res Ther 2017; 88: 7–18.

Moon

Sobolev

Kane

. Digital and mobile health technology in collaborative behavioral health care: scoping review. JMIR Ment Health 2022; 9: e30810.

Crowley

Wolfson

Tarokh

, et al. An update on adolescent sleep: new evidence informing the perfect storm model. J Adolesc 2018; 67: 55–65.

10.

Lewien

Genuneit

Meigen

, et al. Sleep-related difficulties in healthy children and adolescents. BMC Pediatr 2021; 21: 1–11.

11.

Kann

McManus

Harris

, et al. Youth risk behavior surveillance - United States, 2017. Morb Mortal Wkly Rep Surveill Summ Wash DC 2002 2018; 67: 1–114.

12.

Sharma

Aggarwal

Madaan

, et al. Impact of COVID-19 pandemic on sleep in children and adolescents: a systematic review and meta-analysis. Sleep Med 2021; 84: 259–267.

13.

Dutil

Walsh

Featherstone

, et al. Influence of sleep on developing brain functions and structures in children and adolescents: a systematic review. Sleep Med Rev 2018; 42: 184–201.

14.

Spruyt

. A review of developmental consequences of poor sleep in childhood. Sleep Med 2019; 60: 3–12.

15.

Lipari RN, Hedden S, Blau G, et al. Adolescent mental health service use and reasons for using services in specialty, educational, and general medical settings. In: The CBHSQ report. Rockville, MD: Substance Abuse and Mental Health Services Administration (US), 2016. Available from: https://www.ncbi.nlm.nih.gov/books/NBK362074/

16.

Zablotsky

Terlizzi

. Mental health treatment among children aged 5–17 years: United States, 2019. US Department of Health and Human Services, Centers for Disease Control and …; 2020.

17.

Van Dyk

Becker

Byars

. Rates of mental health symptoms and associations with self-reported sleep quality and sleep hygiene in adolescents presenting for insomnia treatment. J Clin Sleep Med 2019; 15: 1433–1442.

18.

Scott

Kallestad

Vedaa

, et al. Sleep disturbances and first onset of major mental disorders in adolescence and early adulthood: a systematic review and meta-analysis. Sleep Med Rev 2021; 57: 101429.

19.

Freeman

Sheaves

Waite

, et al. Sleep disturbance and psychiatric disorders. Lancet Psychiatry 2020; 7: 628–637.

20.

Narmandakh

Roest

Jonge

, et al. The bidirectional association between sleep problems and anxiety symptoms in adolescents: a TRAILS report. Sleep Med 2020; 67: 39–46.

21.

Orchard

Gregory

Gradisar

, et al. Self-reported sleep patterns and quality amongst adolescents: cross-sectional and prospective associations with anxiety and depression. J Child Psychol Psychiatry 2020; 61: 1126–1137.

22.

Craig

Weiss

Hudec

, et al. The functional impact of sleep disorders in children with ADHD. J Atten Disord 2020; 24: 499–508.

23.

Stiles-Shields

Ramos

Ortega

, et al. Increasing digital mental health reach and uptake via youth partnerships. NPJ Ment Health Res 2023; 2: 9.

24.

American Psychiatric Association D, American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-5. Vol. 5. Washington, DC: American psychiatric association, 2013.

25.

Bevans

Gardner

Pajer

, et al. Psychometric evaluation of the PROMIS® pediatric psychological and physical stress experiences measures. J Pediatr Psychol 2018; 43: 678–692.

26.

Forrest

Meltzer

Marcus

, et al. Development and validation of the PROMIS pediatric sleep disturbance and sleep-related impairment item banks. Sleep 2018; 41: zsy054.

27.

Spitzer

Kroenke

Williams

JBW

, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006; 166: 1092–1097.

28.

Johnson

Harris

Spitzer

, et al. The patient health questionnaire for adolescents: validation of an instrument for the assessment of mental disorders among adolescent primary care patients. J Adolesc Health Off Publ Soc Adolesc Med 2002; 30: 196–204.

29.

Irwin

Gross

Stucky

, et al. Development of six PROMIS pediatrics proxy-report item banks. Health Qual Life Outcomes 2012; 10: 22.

30.

Sherlock

Blackwell

Kallen

, et al. Measuring PROMIS® emotional distress in early childhood. J Pediatr Psychol 2022; 47: 547–558.

31.

Swanson

Sandman

Deutsch

, et al. Methylphenidate hydrochloride given with or before breakfast: i. Behavioral, cognitive, and electrophysiologic effects. Pediatrics 1983; 72: 49–55.

32.

Werner-Seidler

Wong

Johnston

, et al. Pilot evaluation of the sleep ninja: a smartphone application for adolescent insomnia symptoms. BMJ Open 2019; 9: e026502.

33.

Tedford

Romano

Gozal

, et al. Digital solutions for sleep problems in children: a pilot study. Pediatr Pulmonol 2022; 57: 1914–1920.

34.

Williamson

Okoroji

Cicalese

, et al. Sleep well! an adapted behavioral sleep intervention implemented in urban primary care. J Clin Sleep Med 2022; 18: 1153–1166.

35.

Lunsford-Avery

Bidopia

Jackson

, et al. Behavioral treatment of insomnia and sleep disturbances in school-aged children and adolescents. Child Adolesc Psychiatr Clin 2021; 30: 101–116.

36.

Cliffe

Croker

Denne

, et al. Digital cognitive behavioral therapy for insomnia for adolescents with mental health problems: feasibility open trial. JMIR Ment Health 2020; 7: e14842.

37.

Lehtimaki

Martic

Wahl

, et al. Evidence on digital mental health interventions for adolescents and young people: systematic overview. JMIR Ment Health 2021; 8: e25847.

38.

Liverpool

Mota

Sales

CMD

, et al. Engaging children and young people in digital mental health interventions: systematic review of modes of delivery, facilitators, and barriers. J Med Internet Res 2020; 22: e16317.

39.

Marten

Keuppens

Baeyens

, et al. Sleep parameters and problems in adolescents with and without ADHD: a systematic review and meta-analysis. JCPP Adv 2023 Mar 11; 3: e12151.

40.

Langberg

Molitor

Oddo

, et al. Prevalence, patterns, and predictors of sleep problems and daytime sleepiness in young adolescents with ADHD. J Atten Disord 2020; 24: 509–523.

41.

Blake

Trinder

Allen

. Mechanisms underlying the association between insomnia, anxiety, and depression in adolescence: implications for behavioral sleep interventions. Clin Psychol Rev 2018; 63: 25–40.

42.

Quach

Nguyen

Williams

, et al. Bidirectional associations between child sleep problems and internalizing and externalizing difficulties from preschool to early adolescence. JAMA Pediatr 2018; 172: e174363–e174363.

43.

Blok

Koopman-Verhoeff

Dickstein

, et al. Sleep and mental health in childhood: a multi-method study in the general pediatric population. Child Adolesc Psychiatry Ment Health 2022; 16: 11.

44.

Dimakos

Gauthier-Gagné

Lin

, et al. The associations between sleep and externalizing and internalizing problems in children and adolescents with attention-deficit/hyperactivity disorder: empirical findings, clinical implications, and future research directions. Child Adolesc Psychiatr Clin 2021; 30: 175–193.

45.

Williamson

Mindell

Hiscock

, et al. Longitudinal sleep problem trajectories are associated with multiple impairments in child well-being. J Child Psychol Psychiatry 2020; 61: 1092–1103.

46.

Leffler

Zelic

Kruser

, et al. Youth and parent report of sleep-based interventions and utilization of technology resources in the treatment of pediatric mood disorders. Clin Child Psychol Psychiatry 2021; 26: 924–937.

47.

Gariépy

Doré

Whitehead

, et al. More than just sleeping in: a late timing of sleep is associated with health problems and unhealthy behaviours in adolescents. Sleep Med 2019; 56: 66–72.

48.

Meltzer

Wahlstrom

Plog

, et al. Changing school start times: impact on sleep in primary and secondary school students. Sleep 2021; 44: zsab048.

49.

Elkhatib Smidt

Hitt

Zemel

, et al. Sex differences in childhood sleep and health implications. Ann Hum Biol 2021; 48: 474–484.

50.

Franco

Putois

Guyon

, et al. Sleep during development: sex and gender differences. Sleep Med Rev 2020; 51: 101276.

51.

Palmer

Oosterhoff

Bower

, et al. Associations among adolescent sleep problems, emotion regulation, and affective disorders: findings from a nationally representative sample. J Psychiatr Res 2018; 96: 1–8.

52.

Short

Chee

. Adolescent sleep restriction effects on cognition and mood. Prog Brain Res 2019; 246: 55–71.

53.

Mazza

Bastuji

Rey

. Objective and subjective assessments of sleep in children: comparison of actigraphy, sleep diary completed by children and parents’ estimation. Front Psychiatry 2020; 11: 495.

54.

Brown

Wilkerson

Boyd

, et al. A review of sleep disturbance in children and adolescents with anxiety. J Sleep Res 2018; 27: e12635.

55.

Perrault

Pomares

Smith

, et al. Effects of cognitive behavioral therapy for insomnia on subjective and objective measures of sleep and cognition. Sleep Med 2022; 97: 13–26.

56.

Guglielmo

Gazmararian

Chung

, et al. Racial/ethnic sleep disparities in US school-aged children and adolescents: a review of the literature. Sleep Health 2018; 4: 68–80.

57.

Jehan

Myers

Zizi

, et al. Sleep health disparity: the putative role of race, ethnicity and socioeconomic status. Sleep Med Disord Int J 2018; 2: 127.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB

0.01 MB