Effects of Community-Based Wraparound Services on Child and Caregiver Outcomes Following Child Protective Service Involvement

Abstract

This study investigated whether statewide delivery of the wraparound service model (WSM) improved child and caregiver outcomes and reduced subsequent child protective service (CPS) contact among families referred to services following a CPS report. Caregivers (n = 247) completed baseline and 6-month interviews to document self-reported engagement in WSM and non-WSM conditions and assess changes in outcomes. Kernel-weighted difference-in-difference (K-DID) models were used to assess program effects, based on reported condition. Child behavior outcomes improved among WSM-engaged families, but differences by condition were non-significant except for internalizing behaviors. Caregiver receipt of WSM was associated with greater retention of behavioral health services, but did not produce statistically significant improvements in their wellbeing. Households in the WSM condition were more likely to be reported to CPS at 6-month follow-up, but this difference was not significant at 12 months and differences in substantiation were not statistically significant. Supplemental analyses compared alternative means of contrasting group effects, highlighting some differences based on method. The WSM produced few significant differential improvements in child or caregiver outcomes and failed to prevent future CPS involvement. Inadequate program fidelity appeared to be a factor in implementation of the WSM, which may have hampered program effectiveness under real-world conditions.

In 2019, state child protective service (CPS) divisions received over 4.4 million referrals for suspected maltreatment and completed investigations for nearly 3.5 million children (U.S. Department of Health and Human Services, 2022). Children involved in CPS investigations are at significant risk of developmental, behavioral, and health related deficits (Horwitz et al., 2012; Lanier et al., 2010; Stahmer et al., 2005). Indeed, over 40% of youth in a national CPS sample showed clinically significant rates of behavioral problems (Burns et al., 2004). Most children remain in the care of their parents after a maltreatment investigation, and less than 25% receive any post-investigation services to address child and family clinical needs (Burns et al., 2004; Hurlburt et al., 2004). Risks to child safety and well-being do not differ markedly even when a case is not substantiated (Kohl et al., 2009; Kugler et al., 2019), thus indicating a need for effective community-based services for families that experience a CPS investigation to address potentially complex behavioral health and family service needs.

The Wraparound Service Model (WSM) is a promising approach to improving child and family outcomes among youth at risk of, or already involved in, child welfare services as a result of CPS involvement (CEBC, 2020). The WSM is a team-based care coordination approach to providing individualized services to children and their families based on identified needs (Coldiron et al., 2017). A defining feature of the model is recurring team meetings with the family, wraparound facilitator, formal providers, and natural supports (Bruns et al., 2010). The model has been implemented in nearly all US states, and several state child welfare systems have adopted variants of the model to coordinate services for children with complex needs or who are involved in child protection or child welfare services (Sather & Bruns, 2016).

The WSM has been shown to improve residential and behavioral outcomes for children and youth with serious emotional disorders (SED; Winters & Metz, 2009) or involved in out-of-home placements in the child welfare system (CWS; Clark et al., 1996; Clark et al., 1998). In a comprehensive review of wraparound services involving over 120 empirical studies, Coldiron et al. (2017) observed positive or mixed support for WSM effects on engagement in community services, improved functioning, and decreased problematic behaviors for youth despite significant variability across studies in study criteria and reporting.

Despite the level of behavioral health challenges among youth involved in CPS, rigorous clinical trials of the WSM are lacking that focus on outcomes among CPS-involved youth and their families. The few existing trials of the model within this context have produced mixed results, which may be attributable to confounding issues such as implementation fidelity and self-selection bias. For example, Browne and colleagues (2016) conducted a randomized controlled trial (RCT) of WSM effects compared to standard care among families substantiated for maltreatment and opened for services. Both conditions were associated with reductions in child functional impairments and parental distress, and improvements in access to family resources. However, the study revealed no added benefit of wraparound services. The authors also observed that fidelity to wraparound principles was poor on several dimensions. These findings parallel another RCT that involved youth with a serious emotional disorder, many of whom were receiving either in-home or foster placement services through the CWS (Bruns et al., 2015). Results revealed non-significant post-treatment group differences and poor wraparound implementation fidelity. Finally, Schneider-Muñoz and colleagues (2015) observed that CPS-involved families that completed county-based wraparound services were less likely to experience a new substantiated maltreatment incident within 6 months than families referred for services who did not complete the program, but the study did not address potential selection effects among program participants or assess program fidelity or other implementation factors.

This study extends prior research to investigate the effects of wraparound services for CPS-referred households in two important ways. First, we recruited participants referred to services as part of a statewide implementation of the WSM for families recently involved in a CPS investigation, providing an opportunity to look at program effects in the contexts of a real-world dissemination of the model in a system-wide implementation. Second, we employed a quasi-experimental causal inference strategy to estimate the conditional average treatment effect of community-based wraparound services, drawing on both administrative data and caregiver assessment data to mitigate selection effects. We used a nonparametric method (i.e., kernel-weighted difference-in-difference modeling (K-DID); Heckman et al., 1998) with a prospective cohort design to reduce the bias that arises from using a comparison group of self-selected non-participants. We addressed the following research questions: (1) does receipt of WSM improve child-level outcomes (e.g., reduced behavioral problems, increased engagement in formal services); (2) does receipt of WSM improve caregiver outcomes (e.g., improved behavioral health outcomes and parental attitudes, increased engagement in formal services); and (3) does receipt of WSM reduce recurrence of CPS contact among families?

Method

Study Context

WSM was delivered in the context of a statewide system of care supported through four regional provider networks contracted with the state’s child welfare agency. Each regional network was comprised of local agencies that completed training in the model from external consultants contracted by the state, and each developed a community-based referral network to provide access to a range of services including crisis stabilization services, case management, child and family counseling, and housing/financial support services. Network staff operated under the practice model that all referred families were eligible for WSM, but that engagement was voluntary. Families that did not engage in the model may have exited the system of care, been referred to an external service provider outside the context of a wraparound team, or received other forms of support not involving WSM involvement (e.g., resource referral).

Study Recruitment and Enrollment

Intake coordinators in each of the regional networks were trained to engage families and introduce them to the WSM. The study team worked with network staff, who provided recruitment materials to potentially eligible families and obtained permission for study outreach to screen and recruit participants. Eligibility criteria included: (1) the child and family were referred to a community provider within the statewide WSM network by a CPS caseworker following a completed maltreatment investigation, (2) the child remained in parental custody (i.e., not placed in an out-of-home child welfare placement) at the conclusion of the CPS investigation, and (3) the child was between the ages of 1.5 and 15 years of age. Study staff conducted a brief phone screen and consent process for families who met eligibility criteria and expressed interest in participating. A total of 523 families were referred to the study by network sites, and 366 were successfully screened for eligibility (70.0%). Of these, 31 (8.5%) were ineligible to participate, and 88 (24.0%) declined to participate or failed to complete an intake interview. A total of 247 families (67.5% of those screened for eligibility) were enrolled in the study and completed a baseline interview.

Measures

Independent Variable

Individuals were classified into treatment or comparison condition (WSM or non-WSM, respectively) based on participant responses to Section A of the Wraparound Fidelity Index-Short Form (WFI-EZ; Pullmann et al., 2013; Sather et al., 2013), completed at the 6-month interview. Section A included four items that assess exposure to fundamental components of the WSM (i.e., presence of a wraparound team, a written wraparound plan of care, regular team meetings, and family input on team decisions) ¹ . Evidence of wraparound team meeting occurrence in agency administrative records (described below) was also used to supplement WFI-EZ items and reflect the affirmative occurrence of team meetings (i.e., an affirmative response to team meeting occurrence or an indication that such meetings had occurred in administrative records was accepted for this indicator).

No formal cut-point of exposure to core components is provided for the WFI-EZ as indicative of WSM exposure. We assigned families to the WSM condition who reported they had experienced at least 3 of 4 components; those who reported they had experienced 0 to 2 of the components were treated as comparison cases in the non-WSM condition. A total of 60 participants (35% of the follow-up sample) met criteria for classification in the WSM treatment condition, and 112 participants (65%) met criteria for classification in the non-WSM comparison condition. Among the WSM condition, 72% indicated exposure to all four WSM components and 28% to three components. Among the non-WSM condition, 45% indicated no exposure to WSM components, 47% to one component, and 8% experienced exposure to two components.

Child and Family Characteristics

Caregivers provided information about child and family demographic characteristics and household information as part of the baseline interview. In addition, children’s history of trauma exposure was assessed using the Trauma History Screen (THS), a measure of trauma history and subjective distress related to each type of 20 different potentially traumatic events. The THS was adapted from the Traumatic Events Screening Inventory (TESI; Ghosh-Ippen et al., 2002) and the Trauma History Questionnaire (Berkowitz & Stover, 2005) to provide a simple tool that screens for a number of types of trauma exposure, as well as multiple exposures to the same type of event.

Child Behavior and Functioning

Parents rated child behavior using the Child Behavior Checklist (CBCL; Achenbach & Rescorla, 2000, 2001), a well-validated and reliable norm-referenced measure of parental ratings of their child’s behavioral and emotional problems, appropriate for children ages 1.5–18 years, using both the preschool and school-age child versions. CBCL computer scoring produces scores for a number of clinical syndromes, as well as a total problem score and ratings of internalizing and externalizing behavior problems. These three primary scores all have excellent reliability and are sensitive to change over time.

Parent Functioning

Parent functioning was rated using four instruments. The Parenting Stress Index-Short Form (PSI-SF; Abidin, 1990) was used to measure parental role stress based on three domains: parental distress, parent-child dysfunctional interactions, and difficult child. The total score has good reliability (alpha = .84). Parental depression was assessed using the Center for Epidemiological Studies (CESD-R; Eaton et al., 1998), a brief measure of frequency of experiencing depressive symptoms with excellent psychometric properties. Parent’s attitude toward caregiver practices was assessed using the Adult-Adolescent Parenting Inventory-2 (AAPI-2; Bavolek & Keene, 1999), which reflects caregivers’ developmentally appropriate expectations, empathy for their child, and views on corporal punishment; subscale scores have good reliability (alpha = .80; Conners et al., 2006).

Service Use

Child and family service access was assessed using the Services Assessment for Children and Adolescents (SACA; Horwitz et al., 2001), a parent survey that assesses use/dosage of 30 different child services including inpatient, outpatient, and school-based services with moderate-to-excellent reliability. Caregiver’s use of behavioral health services was assessed using the NSCAW Caregiver Service Use Instrument (NSCAW Research Group, 2002).

Intervention Fidelity

Section B of the WFI-EZ (Pullmann et al., 2013; Sather et al., 2013) was used to assess WSM implementation fidelity. This section consists of 25 items that assess the degree to which the wraparound process reflects overall and domain-specific fidelity to five ‘key elements’ of implementation. Specific domains include: being needs-based, strengths- and family-driven, and outcomes-oriented; engaging in effective teamwork; and drawing on natural and community supports. Total and domain scores are rated and converted to a score from 0 to 100; a total score of 80 or higher indicates high-fidelity implementation, scores of 75–79 indicate adequate fidelity, scores of 70–74 indicate borderline fidelity, and scores below 70 indicate inadequate fidelity. Separate cut points have been established for subscale scores (Wraparound Evaluation & Research Team, 2021). WFI-EZ scales have excellent reliability (alpha = 0.94).

Administrative Records

Data was extracted from the state’s statewide automated child welfare information system (SACWIS) and a statewide community-based agency information system. The SACWIS is maintained by the state child welfare agency and includes information related to completed CPS investigations. Study referrals were matched to the precipitating CPS investigation in SACWIS based on dates of the incident and of the referral to services. To qualify as a precipitating event, the CPS report had to precede referral by no more than 6 months (Median = 7 days, 86% <30 days). If an incident involving the study child was not located, incidents involving the family and/or caregivers were selected. Ten families were not able to be matched to a precipitating event occurring within 6 months prior to community-agency referral, which may indicate that the case was referred based on a CPS incident that did not result in a completed investigation or that the case was mistakenly not involved in a CPS investigation (though this was asked by the referring agency and as part of a study pre-enrollment screening process with caregivers). Coding of the precipitating event included prior history of substantiated maltreatment, allegation type, case finding, and safety assessment results. SACWIS data was used to identify subsequent occurrences of CPS report and substantiated maltreatment at 6- and 12-months after referral. The community-based agency information system is the web-based data system for the state child welfare authority’s behavioral health system of care and includes child and family data collected by providers for families referred for services. The information system included dates of intake and indicators of wraparound team meeting occurrence.

Data Analysis

We used a quasi-experimental design to assess intervention effects of the WSM compared to like-families referred for WSM but who did not engage in wraparound supports (i.e., non-WSM). For continuous outcomes, we used K-DID methods to account for potential selection bias associated with the quasi-experimental design. DID analyses compare the treatment effect between two non-identical groups while accounting for both within-group differences over time and between group differences at each time (Murray, 2005). Heckman, Ichimura, and Todd (1998) developed a nonparametric extension of the DID approach that uses kernel weights derived from propensity scores (i.e. the probability of being treated). This approach estimates the average treatment effect on the treated (i.e., an estimate of the average effects that would be obtained if all families in the treatment condition received the WSM, compared to if none of the families in the treatment condition received the WSM). Propensity scores were estimated through logistic regression based on an extensive set of observed covariates from both survey and administrative data sources (Stuart & Rubin, 2007). Propensity covariates included caregiver, child, and household demographic indicators; network provider; baseline child and caregiver assessment scores; and characteristics of the index CPS incident precipitating referral to services. A strength of this approach is that it allowed us to match the treated group to the entire control group, rather than a small subset of nearest neighbors, by down-weighting distant observations rather than discarding them. The equation for this K-DID approach is as follows

${\delta }^{\mathrm{K}\mathrm{D}\mathrm{I}\mathrm{D}}=\left[\mathrm{E}\left({\mathrm{Y}}_{\mathrm{i}\mathrm{F}}|T_{iF}=1,\hspace{0.17em}{Z}_i=1\right)-w_i\mathrm{E}\left({\mathrm{Y}}_{\mathrm{i}\mathrm{F}}|T_{iF}=0,\hspace{0.17em}{Z}_i=0\right)\right]-\left[\mathrm{E}\left({\mathrm{Y}}_{\mathrm{i}\mathrm{B}}|T_{iB}=0,\hspace{0.17em}{Z}_i=1\right)-w_i\mathrm{E}\left({\mathrm{Y}}_{\mathrm{i}\mathrm{B}}|T_{iB}=0,\hspace{0.17em}{Z}_i=0\right)\right]$

where, $w_i$ is the estimated kernel weight, Z indicates treatment group (1 = WSM; 0 = non-WSM), T indicates the given treatment status, and the subscripts B and F refer to baseline and follow-up assessments, respectively. We obtained estimates for the following effects using linear regression (Villa, 2016): time period, treatment condition, and a time X treatment interaction effect. A similar approach was used to compare proportional rates of CPS outcomes across condition.

Sample Attrition

A total of 247 households completed baseline data collection, and 181 households completed the 6-month follow-up (73.3% retention). We used logistic regression analyses to predict sample attrition based on demographic variables, baseline survey responses, and case characteristics. Several variables were statistically significantly associated with likelihood of attrition (e.g., lower income, racial minority status, no occurrence of team meetings in the agency information system, and subscales of the PSI-SF). Because treatment engagement was based on 6-month interviews and research on imputation using propensity score methods is limited, only households completing both waves of data collection were included in our model (final n = 172).

Results

Group Comparability

Household sociodemographic information, case characteristics, and baseline child and caregiver outcomes for participants conditions are summarized in Table 1. Analyses compared pre- and post-weighting group differences based on the kernel-weight derived from propensity scores. No statistically significant differences were observed for caregiver, child, or household demographic indicators, though post-weighted differences were generally smaller in magnitude than pre-weighted differences. Significant pre-weighting differences were observed for several characteristics of the CPS incident that precipitated referral to wraparound service providers including a longer time lag from incident to referral, lower rates of alleged and substantiated neglect, higher rates of information service referrals, and fewer ‘conditionally safe’ (rather than safe or not evaluated) determinations from the CPS safety screen among those in the WSM condition. Some differences were observed among groups on baseline measures of child functioning and service involvement including higher ratings of externalizing behavior problems and increased involvement in behavioral health services at baseline among those in the WSM condition. No statistically significant differences were observed for baseline ratings of caregiver behavioral wellbeing (e.g., parenting stress or depression), attitudes toward caregiving practices, or engagement in behavioral health services. All group differences were reduced to non-significance with the application of kernel-weighting procedures.

Table 1.

Participant characteristics by treatment condition.

	WSM		Non-WSM		Pre-weight Difference	Post-weight Difference
	(N = 60)		(N = 112)
	Mean/Proportion	SD	Mean/Proportion	SD
Demographic Characteristics
Caregiver age (years)	31.600	0.979	32.339	0.600	-0.739	0.406
Child age (years)	6.983	0.187	6.125	0.129	0.858	0.312
Caregiver gender (1=female)	0.950	0.001	0.893	0.001	0.057	0.034
Child gender (1=female)	0.367	0.004	0.455	0.002	-0.089	0.044
Caregiver race/Ethnicity (1=minority status)	0.600	0.004	0.571	0.002	0.029	-0.050
Caregiver education
< High school/GED	0.317	0.004	0.268	0.002	0.049	0.039
High School/GED	0.400	0.004	0.375	0.002	0.025	0.006
Some college/College	0.283	0.003	0.357	0.002	-0.074	-0.045
Marital status
Married/Living with partner	0.317	0.004	0.321	0.002	-0.005	0.067
Divorced/Separated/Widowed	0.117	0.002	0.161	0.001	-0.044	-0.007
Never married	0.567	0.004	0.518	0.002	0.049	-0.060
Annual family income
< $20,000	0.567	0.004	0.634	0.002	-0.067	-0.001
$20,000-$39,000	0.317	0.004	0.286	0.002	0.031	0.016
> $40,000	0.117	0.002	0.080	0.001	0.036	-0.015
CPS Incident Characteristics
Time from incident to referral (days)	-18.450	12.570	-11.929	1.925	-6.521*	-1.876
CPS Incident involved study child	0.833	0.002	0.857	0.001	-0.024	0.081
Previous substantiation History	0.450	0.004	0.402	0.002	0.048	-0.020
CPS Allegation type
Neglect	0.550	0.004	0.777	0.002	-0.227**	0.058
Physical abuse/Sexual abuse	0.317	0.004	0.250	0.002	0.067	-0.094
Information service referral/None	0.167	0.002	0.045	0.000	0.122**	0.040
CPS Substantiation type
Neglect	0.367	0.004	0.554	0.002	-0.187*	0.053
Physical abuse/Sexual abuse	0.183	0.003	0.125	0.001	0.058	-0.105
Safety assessment (0=Safe/Not evaluated)
Conditionally safe	0.567	0.004	0.741	0.002	-0.174*	-0.035
Baseline Child Outcome Variables
Total behavior problems	61.183	2.400	57.777	1.454	3.407†	1.659
Externalizing behavior problems	62.600	2.849	57.393	1.497	5.207*	1.208
Internalizing behavior problems	55.317	2.200	55.464	1.179	-0.148	1.614
Child behavioral Health services	1.617	0.046	1.134	0.021	0.483†	0.128
Baseline Caregiver Outcome Variables
Parental stress	85.533	8.155	83.205	3.451	2.328	0.089
Depression	13.700	2.104	13.991	1.507	-0.291	0.158
Parental expectations of child	18.067	0.293	18.330	0.149	-0.264	-0.457
Parental empathy	39.033	0.350	37.991	0.245	1.042	0.299
Use of corporal punishment	42.433	0.719	41.393	0.269	1.040	1.305
Parent and family roles	21.933	0.325	22.705	0.201	-0.772	0.493
Child power and independence	18.967	0.053	19.196	0.057	-0.230	0.050
Caregiver behavioral Health services	1.967	0.051	2.000	0.024	-0.033	0.028

Note. Propensity score estimation also accounted for provider site and legal custody arrangement, not included in table. ** p < .01, * p < .05, † p < .1.

Child Outcomes

The first set of outcome analyses examined effects of WSM receipt on child-level outcomes including caregiver ratings of behavior problems and engagement in formal behavioral health services (see Table 2). A sizeable and statistically significant decrease in total behavior problems and in externalizing and internalizing behavior problems was observed at 6-month follow-up among those in the WSM condition, and a statistically significant change was not observed among those in the non-WSM condition. However, only the K-DID analyses for internalizing behaviors indicated the difference in change scores by condition was statistically significant. Child behavioral health service involvement was unaffected by receipt of the WSM. Children in either condition were involved in between one and two unique behavioral health services at baseline, and at 6-month follow-up this level remained relatively unchanged.

Table 2.

Wraparound service model effects on child outcomes (K-DID Estimation).

	Condition	WSM	Non-WSM	Condition Diff
Behavior Problems
Total problem behavior	Baseline	60.54	58.88	1.66
	6-Month	55.54	58.05	-2.51
	BL-6m Diff	-5.00*	-0.83	-4.17
Externalizing behavior	Baseline	61.81	60.60	1.21
	6-Month	56.04	59.08	-3.04
	BL-6m Diff	-5.77**	-1.52	-4.25
Internalizing behavior	Baseline	54.94	53.33	1.61
	6-Month	50.88	54.37	-3.48*
	BL-6m Diff	-4.06*	1.04	-5.10*
Formal Supports
Behavioral Health services	Baseline	1.54	1.41	0.13
	6-Month	1.67	1.41	0.26
	BL-6m Diff	0.13	0.00	0.13

** p < .01, * p < .05, † p < .1.

Caregiver Outcomes

The second set of outcome analyses examined effects of WSM receipt on caregiver outcomes including caregiver behavioral health indicators, attitudes toward parenting, and engagement in formal behavioral health services (see Table 3). Differences in changes to caregiver ratings of parental stress and depressive symptoms were largely unaffected by the wraparound condition. Both groups demonstrated small, statistically non-significant changes in each of these indicators and the K-DID results were not statistically significant. Similarly, neither group demonstrated statistically significant changes in parenting attitudes, and the K-DID outcomes for these domains were not statistically significant.

Table 3.

Wraparound service model effects on caregiver behavioral health outcomes (K-DID Estimation).

	Condition	WSM	Non-WSM	Condition Diff
Behavioral Health
Parental stress	Baseline	86.08	85.99	0.09
	6-Month	80.58	86.63	-6.05
	BL-6m Diff	-5.50	0.64	-6.14
Depression	Baseline	13.87	13.71	0.16
	6-Month	12.56	13.52	-0.96
	BL-6m Diff	-1.31	-0.19	-1.12
Parenting Attitudes
Expectations of child	Baseline	17.92	18.38	-0.46
	6-Month	18.12	17.81	0.31
	BL-6m Diff	0.19	-0.57	0.76
Parental empathy	Baseline	38.87	38.57	0.30
	6-Month	38.85	38.10	0.75
	BL-6m Diff	-0.02	-0.47	0.45
Use of corporal punishment	Baseline	42.08	40.77	1.31
	6-Month	40.94	40.45	0.50
	BL-6m Diff	-1.13	-0.33	-0.81
Parent-child family roles	Baseline	21.94	21.45	0.49
	6-Month	22.92	22.34	0.59
	BL-6m Diff	0.98	0.89	0.09
Child power and independence	Baseline	19.02	18.97	0.05
	6-Month	18.52	18.43	0.09
	BL-6m Diff	-0.50	-0.54	0.04
Formal Supports
Behavioral Health services	Baseline	2.04	2.01	0.03
	6-Month	2.17	1.20	0.97**
	BL-6m Diff	0.13	-0.81**	0.94*

** p < .01, * p < .05, † p < .1.

Caregiver behavioral health service involvement was affected by treatment condition, however. At baseline, caregivers in the WSM condition were involved in an average of two unique services, and this involvement remained stable with a non-significant increase in involvement. Conversely, caregivers in the non-WSM condition demonstrated a statistically significant decrease in service involvement over the 6-month follow-up period. K-DID results confirmed that this difference in service involvement trajectories across condition was statistically significant. Thus, while WSM caregiver service engagement remained stable, caregivers in the non-WSM condition reduced their engagement in formal services over time.

Safety Outcomes

The final set of analyses compared effects of WSM on recurrence of CPS reports and of new substantiated maltreatment incidents at 6- and 12-month follow-up intervals (see Table 4). At the 6-month follow-up, concurrent with other outcome analyses, there was a statistically significant increase in rates of new reports to CPS among children in the WSM condition—approximately 25% of WSM cases had a new report compared to 10% in the non-WSM condition. Further, there was a trend-level pattern of higher rates of substantiated maltreatment at the 6-month follow-up, as well (10% vs. 2%). At the 12-month follow-up, group differences in reporting were no longer statistically significant, though rates of new substantiated incidents were higher at a trend-level difference among children in the WSM condition.

Table 4.

Wraparound service model effects on child protective service outcomes (K-D Estimation).

	WSM	Non-WSM	Difference
6-month outcomes
New report	0.25	0.10	0.15*
New substantiation	0.10	0.02	0.08†
12-month outcomes
New report	0.33	0.23	0.10
New substantiation	0.15	0.06	0.09†

** p < .01, * p < .05, † p < .1.

Sensitivity Analyses

We conducted a series of post-hoc analyses to assess robustness of findings to alternative means of treatment assignment based on self-reported exposure to WSM components. The first compared outcomes among cases who received all four WSM components with those who received none of the components (i.e., full WSM exposure vs. no exposure/non-WSM). The second compared outcomes among cases who received all four WSM components with those who received three or fewer components (i.e., full WSM exposure vs. partial or no-exposure/non-WSM). The final sensitivity analyses compared cases who were exposed to one or more WSM components with those who received none (i.e., partial-to-full WSM vs. no-exposure/non-WSM). Results of these sensitivity analyses are presented in a supplementary appendix. The comparison of full WSM implementation to no exposure/non-WSM cases produced some changes. Though the direction of effects was consistent with primary findings, differences in child behavioral outcomes were not statistically significant and differences in CPS outcomes were not significantly different across WSM conditions; caregiver service involvement effects remained marginally significant. For the comparison of full WSM implementation to partial or no-exposure/non-WSM conditions—all child and caregiver findings remained, though statistical differences were no longer observed for any CPS outcomes. Finally, the comparison of partial-to-full WSM exposure to no-exposure/non-WSM conditions produced a similar pattern of results to the first sensitivity model: none of the child behavioral improvements were statistically significant (though the directionality remained present for total problems and externalizing behaviors) and CPS effects were also not significant; only caregiver service use effects remained consistent with primary results.

Taken together, the first two sets of analyses suggest that a more stringent criteria for WSM assignment largely maintained directionality of program effects for child and caregiver outcomes, though robustness of effects may have been adversely impacted due to a reduced sample size. Conversely, a broader definition of treatment assignment, including full, partial, and incomplete WSM exposure reduced child-related outcomes, though caregiver service outcomes were maintained relative to the no-exposure condition.

Implementation Fidelity

We conducted follow-up analyses to examine implementation fidelity for families in the WSM condition. Based on existing cut-off scores (WERT, 2021), mean rating of overall fidelity (68.48, SD = 12.53) were in the inadequate range; 19% of cases rated 75 or greater (adequate fidelity) and only 10% rated 80 or greater (high fidelity). Mean fidelity ratings for other subscale domains were largely consistent with this pattern, falling in the inadequate or borderline range for each dimension including using an outcomes-based approach (70.56, SD = 17.23), effective teamwork (68.68, SD = 14.55), addressing client-focused needs (68.97, SD = 14.60), and being strength and family-focused (69.17, SD = 15.72). Caregivers rated efforts to build natural and community supports as poorer (54.68, SD = 18.16). Fidelity was not assessed for those caregivers who reported no exposure to any wraparound components, but overall ratings of fidelity were significantly higher for families who had experienced four wraparound criteria than those who had experienced three or fewer (68.78 vs. 62.55, p = 0.024). A similar pattern was also observed for subscale scores, though only teamwork and natural supports reached statistically significant levels.

Discussion

This study investigated the effects of wraparound services for children and their caregivers who were referred to community-based providers following CPS contact, adding to the limited WSM research base with this population. We evaluated WSM effects using K-DID models, which allowed us to assess effects over time while statistically adjusting for pre-intervention differences on demographic, case, and baseline child and caregiver functioning and service involvement levels across WSM and non-WSM conditions. Our study examined caregiver reports of WSM implementation fidelity and assessed changes across a broad range of child and family-level outcomes.

Before discussing study findings, it is interesting to note that of those enrolled in the study approximately 35% of families referred to services reported they had engaged in WSM through community provider networks, operationalized as receiving three of four components. Though providers operated under a practice model that all families should be engaged, a core tenant of the WSM is the voluntary nature of services. Though our rate of engagement may appear low, successful engagement of CPS-involved families in voluntary services is a known issue. Putnam-Hornstein et al. (2021) reported a 36% rate of successful contact for CPS-involved families randomized to community outreach for prevention services, and only 18% of those contacted engaged in voluntary services. Slack et al. (2022) reported 32% of families randomized to a Community Response Program, a voluntary strengths-based post-CPS intervention, engaged services. Thus, our engagement rates are comparable to those of similar voluntary community-based intervention trials.

Our analyses of effects on both child and caregiver outcomes produced mixed results with limited support for WSM. We observed a statistically significant decrease in children’s total behavior problems and in externalizing and internalizing behaviors among children in the WSM condition, with no such decreases among children in the non-WSM condition, though comparisons of differences in change scores across condition were only significant for internalizing symptoms. Further, there was no effect of WSM exposure on involvement in formal children’s behavioral health services. Youth in both conditions generally remained stable with respect to the number of different services they were involved in at baseline and 6-month follow-up periods. For caregivers, WSM receipt was associated with stable rates of service involvement, while those in the non-WSM condition reported a decline in service involvement. However, this pattern of stable service involvement did not appear to have a significant effect on caregiver wellbeing outcomes. Our final set of analyses examined rates of new and substantiated reports to CPS at 6- and 12-month follow-ups. In the short-term, families in the WSM condition were significantly more likely to be re-reported to CPS at 6 months, but substantiation rates and later re-reports were not statistically significantly different. Even under alternative group assignment conditions, the general pattern of limited effects for child, caregiver, or safety outcomes was fairly stable.

The pattern of findings for child and caregiver wellbeing and service engagement outcomes is consistent with findings from two recent randomized trials of the WSM involving CPS (WSM vs. usual care; Browne et al., 2016) and CWS-involved families (WSM vs. intensive case management; Bruns et al., 2015). In each, youth in the WSM and comparison conditions demonstrated similar reductions in behavioral impairment, with minimal differences in rates of improvement across condition. Caregivers in the Browne study demonstrated an overall slight reduction in psychological distress and no change in parental stress, with no condition-by-time differences. Further, neither of these studies demonstrated dramatic changes in service involvement. Bruns and colleagues reported a trend toward receipt of home-based services and a significant increase in hours of such services among WSM-referred families; and Browne and colleagues reported an overall effect of improvements in family resources, such as necessities and other supports, that did not differ across condition. Our findings suggest WSM contributed to maintenance of existing service levels for caregivers, rather than facilitation of greater engagement in services, and no conditional differences for child services.

The effects observed for new CPS reports or substantiated cases differed from those of Schneider-Muñoz et al. (2015), which observed caregivers referred to WSM had fewer new substantiated reports at 6-month follow-up. One plausible explanation for our findings may be the additional surveillance of families in the WSM condition. In addition to involvement with their WSM facilitator, caregivers maintained higher rates of formal behavioral health services involvement than those in the non-WSM condition, which could contribute to opportunities for identifying subsequent concerns. Increased rates of CPS involvement due to formal service contact have been observed in other studies (e.g., Chaffin & Bard, 2006), though others have argued that the effects of surveillance bias are relatively small (Drake et al., 2017).

The overall pattern of findings adds further evidence of limited effectiveness for WSM with CPS-involved families, despite previous research showing its benefits for youth with SED or in CWS placement (Burns & Goldman, 1999; Clark et al., 1998; Winters & Metz, 2009). Three factors bear further investigation to understand why WSM failed to produce significant improvements for families in the present study. First, our study mirrors the findings of Browne et al. (2016) and Bruns et al. (2015) in demonstrating a consistent pattern of low fidelity implementation of the WSM in large scale dissemination studies. Wraparound is a process, rather than a prescriptive evidence-based intervention. WSM facilitators work to engage the family, articulate family needs and strengths, and develop an individualized service plan that helps the family to mobilize both natural and formal supports to achieve their objectives. Each of these domains was delivered at sub-optimal levels of fidelity, with especially poor ratings on efforts to engage natural and community supports. With poorer implementation quality, it is unclear that the WSM provides families with much additional support beyond those of other community-based services. Ongoing support for implementation delivery is a critical component of maintaining program fidelity, perhaps especially so for less manualized family service models. Similar to the study conducted by Bruns et al. (2015), state supports for ongoing training and quality assurance were curtailed shortly after initial dissemination of the model through statewide provider networks, which may have limited capacity to deliver the program effectively.

Second, the WSM is intended to foster engagement in both formal and informal supports to generate positive outcomes at the child, caregiver, and family levels. Within the current implementation WSM facilitators relied on existing community-based services and supports available through network partnerships. One explanation given for the lack of differences by Browne et al. (2016) was that treatment services were available and sufficient through community settings, thus having a wraparound facilitator was not a necessary condition to promote service outcomes. It is possible a similar situation occurred in this study, and further exploration of services used by families in WSM and non-WSM conditions may help to explicate these findings. More research is warranted to understand the types of services available in the array, how they were accessed by families in both conditions, and whether service variability was associated with differential treatment outcomes among families.

Finally, the fact that WSM has been successful with other populations, yet has shown limited effects with parents involved in CPS highlights a need to address particular application of the model with this service population. Engagement of CPS-involved families can be difficult even with good training and skilled facilitation (Schreiber et al., 2013), and anecdotal reports from WSM providers indicated unique challenges associated with engaging CPS-involved families relative to other service populations in the system (e.g., juvenile justice, youth with SED). Kim et al. (2016) found that engaging CWS-involved families in family team meetings, an important component of WSM, was challenging and often resulted in conflicts between family members and stakeholders due to competing priorities or family concerns that their priorities were not respected by professionals. The authors further observed that additional efforts to prepare families for the context of family team meetings increased engagement and likelihood of follow through with their service plans (Kim et al., 2016).

Limitations

There are a number of limitations to our study that should be considered. First, we employed a quasi-experimental design in which families were provided the opportunity to engage in WSM, and group assignment for analytic purposes was based on self-reported engagement in at least three core components of the WSM. It is possible that caregiver recall of engagement in WSM activities was inaccurate or that the cutpoint of three components was not appropriate. While WFI-EZ measures have demonstrated to have reliabilty and validity compared to the full WFI measure, we bolstered our analyses by including documentation of team meetings in the administrative record as an additional factor in group assignment and reporting of a series of supplemental analyses reflecting alternative cutpoints on the instrument. Another potential limitation is that although our model included an extensive set of survey and administrative covariates for propensity score estimates coupled with K-DID analyses, additional unmeasured covariates could contribute to group-based selection effects. Use of DID methods may also account for unmeasured covariates, and similarity to other randomized trials helps to minimize this concern. Third, child and caregiver outcomes (with the exception of safety outcomes) were derived from caregiver report and may be subject to biased reporting. Concerns are tempered by the use of valid and reliable measures, as well as the fact that these biases would likely be present for both WSM and non-WSM conditions, so the K-DID modeling approach would still provide a means of detecting cross-group differences in trajectories. Finally, families in our study were drawn from a single state CPS system—program results may not generalize to other states or to those that implement the WSM with a higher degree of fidelity.

Implications

A version of wraparound supports currently exists in every state (Coldiron et al., 2019), yet the evidence for its effectiveness is limited among CPS-involved households. Moreover, research has consistently pointed to significant concerns related to program fidelity. Additional efforts must be undertaken, using rigorous experimental and quasi-experimental designs, to demonstrate effects of WSM in the context of supports to maintain high fidelity program implementation among CPS-involved families. It is imperative that state systems provide ongoing training, consultation, and supervision to facilitators to ensure maintenance of program implementation quality as it is taken to scale. Consideration of systems-level administrative structures and funding necessary to support implementation, such as use of formal care management entities rather than community provider networks, may also be necessary to enhance system-level implementation (Olson et al., 2021).

Implementation also must focus on strategies that engage CPS-referred families in the wraparound process more effectively. One way to promote family commitment to WSM is for the child and family teams to include members beyond professional supports (Walker & Schutte, 2005; Wright et al., 2006). Another may be to ensure that formal community services and supports that are leveraged through WSM are responsive to the individualized service plan developed with the family’s direct input and involvement. Research suggests that when targeted services match family need, there are reductions in future service use (Simon & Brooks, 2019) and further CPS involvement (Simon, 2020). An emphasis on adherence to wraparound principals, bolstered through intensive system-level supports to maintain that adherence and coupled with efforts to better engage families in the treatment model to enact individualized formal and natural supports, should increase the likelihood of system-wide WSM implementations that produce more optimal family outcomes.