Sage Journals: Discover world-class research

Abstract

The decision to substantiate a report of child maltreatment represents a key decision point in the child welfare service decision-making continuum. This decision has various potential implications for children and their families, which may include more intensive child welfare involvement or the cessation of services. The substantiation decision is determined by whether there is enough evidence to suggest that maltreatment or the risk of maltreatment has occurred. To date, there has been minimal exploration of whether child welfare worker characteristics might influence this critical decision point. The Decision-Making Ecology would suggest that indeed, worker characteristics play a role in how they carry out their role. Given the importance of this decision point, this study uses secondary data to examine whether worker characteristics, such as education level and type, ethnoracial identity, caseload, and experience, predict substantiation in the Canadian child welfare context. Furthermore, this study utilizes multilevel modeling, a theoretically important and unique method of analyzing organizational data that considers differences in decisions among child welfare workers. The final model included 4,327 children and 567 workers from across Canada. Several case level factors (e.g., child age and functioning, caregiver risk factors) predicted the substantiation decision. Furthermore, and most importantly for this study, worker characteristics significantly predicted their substantiation decision. Workers with fewer years of experience, those in an Ongoing Services role, and with a lower caseload substantiated significantly more often than those with more work experience, in another role, and with higher caseloads. Lastly, caseload and years of experience, and training and caseload both interacted to predict the substantiation decision. Implications for policy and practice and future research areas are discussed.

Keywords

child welfare workers decision-making substantiation multilevel modeling child maltreatment

Introduction

To date, there has been minimal exploration of whether child welfare worker characteristics might influence this critical decision point. As they carry out a dual mandate of promoting the well-being and safety of children, child welfare workers play an important role, as they are the link between the system and families who have come to the attention of child welfare organizations. Child welfare workers are required to provide services offered to families that includes, in part, by making decisions that determine families’ path through the system. These decisions fall on the decision-making continuum, which directs families through the system and include the decision to investigate, substantiate, transfer to Ongoing Services, seek a court supervision order, place a child in out-of-home care, or return a child to their caregiver of origin, for example. Thus, the number of families served by child welfare systems decrease dramatically between the beginning and end of the decision-making continuum (Baumann et al., 1997).

Substantiation of maltreatment allegations is an important decision, as many child welfare systems require substantiated abuse or neglect reports in order for the family to receive continued services. Therefore, this decision may impact whether a family has access to community services, further child welfare services, or results in court involvement. Furthermore, the Decision-Making Ecology (DME; Baumann et al., 1997) suggests that worker characteristics play a role in the decisions they make throughout child welfare services. While there is a U.S. and Canadian body of literature that has examined the clinical predictors of substantiation (e.g., Font & Maguire-Jack, 2019; Fallon et al., 2020; Kohl et al., 2009; Trocmé et al., 2010), we know little about whether worker characteristics predict this key decision.

In Canadian child welfare systems, the rates of substantiation have varied. Findings from the most recent Canadian Incidence Study of Reported Child Abuse and Neglect (CIS-2008; Trocmé et al., 2010) suggest that 36% of child maltreatment (rate 14.19 per 1,000 children) and 5% of risk of future maltreatment (rate 2 per 1,000 children) investigations were substantiated (Trocmé et al., 2010). Over the years, incidence results indicate that substantiation rates were highest in 2003 (rate 18.67 per 1,000 children; 49% of investigations), and in 1998 they were comparable to those in 2008, 9.21 per 1,000 children (43%; Trocmé et al., 2010).

Canadian research suggests that the clinical predictors of substantiated and unfounded child abuse and neglect investigations differ (Trocmé et al., 2009). While the substantiation rates of maltreatment types vary, factors that predict substantiation by maltreatment type do not differ considerably (Trocmé et al., 2009). These findings are contrary to those in the U.S. context (Kohl et al, 2009). Kohl et al.’s (2009) national study that examined the differences in recidivism between substantiated and non-substantiated child maltreatment allegations indicates that their risk of recidivism was similar. Authors suggest more meaningful methods of identifying risk of future maltreatment. Especially, given the differences between the U.S. and Canadian context, it is important to understand which clinical and worker factors predict the substantiation decision in Canada. Indeed, empirical evidence that examines whether worker factors predict child welfare decisions will begin to close the knowledge gap and inform our ultimate goal of evidence informed policy, and training and recruitment strategies.

Exploring whether worker factors predict the substantiation decision is an important part of developing the overall policy and practice knowledge base. Indeed, understanding whether workers make decisions differently may impact how they are trained to provide services and the policies developed to recruit workers. Given the importance of this decision point, this study will focus on whether worker characteristics, such as worker education level and type, ethnoracial identity, caseload, and experience, predict substantiation in the Canadian context. Furthermore, this study utilizes multilevel modeling, a theoretically important and unique method of analyzing organizational data that takes into account differences in decisions among child welfare workers.

This study addresses two research questions: (1) What worker characteristics predict the substantiation decision after a child maltreatment investigation? (2) Does the substantiation decision differ among child welfare workers after their child maltreatment investigation?

Literature Review

Clinical Factors

Several studies that have tested the relationship between child age and substantiation have found that younger children are substantiated significantly more often than older children (e.g., Dettlaff et al., 2011; Font & Maguire-Jack, 2019; King et al., 2013; Maguire-Jack et al., 2019; Trocmé et al., 2005); some studies, however, have found no child age effect (King et al., 2003; Lefebvre et al., 2017; McDaniel, 2006).

Caregiver factors such as those living with functioning difficulties, including physical health, drug use, and mental health, housing issues, and with low income significantly increases the likelihood of substantiation (Dettlaff et al., 2011; Font & Maguire-Jack, 2015; Freisthler et al., 2017; Jedwab et al., 2015; Lefebvre et al., 2017). Recent studies in the United States (Font & Maguire-Jack, 2019; Graham et al., 2018; Janczewski & Mersky, 2016) and Canada (Ma et al., 2019; Maguire-Jack et al., 2019; Sinha et al., 2013; Trocmé et al., 2009) have examined the impact of families’ ethnoracial status and suggest that racialized families are more likely to be substantiated when compared to white or non-Indigenous families.

The referral source may play a role in whether maltreatment allegations are substantiated, such that concerns reported by professional sources are typically substantiated more often than non-professional sources (Dettlaff et al., 2011; King et al., 2013). In a Canadian study, authors found that maltreatment concerns reported by educators were substantiated significantly less than other professionals (King & Scott, 2014). Even controlling for the higher degree of child risk factors reported by educators, their maltreatment concerns were still substantiated significantly less often (King & Scott, 2014). Recently, Font and Maguire-Jack (2019) also found that educator concerns were substantiated less often, but more often when multiple maltreatment types were identified. In the same study, law enforcement referrals predicted higher rates of substantiation than other referral sources (Font & Maguire-Jack, 2019).

Worker Factors

Worker characteristics are less frequently examined than clinical-level factors but have become more commonly explored in recent years and there is some indication that they may play a role in the substantiation decision. In Jent et al.’s (2011) case vignette study, U.S. child welfare workers were asked to assess and make a substantiation decision on different maltreatment case scenarios. Findings suggested that child welfare worker race and education toward corporal punishment were significantly associated with their decisions differently, depending on the severity of the maltreatment in the vignettes. These results suggest that worker factors may impact their decisions; however, the association of factors may present themselves differently depending on the nature of the maltreatment allegations (Jent et al., 2011). Conversely, Font and Maguire-Jack (2015) found that child welfare workers substantiated cases differently based on level of education.

Child welfare worker factors such as child welfare experience and caseload size have also been found to predict the substantiation decision when investigating maltreatment in caregivers with cognitive impairments (McConnell et al., 2011). Similarly, child welfare organizations who report higher workloads have lower substantiation rates (Font & Maguire-Jack, 2019). While these studies have been important for launching the knowledge base, the current study utilizes a strong methodological approach to assessing the link between workers and their substantiation decision. Only one other study (Font & Maguire-Jack, 2015) could be located to use such a methodology. As noted, the decision to substantiate may be applied differently depending on geography, much more exploration into this area, therefore, is required. To build the evidence base, we need a stronger understanding of which factors contribute to the substantiation decision. Thus, this paper aims to begin to close this knowledge gap by focusing on assessing which clinical and worker factors are associated with substantiation, a key child welfare decision.

Method

This study uses secondary data analyses of the most recent and only incidence study conducted in Canada, The Canadian Incidence Study of Reported Child Abuse and Neglect (CIS-2008; Trocmé et al., 2010). The CIS is typically administered on a 5-year cycle (CIS-1993, 1998, 2003, and 2008) and aims to produce national incidence estimates of reporte child abuse and neglect (Trocmé et al., 2010). The current study utilized multilevel modeling and included data at the child and worker levels. This study addresses two research questions: What worker characteristics predict the substantiation decision after a c maltreatment investigation? Does the substantiation decision differ among child welfare workers after their child maltreatment investigation? While there is a dearth of evidence in this area, it is hypothesized that worker characteristics indeed predict the substantiation decision.

Sample

The Canadian Incidence Study of Reported Child Abuse and Neglect (CIS)-2008 was sampled in three stages: (1) a representative sample of child welfare organizations was chosen (N = 112); (2) cases were randomly sampled throughout a 3-month data collection period (October 1 to December 31 N = 9,933); and (3) child investigations (children 15 and under) were identified (N = 15,980; Trocmé et al., 2010). The final sample for multilevel modeling only included investigations that were deemed substantiated or unsubstantiated and included 6,547 child investigations and 763 child welfare workers. Child welfare workers from Quebec did not participate in the comparative worker questionnaire during this CIS cycle, therefore, were not included in the analyses. The CIS has been tested for validity and reliability; for further information, please see the Canadian Incidence Study of Reported Abuse and Neglect Major Findings (Trocmé et al., 2010).

Data Collection Instruments

Analyses in this study include data that were collected via two different assessment forms. The first was the maltreatment assessment form, which was completed by the assigned child welfare worker who provided information about the case (e.g., maltreatment allegation type, investigation outcome), household (e.g., household runs out of money regularly, hazards in the home), caregiver(s) (e.g., age, functioning), and child (e.g., gender, age, functioning). The second form was the worker questionnaire, completed by the assigned child welfare worker and provided information about their characteristics (e.g., education, caseload). Data collected from these instruments were integrated and constructed a two-level data set. Clinical-level data (level one) included any factors used to describe the child (e.g., age), caregiver (e.g., risk factors), or case/investigation (e.g., referral source). Worker-level data (level two) included worker characteristics (e.g., caseload, education).

Measures

Variables in this study were chosen based on a combination of availability and what is known and what is not known in the body of literature about predictors of the substantiation decision. Clinical predictors (level one) of substantiation were chosen based primarily on the findings of the literature review, which includes international studies, but are found primarily in the United States. Conversely, there is a dearth of evidence regarding worker characteristics (level two predictors). Therefore, worker variables were identified based on the limited number of studies previously conducted and available data from the CIS.

Dependent variable

Substantiation decision (0 = allegations not substantiated, 1 = allegations substantiated).

Independent variables

See Table 1 for full list of independent variables.

Table 1.

Independent Variables.

Variable	Definition	Value
Case characteristics
Child age	Age dichotomized into 5 years or younger and older than 5.	0: 5 Years or younger
Child age	Age dichotomized into 5 years or younger and older than 5.	1: Older than five
Child gender	Gender	0: Female
Child gender	Gender	1: Male
Child functioning	Functioning issues either confirmed or suspected: (e.g., depression/anxiety/withdrawal; suicidal thoughts; self-harming behavior; ADD/ADHD; attachment issues; aggression)	0: No functioning issues
Child functioning		1: One or more functioning issue
Caregiver age	Age of caregiver in years. Variable divided at median.	0: Younger than 21
Caregiver age	Age of caregiver in years. Variable divided at median.	1: 21 or older
Caregiver ethnoracial identity	Caregiver ethnoracial identity	0: White
Caregiver ethnoracial identity	Caregiver ethnoracial identity	1: Not white
Caregiver relationship status	Single caregiver or more than one caregiver in the home.	0: More than one caregiver
Caregiver relationship status	Single caregiver or more than one caregiver in the home.	1: Single caregiver
Caregiver risk factors	Caregiver risk factors (e.g., drug or alcohol use, mental health issues, cognitive functioning issue).	0: None noted
Caregiver risk factors		1: One or more issue noted
Caregiver social support	Primary caregiver has few social supports.	0: Not noted
Caregiver social support	Primary caregiver has few social supports.	1: Few supports noted
Caregiver cooperation	Caregiver cooperation with the child welfare investigation.	0: Not cooperative
Caregiver cooperation	Caregiver cooperation with the child welfare investigation.	1: Cooperative
Household finances	Household regularly runs out of money.	0: Not noted
Household finances	Household regularly runs out of money.	1: Noted
Household moves	Number of times household has moved in the past year.	0: None
Household moves	Number of times household has moved in the past year.	1: One or more
Previous involvement	Caregiver previous involvement with child welfare (as a parent).	0: No involvement
Previous involvement		1: Previous involvement
Referral source	Professional source (e.g., teachers) or non-professional source (e.g., neighbor).	0: Not professional source
Referral source		1: Professional source
Worker characteristics
Education field	Worker field of completed education.	0: Other than social work
Education field	Worker field of completed education.	1: Social work
Education level	Worker level of completed education.	0: Bachelor
Education level	Worker level of completed education.	1: Master
Experience	Years of child welfare experience, recoded into three dichotomous variables, based on variable distribution.	0: ≤2 years
		1: >2 years
		0: ≤4 years
		1: >4 years
		0: ≤6 years
		1: >6 years
Age	Worker age. Variable divided at median.	0: 34 or younger
Age	Worker age. Variable divided at median.	1: Older than 34
Ethnoracial identity	Worker ethnoracial identity. “Not white” category is a combination of ethnoracial identities (e.g., Black, Latin American, Aboriginal).	0: White
Ethnoracial identity		1: Not white
Position	Position recoded into three dichotomous variables: Intake, Ongoing, Generalist (e.g., Intake and Ongoing).	0: Not Intake
		1: Intake
		0: Not ongoing
		1: Ongoing
		0: Not generalist
		1: Generalist
Caseload	Caseload at time of data collection. Recoded into three dichotomous variables based on variable distribution.	0: 0–9 cases
		1: >9 cases
		0: 0–13 cases
		1: >13 cases
		0: 0–18 cases
		1: >18 cases
Training	Number of child protection trainings attended in career. Recoded into three dichotomous variables based on variable distribution.	0: 0–6 Training sessions
		1: >6 Training sessions
		0: 0–10 Training sessions
		1: >10 Training sessions
		0: 0–13 Training sessions
		1: >13 Training sessions

Note. ADD = attention deficit hyperactivity disorder; ADHD = attention deficit hyperactivity disorder.

Statistical Analyses Plan

Multilevel modeling is appropriate for data such as those collected in the CIS-2008. By nature, there is dependence in these data, as it is assumed that not all workers function similarly. To answer the research questions, a series of univariate, bivariate, and multilevel tests were run. Initially, empty (no independent variables) one- and two-level models were used to assess dependency in the data and to determine whether a two-level model would be appropriate. After adding clinical-level variables to a single-level logistic model, a random intercept (worker level) was added, resulting in a two-level logistic regression. Level two fixed effects were then added to the model but removed if they did not significantly predict the substantiation decision. Interaction effects at the worker level (e.g., training x caseload) were tested in the two-level models. There is little empirical evidence in the body of literature to guide variable interactions. Rather, interactions were chosen based on hypothetical relationships between worker characteristics and which characteristics may work together to predict substantiation. Log odds, p value, and odds ratios (ORs) were assessed for all fixed effects. Level two variables, even when they did not significantly predict the substantiation decision, remained in the final model if they contributed to model fit. Commonality among worker’s decisions was tested using the intraclass correlation coefficient (ICC). SPSS version 26 was used for univariate and bivariate tests and SAS version 9.4 was used for all multilevel modeling.

Results

Demographics

Results indicate that the majority of allegations were substantiated (n = 4735, 56%). The majority of workers had a social work degree versus another field (n = 794, 67%) and Bachelor degree versus Master degree (any field; n = 742, 86%). Years of experience was almost equally distributed, such that 33% (n = 301) had 2 years or fewer, 33% (n = 288) had between 2 and 6 years, and 33% (n = 298) had more than 6 years. Fifty-four percent (n = 468) of workers were under 34 years and 82% (n = 734) identified as white. Worker roles and organizations are structured differently across the country, there were, however, 44% (n = 401) in an investigative only role, 11% (n = 105) in an Ongoing Service only role, and 37% (n = 341) in a generalist position (e.g., investigation and Ongoing Services). Twenty-one percent (n = 187) of workers had up to nine cases at the time of data collection, 49% (n = 433) had between 10 and 18 cases, and 29% (n = 263) had more than 18 cases. Lastly, 26% (n = 239) of workers attended up to six training sessions in their career, 55% (n = 508) attended between 7 and 13 training sessions, and 18% (n = 169) attended more than 13 training sessions.

Bivariate Analyses

Clinical level

Several clinical-level variables were significantly correlated with the substantiation decision. However, only two child-level variables, functioning concerns and age, were significantly associated with the substantiation decision (X² [1, N = 9109] = 138.28, p < .000; X² [1, N = 9109] = 41.23, p < .000, respectively). Male and female children were not substantiated significantly differently from one another (X² [1, N = 9109] = .084 p = .772). All but one caregiver factor (relationship status; X² [1, N = 9109] = .384 p = .536) were associated with the substantiation decision. Caregiver age (X² [1, N = 9064] = 12.96, p < .000), ethnoracial identity (X² [1, N = 9109] = 51.17, p < .000), functioning (X² [1, N = 9109] = 1116.81, p < .000), social supports (X² [1, N = 9109] = 410.03, p < .000), cooperation (X² [1, N = 8996] = 134.17, p < .000), and previous involvement with child welfare (X² [1, N = 9005] = 18.70, p < .000) were all significantly associated with substantiation. Number of household moves in the last year (X² [1, N = 7004] = 26.38, p < .000), households that regularly ran out of money (X² [1, N = 7462] = 287.49, p < .000), and referral source (professional, not professional; X² [1, N = 9109] = 193.05, p < .000) were also significantly associated with the substantiation decision.

Worker level

Bivariately, some worker-level variables were significantly associated with the substantiation decision; however, this was less frequent than clinical factors. In terms of worker position, Ongoing Services worker was significantly associated with substantiation (X² [1, N = 845] = 6.37, p < .01). However, the Intake position versus other positions (X² [1, N = 845] = .12, p = .72) and Generalist versus other positions (X² [1, N = 845] = 2.24, p = .13) were not significantly associated. Individual factors such as education (social work and other than social work (X² [1, N = 738] = 51.17, p = .44), Bachelor and Master levels (X² [1, N = 692] = .31, p = .57), age (34 or younger and older than 34; X² [1, N = 811] = .23, p = 631), and ethnoracial identity (white and not white; X² [1, N = 833] = .00, p = .96) were not significantly associated with substantiation. Importantly, 82% of workers identified as white, the disproportionate categories (white, not white) may impact results. No experience levels (≤2 years and more than 2 years, X² [1, N = 825] = 1.31, p = .25; ≤4 years and more than 4 years, X² [1, N = 825] = .001, p = .98; and ≤6 years and more than 6 years, X² [1, N = 825] = 1.92, p = .16) were significantly associated with substantiation at the bivariate level. Organizational environmental factors such as caseload (≤9 and more than 9, X² [1, N = 822] = 1.16, p = .28; ≤13 and more than 13, X² [1, N = 822] = .81, p = .36; ≤18 and more than 18, X² [1, N = 822] = 1.67, p = .19) and training (≤6 and more than 6, X² [1, N = 854] = .10, p = .74; ≤10 and more than 10, X² [1, N = 854] = 2.30, p = .12; ≤13 and more than 13, X² [1, N = 854] = .44, p = .50) were not significantly associated with the substantiation decision.

Multilevel Analyses

The final multilevel model was built by conducting several one- and two-level analyses. Examination of a two-level model with no level one or level two predictors (empty model—no independent variables) suggested that a two-level logistic regression was appropriate (X² [N = 8429] = 6.33, p < .0001, −2 log likelihood 36963.10), confirming there is dependence in the data. The final model included 4,327 children (investigations; level one) and 567 workers (level two).

Model fit

The final model (X² [N = 4327] = 263.81, p < .0001) included significant and non-significant level one and level two fixed effects and two significant interactions. This model was chosen because it had the best fit statistics analyzing the Akaike information criterion, Bayesian information criterion, and Log-likelihood. The ICC suggests there is 25% variance among child welfare worker’s substantiation decision.

Clinical factors

Results indicate that several level one clinical factors predict workers’ substantiation decision. Children 5 years or younger were substantiated more frequently than those older than 5 years (OR = 0.75, 95% C.I. [.62–.92]). Children with more than one functioning issue were more than one and a half times more likely to be substantiated than children with no identified functioning concern (OR = 1.59, 95% C.I. [1.34–1.88]). Caregiver risk factor was the strongest predictor of substantiation, considering level one and level two fixed effects. Caregivers with one or more identified risk factor (e.g., substance use, mental health concern) were more than five times as likely to be substantiated in comparison to caregivers with no identified risk factors (OR = 5.29, 95% C.I. [4.42–6.32]). Caregivers with a lack of social supports were substantiated one and a half times more often than caregivers with adequate social supports (OR = 1.65, 95% C.I. [1.35–2.01]). Households that regularly ran out of money were almost twice as likely to be substantiated as households without this identified concern (OR = 1.81, 95% C.I. [1.43–2.49]). Those previously involved with the child welfare system as a parent were slightly less likely to be substantiated than caregivers with no previous involvement (OR = 0.75, 95% C.I. [.63–.89]). Caregivers identified as not cooperating with the investigation were less likely to be substantiated than caregivers who were identified as cooperative (OR = 0.54, 95% C.I. [.36–.82]). Lastly, concerns identified by a professional referral source (e.g., teacher, police) were more than twice as likely to be substantiated compared to reports from non-professional referral sources (OR = 2.31, 95% C.I. [1.93–2.78]).

Worker factors

The majority of worker-level factors did not significantly predict the substantiation decision. There were, however, three single independent level two variables and two level two interactions that did predict workers’ substantiation decision. See Table 2 for full multilevel results. Workers with two or fewer years of experience substantiated maltreatment allegations significantly more frequently than workers with more than 2 years of experience (OR = 1.25, 95% C.I. [1.10–1.51]). Workers in an Ongoing Family Services role were three times more likely to substantiate maltreatment concerns when compared to workers in any other role (OR = 3.22, 95% C.I. [1.42–7.29]). Caseload size significantly predicted the substantiation decision such that workers with a caseload of 13 or fewer substantiated significantly more often than those with more than 13 active cases (OR = 0.59, 95% C.I. [.42–.83]).

Table 2.

Final Multilevel Model.

	Coefficient	Std. Error	p
Level One
Child age	−0.27	0.10	<.005
Child gender	−0.03	0.08	.689
Child functioning	0.46	0.08	<.0001
Caregiver age	0.16	0.22	.457
Caregiver relationship	−0.16	0.08	.060
Caregiver ethnoracial	0.15	0.10	.135
Caregiver risk factors	1.66	0.09	<.0001
Social supports	0.50	0.10	<.0001
Household finances	0.63	0.14	<.0001
Previous involvement	−0.28	0.08	<.005
Caregiver cooperation	−0.60	0.20	<.005
Household moves	−0.14	0.09	.119
Referral source	0.84	0.09	<.0001
Level two
Education field	−0.17	0.13	.192
Education level	−0.09	0.18	.611
Experience (≤2 or >2 years)	−0.57	0.28	<.05
Experience (≤4 or >4 years)	0.12	0.21	.581
Experience (≤6 or >6 years)	−0.24	0.19	.189
Age	0.06	0.14	.669
Ethnoracial	0.13	0.15	.409
Position (intake or other than intake)	0.58	0.31	.064
Position (not ongoing or ongoing)	1.17	0.41	<.001
Position (generalist or other than generalist)	0.66	0.34	.050
Caseload (≤9 or >9)	0.09	0.29	.742
Caseload (≤13 or >13)	−0.51	0.17	<.001
Caseload (≤18 or >18)	0.25	0.17	.153
Training (≤6 or >6)	0.11	0.30	.705
Training (≤10 or >10)	0.04	0.16	.776
Training (≤13 or >13)	0.25	0.16	.133
Training (≤6 more than 6) × caseload (≤9 more than 9)	0.59	0.20	<.001
Experience (≤2 more than 2) × caseload (≤9 more than 9)	−0.57	0.28	<.05

Interactions

Results suggest that caseload size and years of experience interact to predict workers’ substantiation decision. Workers with less than 2 years of experience have a similar likelihood of substantiating allegations of maltreatment no matter their caseload size. However, workers with more than 2 years of experience and a caseload of 10 or more substantiate more frequently than workers with the same experience and a caseload between zero and nine. See Figure 1 for effect plot. Results testing the interaction between training and caseload suggest that training and caseload, taken together, indeed impact the rate of substantiation. Workers with lower rates of training attendance substantiate at similar rates, whether they have a caseload of zero to nine or 10 or more. However, workers who attended more than six training sessions in their tenure substantiate more frequently when they have a lower caseload and less frequently when they have a caseload of 10 or more. See Figure 2 for effect plot.

Figure 1.

Worker caseload size and experience interaction.

Figure 2.

Worker caseload size and training interaction.

Discussion and Implications

There is a gap in the literature when considering whether child welfare worker characteristics predict the substantiation decision. This decision point is important in the continuum of service, as the substantiation of maltreatment concerns has the potential to direct families through the child welfare system, including eligibility for additional community or child welfare services (Baumann et al., 1997). Furthermore, while families being served by child welfare systems should expect service based on familial context, the DME (Baumann et al., 1997) suggests, and has found (e.g., Dettlaff et al., 2011; Lwim, in press; Maguire-Jack et al., 2019), that child welfare worker characteristics, among other factors outside of the clinical case, have the potential to impact their decisions. This study aimed to answer the questions: What worker characteristics predict the substantiation decision after a child maltreatment investigation? Does the substantiation decision differ among child welfare workers after their child maltreatment investigation?

Clinical Factors

Findings, regarding clinical factors (child, caregiver, case), in this study are consistent with the wider body of literature. Several studies have found that younger children are more likely to be substantiated significantly more often than older children (Dettlaff et al., 2011; Font & Maguire-Jack, 2019; King et al., 2013; Maguire-Jack et al., 2019; Trocmé et al., 2005). Similarly, results here suggest that children younger than 5 years were substantiated significantly more often than those 6 years and older (Filippelli et al., 2020; Goulet et al., 2018). Indeed, the vulnerability of young children versus teenagers, for example, may play a role in the rate of substantiation. Findings from this study suggest that children with at least one functioning concern (e.g., attention deficit hyperactivity disorder) were significantly more likely to be substantiated than children with no identified concerns. These results are consistent with the wider body of knowledge (Font & Maguire-Jack, 2015; McDaniel, 2006), which indicates that child functioning plays a role in the level of evidence workers require to substantiate allegations.

Caregiver ethnoracial identification did not predict higher or lower levels of substantiation. The literature in this area is mixed, as some studies have found that ethnoracial identity predicts substantiation (e.g., Fallon et al., 2013; Font & Maguire-Jack, 2019; Graham et al., 2018; Ma et al., 2019; Sinha et al., 2013); however, some have not (e.g., Filippelli et al., 2017; McDaniel, 2006). Indeed, this is a complex issue that must be investigated further, as some research suggests that poverty and race intersect along the service continuum (e.g., King et al., 2018), which may occur not only at the individual, but also at the community level (Wulczyn et al., 2013). Primary caregiver risk factors, such as more than one identified concern (e.g., alcohol or drug use, mental health issue), poor social supports, non-cooperation with the investigation, running out of money regularly, and previous child welfare involvement predicted higher rates of substantiation. These findings were anticipated and correspond to the wider knowledge base (e.g., Dettlaff et al., 2011; Font & Maguire-Jack, 2015; Freisthler et al., 2017; Jedwab et al., 2015; Lefebvre et al., 2017). The greater number of functioning or parenting concerns would inevitably add to the evidence used to substantiate abuse or neglect allegations in an investigation. Lastly, a professional referral source (e.g., teacher, police) predicted the allegations being substantiated significantly more often than a non-professional referral source (e.g., neighbor, anonymous). These findings correspond with the wider literature (Dettlaff et al., 2011; Font & Maguire-Jack, 2019; King & Scott, 2014; King et al., 2013) and suggest that reports of maltreatment concern from professional sources come with additional evidence or authority that increases the likelihood of substantiation.

Expectedly, many clinical-level characteristics predicted the substantiation decision. While these findings are consistent with the wider body of literature, this is the only multilevel study examining the substantiation decision that has been conducted in Canada. Therefore, future research should continue to assess whether clinical factors predict the substantiation and whether these factors may vary by organizational and community factors.

Worker Factors

The majority of worker characteristics did not predict the substantiation decision. However, years of experience, position, caseload, training caseload interaction, and experience caseload interaction did predict workers’ substantiation decision. Workers with fewer years of experience substantiated significantly more frequently than workers with more than 2 years of experience. There is a lack of research in this area; however, findings from this study correspond with a study that found more experienced workers were more likely to unsubstantiate allegations (Child Welfare Information Gateway, 2003).

There is a growing body of literature that explores worker experience and other factors related to practice; number of years of experience did not predict workers’ placement decision (Graham et al., 2015) or the transfer to Ongoing Services decision (Lwin et al., 2018), for example. There are conflicting findings when considering whether experience predicts the perception of risk (Lwin et al., in press; Sullivan et al., 2008) these studies, however, utilized different methodologies, which may account for the difference in findings. In a prospective study, authors found no experience effect for their out-of-home placement assessment; however, less experienced workers, were significantly more likely to act on their original assessment and place a child in out-of-home care than more experienced workers (Davidson-Arad et al., 2003). These findings illustrate that experience may not directly impact assessment and decision-making, rather, it may impact the decision-making process. While this study cannot identify whether the substantiation decision was appropriate, it is hypothesized that less experienced workers may have been more cautious, thus, substantiating allegations with less evidence and, thus, more often. Indeed, further research is needed to assess whether experience plays a role in service and the potential indirect nature of this relationship.

Results from this study suggest that those in an Ongoing Family Services role were more than three times as likely to substantiate maltreatment concerns compared to workers in any other role. There is little research to situate these findings. However, we hypothesize that Ongoing Family Services workers investigate allegations of maltreatment less often than Intake (investigative only) workers, and therefore, may be more cautious and require less evidence when substantiating allegations. Similar to the underpinnings of why less experienced workers may substantiate more often, perhaps Ongoing Family Services workers might be apprehensive, resulting in a more cautious approach. Furthermore, it is also hypothesized that Ongoing Family Services workers, given the nature of their role may already know the family they are investigating, therefore have a different substantiation threshold and decipher the credibility of the evidence differently from a family they do not know as well.

Caseload was the final individual worker characteristics that predicted substantiation. Workers with a caseload of 13 or fewer substantiated significantly more often than those with more than 13 active cases. These findings are consistent with a recent U.S. study (Font and Maguire-Jack, 2019), who found that higher workloads were significantly associated with lower substantiation rates. Similarly, in a synthesis of the literature, authors found that workers with higher workloads tended to substantiate allegations less frequently (Child Welfare Information Gateway, 2003). We are able to situate the findings from this study within the wider body of literature. While the methods in this body of literature differ, it is hypothesized that workers with a higher caseload are unable to conduct an in-depth investigation, resulting in less evidence to substantiate allegations and therefore, lower rates of substantiation. It is, however, also possible that these cases have an inherent (unknown) similarity that makes it less likely to substantiate.

Lastly, the interaction between experience and caseload and training and caseload significantly predicted substantiation. Workers with 2 years or fewer with a caseload between zero and nine or more than nine were consistent in the predicted probability of the substantiation decision. However, workers with more than 2 years of experience with a caseload between zero and nine were less likely to substantiate than workers with more than 2 years of experience with a caseload of more than nine (see Figure 1). These findings are contradictory to the main effect of caseload—the lower the caseload, the less likely one would substantiate. Nevertheless, it is hypothesized that workers with more experience are more likely to be assigned complex cases, which may be more difficult to substantiate than a straightforward maltreatment or neglect concern. Workers who attended between zero and six training sessions, no matter their caseload (0–9 cases vs. more than 9 cases), had similar probability of substantiating maltreatment allegations. However, workers with more than six training sessions substantiated more often when they had a caseload between zero and nine, and substantiated allegations less often when their caseload was 10 or higher (see Figure 2). There is no training main effect, conversely, there is a caseload main effect. Indeed, workers with more experience will have attended more training sessions throughout their tenure. The association consequently may be driven by worker caseload—the number of cases that a worker is investigating predicts the probability of substantiation and this threshold varies depending on a number of factors, including years of experience and amount of training.

These key findings should be used as a foundation for future research. In addition to the noted worker characteristics, other factors that should be assessed for a relationship with the substantiation decision include factors such as worker attitude towards the child welfare system, for example. Indeed, it is important to understand whether worker characteristics, such as those assessed here, directly impact decision-making or are a factor in the decision-making process and indirectly influence the outcome. While the findings are of consequence, they should be utilized cautiously, as there is some evidence to suggest that worker characteristics play a role in child welfare services, but that characteristics may differ depending on the decision and service (e.g., Filippelli et al., 2020; Font & Maguire-Jack, 2015; Lwin et al., in press). Certainly, to promote strong recruitment and onboarding strategies, and professional development, a much more fulsome body of knowledge is required. Nonetheless, in child welfare practice, it should be noted that there is the potential for workers to make decisions differently, based on personal characteristics. A strong foundation for practice, including clinical supervision, would promote a consistent decision-making process and services. Further examination of worker characteristics that predict their likelihood of substantiation is required, as this is a key decision point in the continuum of service and understanding differences among worker characteristics are important.

Limitations

While this study aims to alleviate the knowledge gap, there are limitations. The data analyzed in this study are based only on the first 60 days of the investigation. This study cannot claim that the substantiation decision, whether substantiated or not, is appropriate. Rather, the goal of the study is to assess whether child welfare worker characteristics predict this decision, one way or the other. The data collection tools were completed by the assigned case worker and could not be independently verified. While this may be a limitation, it is a customary method of data collection in incidence studies (e.g., Font & Maguire-Jack, 2019). This study utilizes cross-sectional analyses, which provides only a snapshot in time and cannot suggest causation. Dichotomization of continuous variables (e.g., years of experience, child age, worker age) may have resulted in decreased statistical power. Lastly, the ICC in a logistic regression, should be interpreted with caution, as the probability, not frequency, of the dependent variable is calculated.

Conclusion

This study has made a marked contribution to the literature and is the first multilevel study examining the substantiation decision in Canada. The focus of this study, assessing the role of worker characteristics to the substantiation decision, is an important piece of information required to promote evidence-informed organizational and system policy development. Workers are the link between the system and families, a stronger understanding of their role in service, therefore, is required. The substantiation decision, at the forefront of the decision-making continuum, has the potential to impact subsequent service and families’ experience with the child welfare system and access to community and child welfare services. Importantly, findings suggest that worker characteristics, such as years of experience, caseload size, position, and level of training, play a role in the substantiation decision. These characteristics play a role differently in the substantiation decision, as the interaction between variables suggests a more complex relationship between factors than what may have been previously understood. Given the dearth in literature, to advance effective policies, the field requires a more robust knowledge base that weighs the role of worker characteristics in the practice that they provide.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research and/or authorship of this article.

ORCID iD

Kristen Lwin

Author Biographies

Kristen Lwin is an Assistant Professor at the University of Windsor. Dr. Lwin’s research interests include understanding how worker and organizational factors play a role in child welfare decisions and engagement with Evidence-based Practice in child welfare service delivery.

Barbara Fallon is a full Professor and is the Associate Vice-President, Research at the University of Toronto. Dr. Fallon’s research focuses on reliable, valid national and provincial data to provide an evidence-based understanding of the trajectories of children and families in the child welfare system.

Joanne Filippelli’s research focuses on the distinct vulnerabilities, service needs, and opportunities for supporting young child welfare-involved children and their families. Dr. Filippelli has led Provincial and National research studies involving child welfare involved families in Canada.

Nico Trocme is the Director of the School of Social Work and the Philip Fisher Chair in Social Work at McGill University. Dr. Trocmé’s research focuses on developing a common set of National Outcomes Measures in child welfare and is conducting a research capacity development and knowledge mobilization initiative.

References

Baumann

D. J.

Schwab

Schultz

(1997). The influence of case, organizational and individual factors on the process of decision making. In Kern

Bauman

D. J.

Fluke

(Eds.), WISDOM: The child welfare decision enhancement project (pp. 129–144). Department of Protective and Regulatory Services.

Child Welfare Information Gateway. (2003). Decision-making in unsubstantiated child protective services cases: Synthesis of recent research. U.S. Department of Health and Human Services.

Davidson-Arad

Englechin-Segal

Wozner

Gabriel

(2003). Why social workers do not implement decisions to remove children at risk from home. Child Abuse & Neglect, 27(6), 687–697.

Dettlaff

Rivaux

Baumann

Fluke

Rycraft

James

(2011). Disentangling substantiation: The influence of race, income, and risk on the substantiation decision in child welfare. Children and Youth Services Review, 33(9), 1630–1637.

Drake

Jonson-Reid

(2000). Substantiation and early decision points in child welfare: A conceptual reconsideration. Child Maltreatment, 5(3), 227–235.

Fallon

Chabot

Fluke

Blackstock

MacLaurin

Tonmyr

(2013). Placement decisions and disparities among Aboriginal children: Further analysis of the Canadian incidence study of reported child abuse and neglect part A: Comparisons of the 1998 and 2003 surveys. Child Abuse & Neglect, 37, 47–60.

Fallon

Filippelli

Lefebvre

Joh-Carnella

Trocmé

Black

MacLaurin

Hélie

Morin

Fluke

King

Esposito

Collin-Vézina

Allan

Houston

Harlick

Bonnie

Budau

Goodman

Stoddart

(2020). Ontario incidence study of reported child abuse and neglect-2018 (OIS-2018). Child Welfare Research Portal.

Filippelli

Fallon

Fuller-Thomson

Trocmé

(2017). Infants investigated by the child welfare system: Exploring a distinct profile of risks, service needs, and referrals for support in Ontario. Brain Sciences, 7(8), 101.

Filippelli

Lwin

Fallon

Trocmé

(2020). Young children and ongoing child welfare services: A multilevel examination of clinical and worker characteristics. Child Maltreatment, 26(2), 195–204.

10.

Font

Maguire-Jack

(2015). Reprint of decision-making in child protective services: Influences at multiple levels of the social ecology. Child Abuse & Neglect, 49, 50–62.

11.

Font

Maguire-Jack

(2019). The organizational context of substantiation in child protective services cases. Journal of Interpersonal Violence, 36(15–16), 1–22.

12.

Freisthler

Kepple

N. J.

Price Wolf

Curry

Gregoire

(2017). Substance use behaviors by parents and the decision to substantiate child physical abuse and neglect by caseworkers. Children and Youth Services Review, 79, 576–583.

13.

Goulet

Hélie

Clément

M.-È

. (2018). Child and family needs profiles among cases substantiated by child protection services. Child Abuse & Neglect, 81, 366–379.

14.

Graham

J. C.

Dettlaff

A. J.

Baumann

D. J.

Fluke

J. D.

(2015). The decision making ecology of placing a child into foster care: A structural equation model. Child Abuse & Neglect, 49, 12–23.

15.

Graham

Lanier

Finno-Velasquez

Johnson-Motoyama

(2018). Substantiated reports of sexual abuse among Latinx children: Multilevel models of national data. Journal of Family Violence, 33, 481–490.

16.

Janczewski

Mersky

(2016). What’s so different about differential response? A multilevel and longitudinal analysis of child neglect investigations. Children and Youth Services Review, 67, 123–132.

17.

Jedwab

Benbenishty

Chen

Glasser

Siegal

Lerner-Geva

(2015). Child protection decisions to substantiate hospital child protection teams’ reports of suspected maltreatment. Child Abuse & Neglect, 40, 132–141.

18.

Jent

Eaton

Knickerbocker

Lambert

W. F.

(2011). Multidisciplinary child protection decision making about physical abuse: Determining substantiation thresholds and biases. Children and Youth Services Review, 33, 1673–1682.

19.

King

Fallon

Boyd

Black

Antwi-Boasiako

O’Connor

(2018). Factors associated with racial differences in child welfare investigative decision-making in Ontario, Canada. Child Abuse & Neglect, 73, 89–105.

20.

King

Lawson

Putnam-Hornstein

(2013). Examining the evidence: Reporter identity allegation type, and sociodemographic characteristics as predictors of maltreatment substantiation. Child Maltreatment, 18(4), 232–244.

21.

King

Scott

(2014). Why are suspected cases of child maltreatment referred by educators so often unsubstantiated? Child Abuse & Neglect, 38(1), 1–10.

22.

King

Trocmé

Thatte

(2003). Substantiation as a multitier process: The results of a NIS-3 analysis. Child Maltreatment, 8(3), 172–182.

23.

Kohl

Jonson-Reid

Drake

(2009). Time to leave substantiation behind. Child Maltreatment, 14(1), 17–26.

24.

Lefebvre

Fallon

Van Wert

Filippelli

(2017). Examining the relationship between economic hardship and child maltreatment using data from the Ontario incidence study of reported child abuse and neglect-2013 (OIS-2013). Behavioral Sciences, 7(6), 1–12.

25.

Lwin

Fluke

Trocmé

Fallon

Mishna

(2018). Ongoing child welfare services: Understanding the relationship of worker and organizational characteristics to service provision. Child Abuse & Neglect, 80, 324–334.

26.

Lwin

Filippelli

Fallon

King

Trocmé

(in press). Risk of future maltreatment: Examining whether worker characteristics predict their perception. Child Maltreatment.

27.

Fallon

Richard

(2019). The overrepresentation of First Nations children and families involved with child welfare: Findings from the Ontario incidence study of reported child abuse and neglect 2013. Child Abuse & Neglect, 90, 52–65.

28.

Maguire-Jack

Font

Dillard

(2019). Child protective services decision-making: The role of children’s race and county factors. American Journal of Orthopsychiatry, 90(1), 48–62.

29.

McConnell

Feldman

Aunos

Prasad

(2011). Child maltreatment investigations involving parents with cognitive impairments in Canada. Child Maltreatment, 16(1), 21–32.

30.

McDaniel

(2006). In the eye of the beholder: The role of reporters in bringing families to the attention of child protective services. Children and Youth Services Review, 28, 306–324.

31.

Sinha

Ellenbogen

Trocmé

(2013). Substantiating neglect of first nations and non-Aboriginal children. Children and Youth Services Review, 35(12), 2080–2090.

32.

Sullivan

Whitehead

P. C.

Leschied

Chiodo

Hurley

(2008). Perception of risk among child protection workers. Children and Youth Services Review, 30(7), 699–704.

33.

Trocmé

Fallon

MacLaurin

Neves

(2005). What is driving increasing child welfare caseloads in Ontario? Analysis of the 1993 and 1998 Ontario incidence studies. Child Welfare, 84(3), 341–362.

34.

Trocmé

Fallon

MacLaurin

Sinha

Black

Fast

Felstiner

C. H.

Hélie

Turcotte

Weightman

Douglas

Holroyd

(2010). Canadian incidence study of reported child abuse and neglect, 2008: Major findings. Public Health Agency of Canada. https://cwrp.ca/sites/default/files/publications/CIS-2008-rprt-eng.pdf

35.

Trocmé

Knoke

Fallon

MacLaurin

(2009). Differentiating between substantiated, suspected, and unsubstantiated maltreatment in Canada. Child Maltreatment, 14(1), 4–16.

36.

Wulczyn

Gibbons

Snowden

Lery

(2013). Poverty, social disadvantage, and the black/white placement gap. Children & Youth Services Review, 35, 65–74.

A Multilevel Examination of Whether Child Welfare Worker Characteristics Predict the Substantiation Decision in Canada

Abstract

Keywords

Introduction

Literature Review

Clinical Factors

Worker Factors

Method

Sample

Data Collection Instruments

Measures

Dependent variable

Independent variables

Statistical Analyses Plan

Results

Demographics

Bivariate Analyses

Clinical level

Worker level

Multilevel Analyses

Model fit

Clinical factors

Worker factors

Interactions

Discussion and Implications

Clinical Factors

Worker Factors

Limitations

Conclusion

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iD

Author Biographies

References