Abstract
Objective:
This study evaluated the impact of a critical time intervention (CTI) adaptation on health care utilization outcomes among adults experiencing homelessness and mental health needs in a large urban center.
Methods:
Provincial population-based administrative data from Ontario, Canada, were used in a pre–post design for a cohort of 197 individuals who received the intervention between January 2013 and May 2014 and were matched to a cohort of adults experiencing homelessness who did not receive the intervention over the same time period. Changes in health care utilization outcomes in the year pre- and postintervention were evaluated using generalized estimating equations, and post hoc analyses evaluated differences between groups.
Results:
Pre–post analyses revealed statistically significant changes in health care utilization patterns among intervention recipients, including reduced inpatient service use and increased outpatient service use in the year following the intervention compared to the year prior. However, the matched cohort analysis found nonsignificant differences in health service use changes between a subgroup of intervention recipients and their matched counterparts.
Conclusions:
An adapted CTI model was associated with changes in health care utilization among people experiencing homelessness and mental health needs. However, changes were not different from those observed in a matched cohort. Rigorous study designs with adequate samples are needed to examine the effectiveness of CTI and local adaptations in diverse health care contexts.
Introduction
Internationally, rates of homelessness continue to rise 1,2 ; and in Canada, national data show an increase in demand for shelter beds over time, with at least 235,000 Canadians experiencing homelessness in a year. 3 People experiencing homelessness also experience higher rates of mental health and addiction challenges, 4 neurocognitive impairment, 5,6 physical health problems, and premature mortality, 7,8 compared to their housed counterparts. Despite the health disparities, this population generally has limited access to appropriate, high-quality health care and supports, even in settings with universal health insurance. 9 –12 Together, these factors contribute to high rates of hospital service use and costs. 13 –15
Given increasing rates of homelessness and the impact of homelessness on health, 16 there is an urgent need for interventions to improve health outcomes in this population, including avoidable hospital service utilization. 14 For adults experiencing mental illness, care following discharge from hospital for a mental health condition is reportedly the most important factor in reducing reliance on subsequent inpatient care. 17 Adults experiencing mental illness and homelessness, however, are less likely to be referred to and access community-based services upon discharge from hospital, compared to housed individuals. 18,19
Among people experiencing homelessness, strategies to improve continuity of care by offering intensive time-limited services to support the transition from hospital to community-based services have shown promising findings. 20 –25 Critical time intervention (CTI) was designed to support homeless individuals during transitions of care, including the period posthospital discharge in which people experiencing mental illness are at high risk of experiencing first-episode or recurrent homelessness. 26,27 CTIs involve time-limited intensive case management over a period of 6 to 9 months and aim to help service users navigate the complex service system and establish (or reestablish) access to longer-term community-based connections, resources, and interventions. This particular model has been shown to decrease early psychiatric readmission rates, 23,28,29 improve perceived quality of care, 21,24 improve health and quality of life outcomes, 23,29 and reduce rates of homelessness. 26,30 Past evaluations of CTIs have also suggested that this approach is cost-effective 31 in supporting transitions of care.
More evidence of the impact of CTI models, and their adaptations in diverse settings, is urgently needed to support implementation of effective and cost-effective strategies to improve health and housing outcomes among people experiencing homelessness and who have unmet mental health needs. A recent pre–post evaluation of a CTI adaptation for adults experiencing homelessness and mental health needs who were transitioning from hospital to community services in Toronto, Canada, reported significant improvements in mental and physical health, substance use, and quality of life in the 6 months following the intervention. 23 Expanding on this work, the primary objective of this study was to use administrative health care utilization data to evaluate whether a brief CTI adaptation for adults experiencing homelessness and mental health needs who were discharged from hospital services was associated with significant changes in health care utilization outcomes in the year postintervention compared to the year prior. This analysis in addition compared changes in health care utilization among participants with a recent hospitalization to those of a matched cohort of people experiencing homelessness over the same time period who did not receive the intervention.
Methods
Intervention
The Coordinated Access to Care for Homeless adults (CATCH-Homeless) program is a CTI adaptation in Toronto, Canada, supporting adults experiencing homelessness and mental health needs following discharge from hospital. The program is facilitated through partnerships between 3 local hospitals serving large numbers of people experiencing homelessness, a primary care team, a homeless shelter, a large community mental health agency, and a physician practice plan. The program accepts referrals from all partner hospital emergency departments (EDs) and inpatient units, or from community agencies, and connects participants with transitional case managers who coordinate access to a full range of postdischarge community-based services, including mental health and addiction services, medical care, peer support, housing assistance, and other resources described in depth elsewhere. 32
Study Design
This study is part of a larger mixed-methods evaluation described elsewhere. 21,32 Using provincial population-based administrative databases at ICES (formerly known as the Institute for Clinical Evaluative Sciences), a single-arm pre–post analysis among intervention participants was conducted to evaluate health care utilization outcomes in the year following enrollment in the intervention compared to the year prior. In addition, changes in health service utilization of a subgroup of intervention participants with a recent hospitalization were compared to those of a matched cohort of adults experiencing homelessness with at least 1 psychiatric hospitalization over the same time period who did not receive the intervention.
Data Sources
All administrative health care data were obtained from ICES. ICES is an independent, nonprofit research institute funded by an annual grant from the Ontario Ministry of Health. As a prescribed entity under Ontario’s privacy legislation, ICES is authorized to collect and use health care data for the purposes of health system analysis, evaluation, and decision support. Secure access to these data is governed by policies and procedures that are approved by the Information and Privacy Commissioner of Ontario. The Ontario Mental Health Reporting System (OMHRS) and the Canadian Institute for Health Information’s Discharge Abstract Database (DAD) include data on all psychiatric and acute hospitalizations, respectively. The Ontario Health Insurance Plan (OHIP) claims database contains physician billings data, and the National Ambulatory Care Reporting System gathers data on hospital- and community-based ambulatory care, such as ED visits. The Ontario Registered Persons Database is a registry of all individuals living in Ontario who are eligible for public health care insurance and holds data on patient demographics, such as age, sex, and postal code.
Sample
Intervention participants
Intervention participants were those referred to the intervention by partner sites following discharge from EDs and inpatient units. Intervention eligibility criteria included current homelessness status (defined as living in a crisis/emergency shelter, living on the street, or couch surfing), provider-determined unmet mental health needs, and participant-identified need for support services. Individuals were ineligible for this community-based intervention if they had recent severe aggression or illness severity that required institutional care. Study eligibility criteria included being a resident of the province of Ontario, age 18 years and older, and not having previously received services from the program. For the current analysis, 197 individuals who received the intervention between an index date of January 7, 2013, and May 16, 2014, completed a baseline interview, and agreed to health care record linkage were included. Twenty-six individuals (11.7% of total participants enrolled in the intervention) did not consent to health care record linkage and were excluded from the current analysis. These individuals were not different from consenting individuals in terms of gender but were younger (mean age: 34.9 years vs. 40.3 years; P = 0.021). Furthermore, they were not different in terms of baseline ED visits but had fewer baseline hospitalizations (mean: 0.7 vs. 1.2; P = 0.022). Additional exclusion criteria for this analysis included missing patient identifier (required for data linkage), not being enrolled in OHIP, and/or missing data on sex and/or age.
Matched cohort participants
The comparison group was composed of individuals aged 18 years and older with recorded homelessness (as identified by a residence variable in the OMHRS or a homelessness variable in the DAD), who had at least 1 hospitalization for mental health or substance use during the exposure period (index date: January 1, 2013, to May 31, 2014). Individuals who received the intervention and who had at least 1 hospitalization in the 12 months prior to the index date were then matched 1:2 with the comparison group of adults experiencing homelessness who had not received the intervention using propensity score-based matching. Variables used to calculate the propensity score included having a hospitalization in the 12 months prior to the index date, age, sex, neighborhood income quintile, and administrative health region of residence. We selected the closest control that met the following criteria: age within 2 years at the index date, same sex (hard match), and a propensity score within a caliper width of 0.20. To evaluate matching success, standardized baseline differences were calculated between the subset of intervention participants and the matched cohort. Standardized differences of 0.10 or less are considered negligible. 33 Individuals were excluded from the matched group if they were ineligible for OHIP and/or had missing sex and/or age data.
Outcome Measures
To evaluate changes in acute health care utilization, count outcomes including the total number of all-cause and mental health and substance use-specific ED visits, hospitalizations, and total hospital days were calculated. Dichotomized outcomes indicating whether or not a participant had any all-cause and mental health and substance use-specific ED visits and hospitalizations during the observation period were also calculated. To evaluate changes in outpatient service use, count outcomes including the total number of psychiatrist visits, all-cause and mental health and substance use-specific general practitioner (GP) visits, and the sum total of outpatient visits were calculated, as well as dichotomized outcomes indicating whether or not a participant had any psychiatrist visits and all-cause and mental health and substance use-specific GP visits during the observation period were calculated. Diagnostic codes used to determine mental health and substance use-specific hospital and outpatient visits are provided in Supplemental Table 1.
Data Collection
Intervention participants’ demographic and administrative health care records, held by ICES, were linked using unique encoded identifiers; deterministic linkage was applied using OHIP numbers. Health care records were examined for the 12 months pre- and postindex date. The index date for exposure was the intervention enrollment date, ranging from January 7, 2013, to May 16, 2014; the study period was January 2012 to May 2015. All participants provided written informed consent and the study received Research Ethics Board approval from Unity Health Toronto. To identify the matched cohort used in comparative analyses, population-based administrative demographic and health records were retrieved and reviewed for the same time periods.
Statistical Analyses
Single-arm pre–post analysis
Baseline characteristics for both intervention participants and individuals included in the matched cohort were calculated using frequencies and proportions for categorical variables and means and standard deviation (SD) for continuous variables. In the pre–post evaluation, inferential analyses for each count outcome included rates and rate ratios (RRs) and their respective 95% confidence intervals estimated using generalized estimating equation (GEE) models with a negative binomial distribution and a log link. For each binary outcome, predicted probabilities and prevalence ratios (PRs) and their respective 95% confidence intervals were estimated using GEE models with a Poisson distribution and a log link. All models were adjusted for age, sex, and neighborhood income quintile (measured at the census tract level). An offset variable for person-years was included to adjust for different follow-up lengths.
All analyses were conducted using SAS Enterprise Guide 7.1. A significance level of 0.05 was used to interpret statistical significance.
Post hoc comparative analysis
Baseline demographic characteristics were calculated between intervention participants with at least 1 hospitalization in the 12 months preindex date and their matched counterparts using frequencies and proportions for categorical variables and means and SD for continuous variables. To show trends between groups and over time, GEE models specifying a negative binomial distribution and a log link were conducted for each count outcome to estimate RRs and relative rate ratios (RRRs); for each binary outcome, GEE models specifying a Poisson distribution and a log link were conducted to calculate PRs and relative prevalence ratios (RPRs). All post hoc models were adjusted for age, sex, and neighborhood income quintiles and included group, time, and group by time interaction variables.
Results
The cohort selection is detailed in Figure 1. Of the 197 eligible individuals who received the intervention during the study period, 11 health records were unlinkable (6 were blank and 5 were invalid), resulting in a cohort of 186 participants. Of those, 51 individuals were excluded from the matched analysis because they did not have a hospitalization for mental health or substance use in the 12 months prior to the index date required for comparison with the matched sample, leaving a subset of 125 individuals. Among 3,555 administrative records of people experiencing homelessness with a mental health or substance use-specific hospitalization during the exposure period, 250 individuals (1:2 matching) were selected for inclusion in the matched cohort.

Exposed and matched cohort selection.
Demographic Characteristics
The demographic characteristics of intervention participants (n = 186) are presented in Supplemental Table 2. The majority of the sample (78.5%) was male. Participants had a mean age of 40.3 years (SD = 12.0), with an even distribution across age brackets between 18 and 59 years; only 4.3% of participants were over the age of 60.
Table 1 shows the demographic characteristics of the 125 intervention participants included in the matched cohort analysis and their matched counterparts; these were similar to those of the full cohort of intervention participants in age, sex, and neighborhood income quintile as indicated by no standardized difference between the 2 groups being greater than 0.10.
Demographic Characteristics of CATCH-H Participants with at Least 1 Hospitalization in the 12 Months Preintervention (n = 125) and Matched Controls (n = 250).
Note. CATCH-H = Coordinated Access to Care for Homeless; SD = standard deviation.
a1: includes individuals in the first income quintile and those where this information was missing.
Pre–Post Analysis
Rates per person-year and RR for intervention participants (n = 186) in the 12 months pre- and postintervention enrollment for the number of hospital admissions, hospital days, ED visits, and outpatient visits are presented in Table 2; results for binary outcomes are presented in Supplemental Table 3. Overall, results reveal changes in health care utilization patterns, with decreased inpatient care and increased outpatient care in the 12 months following enrollment in the intervention, compared to the 12 months prior. Inpatient hospital use decreased significantly among intervention participants in the 12 months following enrollment, especially for mental health and substance use-specific visits. Adjusted admission rates decreased significantly for both all-cause and mental health and substance use-specific causes, by 33% (95% CI, 15 to 47; P = 0.001) and 43% (95% CI, 26 to 66; P < 0.001), respectively, in the 12 months postintervention relative to 12 months prior. The total number of hospital days also decreased significantly, by 54% (95% CI, 36 to 77; P < 0.001) for all-cause days and by 63% (95% CI, 48 to 74; P < 0.001) for mental health and substance use days. While the number of ED visits did not change significantly over time, the overall prevalence of ED visits did decrease significantly, by 14% for any reason (95% CI, 7% to 20%; P < 0.001) and by 31% for mental health and substance use reasons (95% CI, 12% to 39%; P < 0.001; Supplemental Table 3). In contrast to the decrease in acute hospital service use, outpatient psychiatrist visits increased by 40% (95% CI, 1.15 to 1.70; P = 0.001) in the 12 months following the intervention compared to the 12 months prior; GP visits did not change significantly.
Rates Per Person-Year and RR for CATCH-H Participants (n = 186) 12 Months Preintervention versus 12 Months Postintervention for Number of Hospital Admissions, Hospital Days, ED Visits, and Outpatient Visits Estimated from Adjusted Generalized Estimating Equations with Negative Binomial Distribution.
Note. CATCH-H = Coordinated Access to Care for Homeless; CI = confidence interval; ED = emergency department; GP = general practitioner; RR = rate ratio.
Matched Cohort Analysis
RRs and RRRs for the subset of intervention participants with at least 1 hospitalization in the 12 months prior to the index date (n = 125), compared to matched controls (n = 250) 12 months pre- and postintervention, for the number of hospital admissions, hospital days, ED visits, and outpatient visits are presented in Table 3. PRs and RPRs are presented in Supplemental Table 4. For both count and binary outcomes, results of the matched cohort analysis suggest that while results generally trended in the same direction as the pre–post analysis, the changes in health service use experienced by intervention participants in the year following receipt of the intervention as compared to the year prior were not significantly different than the changes experienced by their matched counterparts (Table 3).
Rate ratios and RRR for CATCH-H Participants with at Least 1 Hospitalization in the 12 Months Preintervention (n = 125) Compared to Matched Controls (n = 250) 12 Months Preintervention versus 12 Months Postintervention for Number of Hospital Admissions, Hospital Days, ED Visits and Outpatient Visits Estimated using Adjusted Generalized Estimating Equations with Negative Binomial Distribution.
Note. CATCH-H = Coordinated Access to Care for Homeless; CI = confidence interval; ED = emergency department; GP = general practitioner; RR = rate ratio; RRR = relative rate ratio.
Discussion
Care continuity 34 is essential to the delivery of high-quality services for people living with chronic health conditions, including mental illness and addictions, and is associated with improved health and service use outcomes. 35 Our findings of decreased acute care service use and increased outpatient service use in the 12 months following the intervention as compared to the 12 months prior are consistent with previous outcomes of CTI studies, finding significant changes in outcome measures. 23,28,29 In addition, our findings further support and are supported by previous qualitative research suggesting improved experiences of continuity of care among adults experiencing homelessness receiving CTI or CTI adaptations. 21,36 Notably, our findings highlight continued high rates of ED visits postintervention among participants, suggesting that this population may have more acute and/or enduring needs than the brief CTI adaptation can address in our local context. Access to housing and high-quality intensive case management in Toronto, for example, are extremely limited and hindered by long wait lists; it is possible that participants continued to visit the ED for immediate shelter and support postintervention, as intervention resources were limited and not linked to housing. Persisting high ED use among participants reaffirms the need for inclusion of housing in mental health policy priorities. 16
Although prior controlled studies of CTI have demonstrated improvements in acute care utilization in some settings, 25 the addition of a matched cohort analysis in the current study indicates that changes in service use patterns of recipients of a brief CTI adaptation within our setting of universal health insurance and a relatively service-rich environment are not significantly different than those of their matched counterparts. Similar to this analysis, a prior post hoc analysis of intervention participants using self-reported data and a comparison group of adults experiencing homelessness and mental illness who were receiving usual care found successful reduction in mental health symptoms and alcohol and drug use problems among intervention participants, but increased ED visits and days spent in hospital over 6 months. 23 The current post hoc analysis, in using administrative data and a cohort matched on prior hospitalizations, was intended to build off the limitations of the aforementioned prior analysis 23 by focusing on service use outcomes using administrative data and exploring between-group differences with added methodological rigor. Additional measures including clinical characteristics such as diagnosis and acuity, and demographic and service use factors such as geographical location and resource availability, may help to better explain the identified nonsignificant differences between groups in future studies.
Our findings further highlight the need for rigorous methods in evaluating new interventions in diverse contexts. The brief CTI adaptation evaluated in the current study was set in a large urban center under a universal health insurance system and included a dedicated, low-barrier, multidisciplinary weekly physician clinic in addition to brief case management support with limited training and supervision of frontline staff, in keeping with a pragmatic field intervention. Our findings suggest that participants’ access to preventative outpatient services was successfully facilitated by streamlined access to GPs and psychiatrists. Such an approach may be particularly relevant to similarly large urban center where timely access to physician and case management resources is hindered by rapid general and homeless population growth and corresponding demand for services. While fidelity standards for CTI have been developed, detailing the key model components requiring adherence, and the contextual structures and staff competencies needed to ensure model integrity, 37 –39 a recent systematic review indicates that fidelity and adaptations remain highly variable 25 and should be a required component of rigorous evaluations. The fidelity of the intervention to the CTI model was not formally assessed in this study. Future efforts should ensure that local adaptations of evidence-based interventions balance local needs and resources with fidelity to key intervention ingredients, so that evidence generated can be reliably attributed to the model of interest.
Strengths and Limitations
Our findings contribute to the growing evidence on the impact of CTI and its adaptations on health service use. Results are strengthened by the study’s methodological approach, which included the use of population-based administrative data and a rigorous selection process to ensure the appropriateness of comparators. However, the identification of a matched cohort was challenged by the lack of a baseline acuity measure, requiring us to match on recent hospitalizations as a proxy for acuity, although postmatching demographic comparisons suggested the groups were minimally different. Additionally, intervention participants were selected using some clinical criteria that could not be applied to matched individuals identified from administrative data. In the absence of a built-in control group, this an acknowledged design limitation could have resulted in selection bias.
We were also limited by our sample size and underpowered to detect hospitalization differences. It is possible that the lower relative rates of psychiatric hospitalizations observed in the intervention group would be significantly different with a larger sample. Still, without a randomized design, it is possible the improvements in health care use observed in the pre–post analysis are due to regression to the mean, rather than the effect of the intervention.
Given the high cost of hospitalizations and from a quality of care perspective, future research should focus on rigorous evaluations using experimental methods with large samples and robust sets of explanatory variables, in addition to approaches to strengthening fidelity of local adaptations to evidence-supported treatments. Related research should investigate the cost-effectiveness of these interventions to maximize health outcomes given the limited resources available.
Conclusions
Study findings suggest that participants of a brief CTI for adults experiencing homelessness and unmet mental health needs had decreased acute care use and increased outpatient service use postintervention. Post hoc analyses, however, found that changes in service use patterns were not significantly different from those of a matched cohort of adults experiencing homelessness. While CTI and its adaptations hold promise in improving continuity of care and health outcomes for this population, more rigorous study designs with adequate sample sizes are needed to further examine the effectiveness of CTI and local adaptations in diverse health care contexts.
Supplemental Material
Supplemental Material, sj-docx-1-cpa-10.1177_0706743721996114 - Evaluating the Impact of a Critical Time Intervention Adaptation on Health Care Utilization among Homeless Adults with Mental Health Needs in a Large Urban Center: Évaluer l’effet d’une Adaptation De l’intervention En Temps Critique Sur l’utilisation des Soins De Santé Chez Des Adultes Itinérants Ayant Des Besoins De Santé Mentale Dans Un Grand Centre Urbain
Supplemental Material, sj-docx-1-cpa-10.1177_0706743721996114 for Evaluating the Impact of a Critical Time Intervention Adaptation on Health Care Utilization among Homeless Adults with Mental Health Needs in a Large Urban Center: Évaluer l’effet d’une Adaptation De l’intervention En Temps Critique Sur l’utilisation des Soins De Santé Chez Des Adultes Itinérants Ayant Des Besoins De Santé Mentale Dans Un Grand Centre Urbain by Nadine Reid, Joyce Mason, Paul Kurdyak, Rosane Nisenbaum, Claire de Oliveira, Stephen Hwang and Vicky Stergiopoulos in The Canadian Journal of Psychiatry
Footnotes
Authors’ Note
Data included in this publication are not publicly available and were collected by ICES. As a prescribed entity under Ontario’s privacy legislation, ICES is authorized to collect and use health care data for the purposes of health system analysis, evaluation, and decision support. Secure access to these data is governed by policies and procedures that are approved by the Information and Privacy Commissioner of Ontario. For further questions regarding data accessibility, please contact the corresponding author.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health (MOH). Parts of this material are based on data and information compiled and provided by MOH and the Canadian Institute for Health Information. The analyses, conclusions, opinions and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
