Abstract
Background and aims
Practice-based research holds potential as a promising solution to closing the research-practice gap, because it addresses research questions based on problems that arise in clinical practice and tests whether systems and interventions are effective and sustainable in a clinical setting. One type of practice-based research involves capturing practice by collecting evidence within clinical settings to evaluate the effectiveness of current practices. Here, we describe our collaboration between researchers and clinicians that sought to answer clinician-driven questions about community-based language interventions for young children (Are our interventions effective? What predicts response to our interventions?) and to address questions about the characteristics, strengths, and challenges of engaging in practice-based research.
Methods
We performed a retrospective chart review of 59 young children who had participated in three group language interventions at one publicly funded community clinic between 2012 and 2017. Change on the Focus on the Outcomes of Communication Under Six (FOCUS), a government mandated communicative participation measure, was extracted as the main outcome measure. Potential predictors of growth during intervention were also extracted from the charts, including type of intervention received, attendance, age at the start of intervention, functional communication ability pre-intervention, and time between pre- and post-intervention FOCUS scores.
Results
Overall, 49% of children demonstrated meaningful clinical change on the FOCUS after their participation in the language groups. Only 3% of participants showed possibly meaningful clinical change, while the remaining 46% of participants demonstrated not likely meaningful clinical change. There were no significant predictors of communicative participation growth during intervention.
Conclusions
Using a practice-based research approach aimed at capturing current practice, we were able to answer questions about the effectiveness of interventions delivered in real-world settings and learn about factors that do not appear to influence growth during these interventions. We also learned about benefits associated with engaging in practice-based research, including high clinical motivation, high external validity, and minimal time/cost investment. Challenges identified were helpful in informing our future efforts to examine other possible predictors through development of a new, clinically feasible checklist, and to pursue methods for improving collection of outcome data in the clinical setting.
Inherent differences between how evidence is developed in a laboratory setting and how evidence is applied clinically can create a gap between research and practice. Similar to the situation in various health, medical, and educational fields, this research-practice gap also exists in the field of communication sciences and disorders (Crooke & Olswang, 2015; Green, Ottoson, et al., 2009). In order to make research more accessible to and useful for clinicians and to partner in the implementation of evidence-based practice in the field of communication sciences and disorders, the gap between research and practice needs to be reduced (Allen et al., 2017; Crooke & Olswang, 2015; Green, Ottoson, et al., 2009). In this paper, we describe our collaboration between a group of community clinicians and researchers that created an opportunity to engage in practice-based research through a retrospective chart review. Our collaboration sought to answer clinician-driven questions about community-based early language interventions and to address questions about the characteristics, strengths, and challenges of engaging in practice-based research.
Olswang and Prelock (2015) describe some of the specific challenges associated with attempting to translate research into clinical practice both generally and particular to the field of communication sciences and disorders. They argue that pushing research into practice can be challenging because (a) research findings may not be clinically relevant, (b) descriptions of treatment may not be sufficient to allow adequate fidelity of implementation, (c) organizations may challenge the introduction of new treatment innovations, (d) clinicians may lack motivation to change their practice, and (e) the benefits of a new treatment for a clinical population may not be sufficient to warrant implementation (Olswang & Prelock, 2015). One critical component of the clinical application of evidence-based practice is fidelity. To effectively implement an evidence-based intervention, fidelity to the original intervention must be maintained (Allen et al., 2017). When studies fail to report interventions in sufficient detail, it becomes difficult to ensure that interventions have been implemented with fidelity. When interventions are not implemented with high fidelity, the research outcomes may not carry over to clinical settings (Kaderavek & Justice, 2010). To respond to such challenges, we need new approaches to narrowing this gap from both sides, namely, not only bringing evidence-based research into practice but also bringing real-world practice into research. Practice-based research is a bi-directional approach to achieving this goal.
Practice-based research has been described as the application of research principles and methods to existing clinical practice through collecting data and information in order to investigate clinical questions that realistically inform clinical practice (Crooke & Olswang, 2015; Epstein, 2001). Practice-based research has potential as a promising solution to the challenges that have led to the research-practice gap, because it addresses research questions based on problems that arise in clinical practice, and tests whether the implementation of systems and interventions are effective and sustainable in a clinical setting (Westfall et al., 2007). Practice-based research provides an opportunity to further engage clinicians in the research process by offering opportunities for participatory research processes and evidence generation from settings similar to theirs (Green & Nasser, 2017). While laboratory studies offer more control than practice-based research, and thus boast high internal validity, evidence can be harder to apply to a real-world setting. Studies based on real-world clinical practice are important because evidence from real-life practice creates evidence with higher external validity or evidence that can be more easily applied to clinical practice (Green, Glasgow, et al., 2009; Olswang & Prelock, 2015). Despite the benefits of engaging in practice-based research, Olswang and Bain (2013) determined that based on a review of articles published in three communication disorders journals, less than 10% of studies published between 2006 and 2010 examined the effectiveness of speech-language pathology treatment in practice or natural settings.
Vollebregt et al. (2018) proposed a model to categorize practice-based research practice partnerships. The model consists of three stages of cocreation: (a) research capturing practice, where evidence is collected from the clinical setting to evaluate the effectiveness of current or ongoing clinical practices, (b) research changing practice, which involves the implementation of evidence-based approaches from research into the clinical setting, and (c) research creating practice, where researchers and clinicians cocreate practice-based approaches and evaluate their effectiveness and implementation. These three stages of cocreation span the research process and each reflect an important component of the research-practice partnership.
Capturing practice is one level of cocreation that may help close the practice-research gap and improve clinical practice by determining whether current clinical practices are or are not effective. Because research is embedded in clinical practice, the research and its conclusions emerge from a real-world clinical setting, making them more likely to be clinically feasible and therefore implementable. Research that captures current practice can be used to inform clinical decision-making and inform future research-practice examination of alternative practices.
This paper outlines our first effort to conduct practice-based research that involved capturing practice, specifically, through a retrospective clinical chart review. This project grew out of a partnership between researchers and community clinicians, who came together out of a mutual desire to close the gap between research and practice in the area of group early language interventions by applying research-inspired principles to evaluate the effectiveness of current practice.
Research questions and aims
The clinicians and researchers entered this partnership with similar primary research questions surrounding capturing practice. A precursory examination based on a model of cataloging the current clinical practices led to the determination that we would investigate the outcomes and predictors of intervention. The clinically focused research questions were the following: (a) are the group language interventions being delivered in the community clinic effective and (b) what factors predict response to these interventions? The clinicians asked these questions to determine whether what they were doing clinically was working and with the hope of improving their ability to triage clients to different interventions based on their needs and adjust their interventions for individual clients. As a partnership, we hypothesized that young children would show language gain following intervention and that a combination of child, family, and program variables would predict response to these group interventions.
Our research team also had several motivators in conducting this practice-based research study. From a clinical implications perspective, the research team’s aims were to (a) evaluate whether interventions that were designed by clinicians and structured to fit within the constraints of the practice and administrative environment were effective, (b) determine what interventions are and are not working and for whom, (c) work toward making necessary adjustments to the interventions to improve effectiveness where needed, and (d) be able to report results to the broader scientific community to benefit children outside of this one clinic. Additionally, the research team was interested in assessing the utility of practice-based research to (a) determine whether capturing practice through practice-based research is feasible and useful, (b) identify barriers and facilitators to success in this level of practice-based research, and (c) use lessons learned in this study to inform future efforts to conduct studies capturing practice. Potential learning opportunities included ideal outcome measures from both measurement and feasibility perspectives, identification of additional data that are not currently being collected in the course of routine practice that might be informative and feasible to collect, and overall practicality in terms of clinician time and clinical benefit. To inform our study planning, we consulted the extant literature on early language interventions and possible predictors of response to these interventions.
Group language interventions
Parent and child group language interventions like those offered by our community partners (see Methods section for further details) have been shown to facilitate improvements in expressive language and vocabulary growth (Fricke et al., 2013; Lederer, 2001; Robertson & Weismer, 1999). While intervention decisions must be made based on the needs of the child and their family (Baxendale & Hesketh, 2003; Nye et al., 1987; Olswang et al., 1998), parent and child group interventions have been shown to produce similar outcomes to individual therapy (Boyle et al., 2007; Lederer, 2001). Research has shown that direct and indirect group language intervention programs run by speech-language pathologist (SLPs) or a communication disorders assistant (CDAs) result in similar language outcomes (Boyle et al., 2007; Law et al., 2005).
Predictors of response to intervention and change during intervention
Of particular importance to identifying predictors is the distinction between response to intervention and change or growth during intervention. Response to intervention and growth can co-occur throughout a period of intervention, but in order to determine what specifically predicts an individual’s response to intervention, the change in response to intervention and the change associated with other factors as they relate to an individual’s growth must be separated (Yoder & Compton, 2004). Unfortunately, the extant literature on predictors of response to intervention has been inconsistent in making this important distinction. We have made an effort below to differentiate literature on predictors of response to intervention and predictors of growth.
Predictors of response to intervention include maternal responsiveness and maternal education for children with developmental delays participating in language intervention (Yoder & Warren, 2001). As summarized by Leonard (2014), other variables that can predict a child’s response to intervention include variables specific to the program, such as intervention dosage, duration, and intensity. These variables encompass the number of presentations of a particular form during a session and across a period of time, the length of time of intervention, and the overall number of presentations of a particular form based on presentations per session across the time of intervention (Conn & Chan, 2016; Leonard, 2014; Proctor-Williams et al., 2001). Additionally, a child or family’s attendance at intervention is predictive of language growth that occurs as a response to intervention and has been found to moderate the impact that intervention has on language outcomes (Arbour et al., 2016; Justice et al., 2008; Leonard, 2014).
Predictors of language growth reflect those variables that have been shown to predict progress in language development but have not been specifically attributed to intervention over other factors (see Yoder & Compton, 2004 for a more detailed description). Predictors of growth in language include variables such as pre-intervention language level or ability (Johanson et al., 2016; McLean & Woods Cripe, 1998; Olswang & Bain, 1996), maternal language use (Abraham et al., 2013; Girolametto et al., 1999), and parental interaction characteristics and style (Brady et al., 2004; Garcia et al., 2015; Hart & Risley, 1995; Olswang et al., 1998). Although it is possible that these factors might also predict response to intervention, the necessary evidence is currently lacking in the extant literature. More generally, it can be expected that developmental factors such as age could impact language growth.
The lack of differentiation of response to versus change during intervention in much of the literature on early language interventions meant that we would not be able to fulfill our original goal of identifying predictors of response to intervention specifically. However, given the data we anticipated being able to access in the clinical charts, we did see value in trying to identify child, family, and intervention factors that had potential to predict growth during intervention using the current dataset in order to inform future prospective studies of possible response to intervention predictors. As described in further detail in the Method section, we thus developed a list of potential predictors to collect from the charts that included (a) predictors of response to intervention or change during intervention from the literature and (b) factors that the clinicians in the participating clinic believed to be predictive of how well a child progressed during their group interventions.
Method
Practice-based research processes
The first step in engaging in our practice-based research project was to determine whether it would be possible to investigate our research questions with the information already available in the clinical charts, that is, to conduct a retrospective chart review. While traditionally, retrospective studies are not considered to provide a strong source of evidence, in practice-based research, retrospective studies offer a unique opportunity to study what occurs naturally in practice without external interference or suggested change (Crooke & Olswang, 2015). A retrospective chart review offers one way to capture current practice (the category described by Vollebregt et al., 2018). Although its original purpose is to support clinical service delivery, the information that is routinely collected through clinical practice can be mined to provide valuable retrospective information about patient needs, interventions, and outcomes that can help capture what is occurring in practice (Epstein, 2001).
Through a feasibility assessment, our group determined that we met three requirements in order to be able to attempt a retrospective chart review: (a) an intervention model or program was being delivered in the clinic that had foundations in the scientific evidence in the literature to date (e.g., Boyle et al., 2007; Fricke et al., 2013; Lederer, 2001; Robertson & Weismer, 1999), was being delivered in a similar or consistent way across children, and had been completed by enough children to provide an adequate sample size; (b) results of outcome measures were available for the children who had participated in this intervention, and these measures had been administered both pre- and post-intervention and (in some instances) also during a pre-treatment waiting period; and (c) information was available about child, family, and service factors suggested in our literature review to be potential predictors of response to intervention. Ultimately, our consideration of these questions and consultation with the literature led us to believe that we had (or at least thought we might have) sufficient information available to conduct a retrospective clinical chart review capturing practice to evaluate the effectiveness of group language interventions for 1 to 5 years olds. How we arrived at this decision is described in further detail below.
The interventions
Our review of the interventions offered in the clinic identified three parent and child group interventions that met our criteria of being grounded in evidence and being delivered consistently to a sufficiently large sample (Talking with Tots, Early Language Group, and Word Combination Group). Overall, each program had been running in a similar way over a 4-year period. All groups were designed for a child and their parent or caregiver and were run by either a SLP or a CDA. Each group intervention was followed by a 3- to 4-month consolidation period during which time the parents practiced the skills learned in the group at home. The programs had different age of entry criteria (described below). Each program required that the child demonstrate expressive language difficulties, but this could be primary or secondary to another developmental disorder, and the child may also demonstrate receptive language difficulties. Clinicians made decisions about which intervention groups children were placed in based on the child’s age, as well as considerations about family schedules and length of wait lists. The programs were not designed to target speech sound difficulties, which are addressed in individual intervention in the clinic or at daycare.
Talking with Tots
Both parents or caregivers and their child attend the Talking with Tots group, which is designed for toddlers who are between the ages of 16 and 20 months and have limited speech with few words in their vocabulary. Toddlers with any combination of expressive, receptive, or social communication difficulties can be included in this group. It consists of three sessions run by a SLP, and each group contains no more than four families to build a community of trust and openness between families. The group focuses on training parents and caregivers to use early language facilitation strategies in interacting with their child. Talking with Tots teaches parents early language facilitation strategies that are commonly used in various early language communication interventions and supported by the literature, such as modelling; repeating; focused stimulation; observing, waiting, and listening (OWL’ing); engaging in face to face interactions; imitating; interpreting gestures; asking good questions; following the child’s lead; and expanding (Dunst et al., 1990; Girolametto et al., 1996, 1999; McDonald et al., 2015; Özçalişkan & Goldin-Meadow, 2005; Rhyner et al., 2013). Because this group focuses on parent training, individual goals for toddlers are neither set nor targeted. Instead, the SLP explains strategies, demonstrates their use, and allows parents to practice implementing them during a session.
Early Language Group
The Early Language Group is run by a CDA, and each family receives six group sessions and one one-to-one session, which are all attended by a parent or caregiver and the toddler. This group is designed for toddlers between 24 and 30 months and focuses on daily routines to target development of vocabulary, functional words, social skills, and the ability to sit and participate in circle time activities. Each session revolves around a theme (e.g., toys and the park, birthdays) designed to incorporate vocabulary, songs, stories, and language opportunities with particular goals. There is some parent training in this program involving strategies similar to those introduced in Talking with Tots, but the primary focus is on the toddler’s progress. Goals are set based on where the toddler’s language begins. Goals can range from increasing vocalizations or functional gestures for non-verbal toddlers to building spontaneous functional vocabulary for toddlers who start the Early Language Group with some basic vocabulary.
Word Combination Group
Toddlers who are about 30 months are referred to the Word Combination Group. It also consists of six group sessions and one one-to-one session, is attended by a parent or caregiver and their child, and is run by a CDA. The focus of the Word Combination Group is vocabulary themes (e.g., animals) that promote the development of vocabulary, functional phrases, word combinations, and social skills. This group is recommended once vocabulary has developed and focuses on the child’s ability to put words together. Much like the Early Language Group, this group continues developing early language facilitation strategies for parents, with a particular focus on expanding, but primarily targets the language development of the toddlers. In this program, goals set for each child focus on the development of functional phrases. Depending on the toddler’s current language and communication abilities, this could mean two-word phrases or longer functional phrases. The Word Combination Group pays particular attention to the development of categories of words (e.g., location words, action words, concept words) to facilitate the production of phrases and sentences.
Participants
Participants were 59 children between the ages of one and five years who had completed one of three group language interventions at the community speech and language clinic in London, Ontario. While no child could participate in multiple intervention groups at once, it was possible for a child to have participated in groups sequentially. In these cases, outcome measures were collected before and after each group so that participants’ outcome measure scores did not span participation in two groups. Sixteen children received intervention through the Talking with Tots group, 20 received the Early Language group intervention, and 23 participated in the Word Combination Group. Twenty-one children had previously participated in one of the other group interventions in addition to the intervention group for which their data were used in this analysis. Table 1 displays the demographic information for the sample as a whole and for each group individually.
Participant demographics.
M: male; F: female.
The outcome measures
The community clinic in which the study took place is a regional site of Ontario’s Preschool Speech and Language (PSL) program, which provides self-referred, provincially funded access to speech-language pathology services for children from birth until school entry. The PSL program requires its clinicians to use two outcome measures (described in further detail below): the FOCUS (Thomas-Stonell et al., 2012) and the Communication Function Classification System (CFCS, Hidecker et al., 2011). These outcome measures assess a child’s communicative participation and functional communication ability, respectively. The PSL program guidelines indicate that clinicians should complete the FOCUS and the CFCS (a) at the initial assessment if the first intervention is accessed at the same visit, (b) at the beginning of the first intervention if not completed at the initial assessment, (c) at all re-assessments, and (d) at discharge from the PSL program (Ontario Ministry of Children and Youth Services, 2015). Due to the nature of services offered at this particular community clinic, the measures were completed at the initial assessment visit, and depending on time spent on the waitlist, sometimes the first intervention visit. Because of these guidelines, how they were applied to this community clinic’s practice, and the fact that they are applied to all children regardless of intervention type, we anticipated that there might be consistent FOCUS and CFCS data across children from the three groups during a waiting period and pre- and post-intervention. Ultimately, waiting period data (initial assessment and start of intervention) were not available for enough participants to be included. As such, one pre-intervention time point was included.
FOCUS
The FOCUS is a 50-item questionnaire with 7-point Likert scale questions that is filled out by a parent, caregiver, or teacher and is designed to measure changes in communicative participation skills during speech-language intervention in children under six years. The questions occur in two forms: a range from Not at all like my child to Exactly like my child and a range from Cannot do at all to Can always do without help. Children’s scores on the FOCUS can range from 50 to 350, and outcome is measured by comparing score change across administrations. A 16-point change in FOCUS score from pre-intervention to post-intervention is considered a minimally clinically important difference, which was established using Kappa to assess the agreement that meaningful change had occurred between parents and clinicians who had filled out FOCUS forms for the same children (Thomas-Stonell et al., 2012; Thomas-Stonell, Oddson, et al., 2013). Research has shown that over a 2-month waiting period, change in FOCUS scores do not demonstrate clinically meaningful change in communicative participation, with an average change over 2 months of 5.9 points (Thomas-Stonell, Washington, et al., 2013). This suggests that a change in FOCUS scores that is considered a minimally clinically important difference demonstrates a change beyond what is seen during a period of waiting for intervention.
CFCS
The CFCS, a measure of functional communication ability, is designed for use as a classification tool, not an assessment tool (Hidecker et al., 2017) and reflects an adult’s evaluation of the child’s ability to communicate with others. In the PSL Program, SLPs make the CFCS classifications. Scores can range from Level I to V with each level representing a different profile of communication. Level I indicates that the child is an “effective sender and receiver with unfamiliar and familiar partners,” Level II describes an “effective but slower paced sender and/or receiver with unfamiliar and familiar partners,” Level III reflects an “effective sender AND effective receiver with familiar partners,” Level IV indicates an “inconsistent sender and/or receiver with familiar partners,” and Level V describes a child who is a “seldom effective sender and receiver with familiar partners.” Construct validity and predictive validity have been established for use of the CFCS in samples of young children (birth–6 years) with communication disorders (Hidecker et al., 2017).
Once we came to understand the nature of this classification tool, we realized that the CFCS score was best treated as a predictor as opposed to an outcome measure for our study. Because the CFCS provides a classification score, the degree of change in classification that represents significant growth is not reported. Additionally, the CFCS classifies children based on their ability to communicate with others irrespective of their age. Even children with typical development start off at Level V and progress to Level I. Because the classification system does not consider age, a 2-year-old developing typically would rarely be classified as a Level I. A 2-year-old child with typical development may score the same as a 2-year-old child with language difficulties because almost all children who are two years old experience some degree of communication breakdown. A change in classification indicates growth in functional communication ability, but it is difficult to quantify what amount of growth constitutes growth beyond normal development, particularly in cases such as ours where there is a broad age range of participants.
Data collection
Ethical approval was obtained through the University of Western Ontario’s Research Ethics Boards. Charts were reviewed between May 2016 and August 2017 inclusive for clients who had participated in the group interventions between January 1, 2012 and April 27, 2017, and outcome and predictor data were extracted where available.
The outcome data collected included the FOCUS scores pre- and post-intervention. As described previously, the limited number of factors specifically identified as predictors of response to intervention in the literature to date led us to expand the predictors we planned to collect from the charts in two ways. First, we attempted to gather information on factors identified in the literature as predictors of either response to intervention or change during intervention that we believed would be recorded in the charts. These included intervention duration, attendance, parent interaction style, and pretreatment communication abilities. Other possible predictors from the literature had not been collected as a part of routine clinical practice and thus could not be collected, namely, two response to intervention predictors (maternal responsiveness and maternal education) and one language growth predictor (maternal language use). Second, the clinicians in the participating clinic identified additional factors they believed to be predictive of how well a child progressed during their group interventions. These included age, sex, relationship of the adult participant to the child, other interventions previously received, caregiver involvement, carryover at home, parent use of communication strategies taught, and discharge plan, as well as a variety of developmental indicators (i.e., ability to sit, ability to take turns, ability to stay on task, joint attention, ability to wait, emerging play skills, and emergent literacy skills). For some children, the variables had been coded in the chart as either at risk or not at risk based on clinician judgment. For others, any information relevant to each variable available through clinic notes (e.g., clinician expressed concern about a child, family, or program variable) was assigned a rating of at risk or not at risk by the first author during data extraction.
This list of potential predictors was quite long at the outset of data collection but became much shorter once the chart review was complete. Program variables predicting response to intervention such as dose, duration, and intensity were not tracked directly, although due to differences in the interventions offered, were considered implicitly through the inclusion of intervention received as a predictor. Attendance is another program variable that moderates response to intervention and was tracked and included as a predictor in the analysis. Of the remaining predictors of growth from the literature, the only variable with consistently recorded data was the pre-intervention communication level (CFCS level pre intervention). Age of the child at the start of intervention was also available, but the remaining child, family, and program variables suggested by the clinicians could not be considered for inclusion in the analyses due to large amounts of missing data.
It became clear during data extraction that timing of outcome measure collection was quite variable, and as such, one additional code was added to improve accuracy of our data collection. The timing of data collection was recorded as a continuous variable reflecting the time in months between the pre- and post-data collection points and the time between the pre and follow-up data collection points, when follow-up assessments were available. Collecting these data showed us that no participants had FOCUS data collected at both the initial assessment and the start of intervention. While likely the result of both the short wait times to access group interventions and the PSL program guidelines that outcome measures should be recorded at the initial assessment or the start of intervention, this made it impossible to compare gain during pre-treatment waiting with gain during treatment. Despite the absence of data measuring pre-intervention waiting periods, we can feel relatively confident, based on research investigating FOCUS score changes in periods of waiting (Thomas-Stonell, Washington, et al., 2013), that a meaningful clinically important difference on the FOCUS is representative of a change related to intervention and not one occurring solely because of natural development.
To summarize, based on the literature, clinician judgment, and data availability in the charts, the predictor variables included in the statistical analyses were the following: (a) intervention received, (b) attendance, (c) age at the start of intervention, (d) CFCS level pre-intervention, and (e) timing of data collection.
Statistical analyses
Analyses were chosen to provide meaningful answers to the research questions posed by the clinician–researcher team. First, the proportion of children who demonstrated meaningful clinical change on the FOCUS (change >15 points), possibly meaningful clinical change on the FOCUS (change of 9–15 points), and not likely meaningful clinical change (change <9 points) for each intervention type was calculated (Thomas-Stonell et al., 2012; Thomas-Stonell, Oddson, et al., 2013). Second, a direct-entry regression was run on the change in FOCUS scores with group as the predictor. Group was a categorical variable because each participant could only be enrolled in one group at a time. To assess whether group significantly predicted change on the FOCUS, participation in each group was dichotomously coded as yes or no, and each dichotomous variable was entered into the regression. Third, we performed a direct-entry regression on change in FOCUS with attendance (% of sessions attended), age at the start of intervention (in months), CFCS level before intervention, and timing of data recording (months between time 1 and time 2) as predictors. We chose to use direct-entry regression, because we did not have a theoretical basis for considering any chosen variable prior to any other chosen variable. The literature describes many possible predictors, but contexts are quite variable and the determination that any one variable should be included before any others would not have had theoretical support.
Results
The retrospective chart review provided FOCUS change scores for 59 children across the three intervention groups.
Change in FOCUS score
Table 2 displays the number of children who demonstrated meaningful clinical change, possibly meaningful clinical change, and not likely meaningful clinical change. Overall, 49% of all group language intervention participants demonstrated meaningful clinical change after their participation in the language groups. Only 3% of participants showed possibly meaningful clinical change, whereas the remaining 46% of participants demonstrated not likely meaningful clinical change.
The number (percentage) of children who demonstrated change after completing each of the preschool group language intervention programs.
Regressions
A direct-entry regression run with change in FOCUS score as the dependent variable and group as the independent variable demonstrated that group was not a significant predictor of change in FOCUS score. This model predicted less than 1% of the variance in change in FOCUS score (p = .30). Because there were no differences in change in FOCUS score by group, the second regression was run on the entire sample across the three intervention groups. Table 3 provides a summary of change in FOCUS score by intervention group.
Change in FOCUS score by intervention group.
A second direct-entry regression was run with change in FOCUS score as the dependent variable and attendance, age pre-intervention, CFCS classification pre-intervention (as a functional measure of pre-intervention language ability), and timing between data collection points (i.e., months between FOCUS scores) as independent variables. As described previously, these variables have either support in the literature indicating their potential influence on a child’s responsiveness to language intervention or were used because of the nature of the data we were able to gather (i.e., timing between data collection points). Table 4 provides a summary of the regression model. The model explained <1% of the variance in FOCUS score change suggesting that no factors significantly predict FOCUS score change.
Summary of coefficients, standard error, t values, and p values for change in FOCUS score.
Note. Model accounts for <1% of variability in change in FOCUS score. p = .55.
CFCS: Communication Function Classification System; FOCUS: Focus on the Outcomes of Communication Under Six.
Discussion
Our clinician–researcher collaboration sought to answer a number of clinical and process-related research questions. Clinically, we were interested in determining whether the group language interventions being delivered in this community clinic were effective and what factors predicted change occurring during these interventions. From a research perspective, we were also interested in assessing the utility of practice-based research, specifically, to determine whether capturing practice is a feasible and useful method of performing practice-based research, to identify barriers and facilitators of capturing practice, and to inform future studies with lessons learned from our efforts.
Were the group interventions effective?
Overall, about half of all children who participated in the group language interventions at this clinic demonstrated a clinically meaningful change in communicative participation from pre-intervention to post-intervention. These children all had change scores well above what would be expected for a waiting or baseline period (Thomas-Stonell, Washington, et al., 2013). Interestingly, very few participants demonstrated possible meaningful clinical change. Participants seemed to either respond well to intervention or not. While we cannot conclude that the group interventions are or are not effective for all participants, we can determine that they are effective interventions for some participants. All three of the groups showed similar proportions of children (∼50% in each group) who demonstrated clinically meaningful change, suggesting that all of the interventions evaluated in this study showed similar levels of effectiveness. Of note is that the variability in change in FOCUS score was quite large in all three groups, potentially influencing the lack of difference in change in FOCUS scores between groups.
What predicted change during the interventions?
Contrary to our expectations based on the literature and clinical judgment, our regression analysis did not identify any significant predictors of change in communicative participation during these interventions. That is, change was not predicted by intervention group, attendance, age pre-intervention, functional communication ability pre-intervention, or time between FOCUS scores. Therefore, our analysis did not provide any information about what variables do predict change during these interventions. It is possible that the variables we examined are indeed not significant predictors of change in communicative participation during intervention, which is instead predicted by other variables we did not measure or analyze. We included variables that had full or near full sets of data and that had scientific support as predictors of response to intervention or growth, but, in theory, there were certainly other variables that could have been included. For example, the child’s CFCS classification was used as a predictor of communication level pre-intervention. We decided to use this variable because it was reliably collected, it is not directly related to any child’s FOCUS score, and it prevented potential issues of ceiling effects that may have occurred with the initial FOCUS score, while still providing a measure of communicative ability. There may be other measures of pre-intervention communication ability, or language ability specifically, that might predict a child’s change in group language intervention. Different constructs may have different predictive value of change in communicative participation during intervention. Perhaps, functional communication ability does not predict change during a period of intervention, but a measure of, for example, expressive language form, might.
Another possibility is that the intervention was simply not sufficient for change for some participants. Because only half of the sample demonstrated clinically meaningful change in their communicative participation, it may be difficult to determine what predicts gains in communicative participation. The range of change in FOCUS score (−40 to 188) was very large. This may have impacted the results of the regression analyses. Perhaps in different ranges of change, different variables are predicting that change. If this were the case, a range this size might make it difficult to identify the predictors of change across the entire range as opposed to within smaller ranges. For example, predictors of negative change may be different than predictors of positive change. In another example, small changes in the Not Meaningful Clinical Change range may be predicted by different variables than changes in the Meaningful Clinical Change range. These questions are worth pursuing in future studies.
Based on the available predictors, the models presented in this study did not predict any variance in the change in communicative participation experienced during a period of intervention, but that is not to say that the overall constructs being investigated do not play a role. It may simply be that we were limited by the way in which we were able to measure the constructs.
Challenges and considerations moving forward
We acknowledge several limitations in our effort to gather evidence to answer our research questions, which both lend some caution to the strength of our conclusions and highlight some challenges in trying to capture practice retrospectively. The data were collected across different time points for each child. Due to the short time spent waiting for intervention for most children in the study and the PSL program guidelines, there were no children who had the FOCUS completed both at the initial assessment and at the start of intervention, if they were not the same time. Additionally, the amount of time between FOCUS scores varied quite dramatically across participants. The time between FOCUS scores used as the pre-intervention and post-intervention time points ranged from about 1 to 7 months. While some children had the FOCUS completed at the initial assessment and the end of intervention, others had the FOCUS completed at the start of intervention and at their consolidation appointment. Many of these inconsistencies stem from families missing appointments or forgetting to fill out the outcome measure forms when they attended appointments. With more consistent data collection time points, including at baseline, pre-intervention, post-intervention, and after a consolidation period, more information would be available about the effectiveness of the intervention programs. For example, more consistent, longitudinal data would allow for analysis of gain that occurs during the program and gain that occurs in the consolidation period that follows an intervention, in relation to gain that occurs in the wait time before an intervention.
Our use of a communicative participation measure to evaluate gain may have also impacted the results of our predictor analysis. Research investigating the effectiveness of language intervention has used a range of language measures (e.g., standardized tests, developmental scales, language sample analysis, specific language skills, and speech sound indices, to name a few) but have rarely included communicative participation measures (Cunningham et al., 2019; Kwok et al., 2019). Interventions that show demonstrable change in one language measure may not necessarily show the same effects on other methods of language measurement. In the same way, predictors of one outcome measure may not predict others. Dempsey and Skarakis-Doyle (2010) highlighted the value in assessing the impact of a language intervention on children’s ability to use language in their daily life and experiences, and parents indicate that such outcomes are most important and meaningful to them (Roulstone et al., 2012). Therefore, the selection of an outcome measure focusing on communicative participation by the PSL program seems justified. These outcome measures assess important and relevant communicative abilities, particularly given that one of the goals of intervention is to improve one’s ability to functionally use language in day to day life (Eadie et al., 2006). Clearly, more research is needed to understand the relations between factors that predict growth in language versus growth in communicative participation during early language interventions.
As is to be expected, there were also challenges in carrying out this type of study. While the provincial mandate provides outcome measurement tools, various circumstances led to these outcome measures not being available for all participants. These outcome measurement tools are completed by parents and, by clinician report, factors such as attendance, forgetfulness, distraction, and motivation, can influence the completeness of data collection. This issue of missing outcome measure data is not restricted to this community clinic but is known to affect the provincial program more broadly. In addition to the way in which outcomes were measured, the mandated timing of measuring outcomes can be easily impacted by real-life practicalities such as missed appointments. In the context of these real-world challenges, attempting to measure a construct as specific as response to intervention becomes next to impossible with the variation in timing that occurs naturally in clinical settings. Finally, the lack of information on the nature of each child’s language difficulties, beyond knowing that their expressive language difficulties identified by SLPs qualified the children for the intervention groups, makes it challenging to make more specific determinations about change occurring during intervention. There are no consistent procedures for identification of language difficulties and disorders in the PSL Program or in this community clinic and, as is common, individual SLPs use the best available evidence and their clinical judgement on an individual case basis. Examples of tools or procedures that SLPs working in the PSL Program might use in making these determinations include the Clinical Evaluation of Language Fundamentals-Preschool (CELF-P; CELF-P2; Wiig, Secord, & Semel, 1992, 2004), the Preschool Language Scales (PLS-4; PLS-5; Zimmerman, Steiner, & Evatt Pond, 2002, 2011), observation and interaction with the child, and interviewing the parent. While each of these procedures can provide valuable information in an assessment about a child’s language, the lack of consistent guidelines can be challenging in systematically tracking the language growth of children receiving intervention. These challenges can impact the ease with which community-based research is carried out and thoughtful, workable solutions to challenges that arise is an important skill in engaging in practice-based research.
The value of capturing practice through retrospective chart review
Based on our experience, we would argue that our attempt to capture real-world clinical practice was indeed worth the effort. However, we must admit that the picture we obtained was not as complete as what we originally expected. Perhaps one of the biggest accomplishments of this study was the ability to collect information about the effectiveness of three group interventions being offered in a sample of children who were receiving publicly funded intervention in a real-world community clinic.
We were able to determine that about half of the children with outcome measures recorded pre- and post-intervention showed clinically meaningful change in their communicative participation ability following these three interventions. The groups were relatively short in duration (3–7 sessions). A short group intervention that results in communicative participation improvements for half of the children may be a useful entry point. Programs like these could very well reduce the number of young children who will enter a more intensive intervention program. An additional consideration is that the full impact of intervention on communicative participation may not occur immediately at post-intervention, but sometime later. If this is the case, the change in communicative participation from pre-intervention to post-intervention may not represent the full extent of the effect of the group interventions on communicative participation.
We also learned what variables did not appear to predict change in communicative participation during these interventions (i.e., attendance, communication ability pre-intervention, age). While it is possible that these variables do not predict change at all, it is also possible that they do not predict change in communicative participation, specifically. Nonetheless, if the ultimate goal of early language intervention is to improve how children are able to communicate in their daily lives, then it remains important for us to identify the factors that predict who does and does not benefit in this regard from the types of interventions being offered. The current results direct us to explore other possible predictors, as well as other outcome measures, in future research in order to better understand who is most likely to benefit from group language interventions. It remains worthwhile to explore these questions in a community-based research setting because of their potential impact on guiding real-world intervention decisions for young children and measuring the language and participation outcomes of various language interventions.
We also believe that our study was worth the effort from the perspective of how it informed us about the process of practice-based research that involves capturing practice through a retrospective chart review. Although our intervention-related questions were not fully answered by this study, engaging in this type of research-practice partnership provided valuable information about the facilitators and barriers associated with attempting to capture practice.
A number of the benefits we anticipated at the beginning of this collaboration were realized. The collaboration between clinicians and researchers developed very seamlessly. Because clinicians were engaging in their usual practice, it was a transition that required little extra work on their part. The clinicians in this collaboration were also highly motivated because the research questions being posed were of mutual interest. This type of study also ensured high external validity, a concern often described in applying research findings to clinical practice (Green & Nasser, 2017; Law et al., 2004). Intervention was not adjusted in any way, and the study sample was representative of the clinical population clinicians see in a real clinical setting. We can, therefore, determine that the intervention being provided was both realistic and feasible for clinicians. Additionally, due to the provincial mandates for measuring outcomes in the PSL Program, the outcome measurement tool can be used to assess functional language and communicative participation in children across a broad range of ages and abilities (Hidecker et al., 2017; Thomas-Stonell et al., 2012). Finally, comparatively speaking, this type of study was a time and cost-efficient place to begin practice-based research by capturing practice. The data collection phase of this study was completed over the course of 2 months, and the cost was limited to the time of the first author (RES) required to extract and analyze the data. There was no cost associated with recruitment, intervention delivery, or outcome measurement. These realizations support continuing engagement in community-based research. Partnerships between community organizations and research teams can bear fruitful and informative collaborative opportunities. This type of study fosters relationship growth between partners, makes research more accessible to and likely to reach clinicians, and encourages researchers to address important and relevant clinical questions in realistic ways.
As our practice-based research collaboration moves forward from capturing practice toward changing and creating practice (Vollebregt et al., 2018), several shared plans have been made to address the barriers experienced in this study. The final author (JOC) is involved in a series of program evaluation and quality improvement projects in partnership with the provincial ministry in an effort to (a) improve the implementation of the FOCUS and (b) evaluate the optimal timing of outcome measurement using the FOCUS. We also were able to identify additional data that have not been historically collected but are feasible and might be informative. This led to the collaborative development of a checklist that has been implemented within the clinical setting to track potential predictors of growth in a period of intervention in an expanded, consistent, and clinically feasible way.
Practice-based research provides a unique and valuable opportunity for clinicians and researchers to partner and investigate mutually important research questions that drive clinical practice in a realistic way. Researchers are able to approach questions with a clinical perspective, ensuring that their research is more accessible and applicable to questions that clinicians may be asking. Clinicians, in turn, have input throughout the research process and have a voice in the utility and prioritization of recommendations being made. We are hopeful that the lessons we learned in this study will be useful in informing not only our own future efforts to conduct practice-based studies that capture practice, but those of other clinician–researcher partnerships in our field.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article
