Abstract
Check-in/Check-out (CICO) behavioral support has been implemented in Finnish School-Wide Positive Behavior Interventions and Support (SWPBIS) schools to cater to students who require personalized behavior support beyond the universal level. Previous studies have demonstrated the effectiveness of CICO as a behavioral support method. However, further research is needed to investigate its effectiveness in a larger sample and to analyze the timeline for behavioral change. This study focused on 51 elementary school students in Finland, assessing their behavior at baseline and during the CICO intervention phase using two data collection methods: the Daily Report Card (DRC) and the School Situation Questionnaire (SSQ). Nonlinear growth modeling was employed to examine the effects of the intervention. The results indicated that CICO yielded significant positive effects on behavior within 1 week of initiating support. After the outcomes stabilized, the behavior change remained stable beyond the first week of the intervention. These effects were detected in both the target behavior measured with DRC and the problem behavior measured with SSQ. These findings suggest that CICO interventions produce rapid and sustained changes in behavior. Further, the effects of CICO were observed in various settings within the school environment, indicating distal outcomes.
Keywords
Check-in/Check-out (CICO) serves as a Tier 2 intervention designed for students who exhibit an insufficient response to universal-level behavior support in schools (Bundock et al., 2020; Crone et al., 2010). The CICO intervention is frequently employed in schools implementing School-Wide Positive Behavior Interventions and Support (SWPBIS; Wolfe et al., 2016). The tier levels within SWPBIS offer a systematic framework for implementing interventions and providing support. The main goals of CICO are to prevent exacerbation and the long-term effects of problem behavior, to promote appropriate behavior during instruction, and to support positive social behaviors in different school contexts.
The CICO implementation should combine the necessary implementation components and core practice elements (Filter et al., 2022). The components of implementation include comprehensive training to effectively implement the intervention, data-driven decision-making, and ongoing support and feedback processes for the personnel involved in delivering the intervention. The intervention practice elements involve a structured daily check-in with a designated staff member, reminding students of their own goals, and receiving feedback throughout the day using the point card. At the end of the school day, students attend check-out meetings with the CICO coach, where they can reflect on their progress and receive reinforcement or rewards for their efforts. Parents and guardians play a crucial role in providing support and staying informed about a student’s progress and behavior in school (Filter et al., 2022).
In Finland, there is a comprehensive manual for CICO support that encompasses both theoretical principles and practical strategies (Karhu et al., 2017). This contextually adapted CICO support adheres to the guidelines outlined in the manual developed by Crone et al. (2010) while also aligning with Finnish legislation (Basic Education Act, 2010) and the principles of support organization in Finnish primary schools (Finnish National Agency for Education, 2014). The CICO team, which includes the teacher coordinating CICO activities, is responsible for organizing the CICO support in school and overseeing the implementation of the elements of the support. The CICO support begins with a brief student-specific behavioral assessment akin to a modified functional behavior assessment, as proposed by Carr et al. (2002). During CICO support, each student in Finland receives individual behavioral goal(s) tailored to their classroom. Behavioral goals are set by the CICO team and classroom teacher together with the student, and aligned with universal behavior expectations.
Several studies have shown positive outcomes for CICO. These outcomes include reductions in problem behavior (Hawken et al., 2014), improvements in academic engagement and performance (Miller et al., 2015), and improved social skills (Karhu et al., 2020). Past literature suggests around 70% of students who completed CICO had a significant reduction in problem behavior (Filter et al., 2007; Hawken et al., 2007). However, subsequent meta-analyses and systematic reviews have provided a more mixed picture of the impact of CICO. Drevon et al. (2019) conducted a meta-analysis synthesizing the effect sizes from 32 CICO studies. Their findings indicated that CICO had significant effects on student behavior in both single-case and randomized control trials, surpassing one standard deviation (Drevon et al., 2019). According to a meta-study conducted by Park and Blair (2020), medium effect sizes were found in conventional group studies. Some group design studies have not shown equally strong evidence of the effects of CICO. Hawken et al. (2014) conducted a systematic review and discovered varying effect sizes for CICO, ranging from small to large. A systematic review by Maggin et al. (2015) reported that group-based studies did not yield any intervention effects, whereas Wolfe et al. (2016) showed that CICO has a minimal effect on promoting desirable or appropriate behavior and a limited positive impact on academic engagement. A student’s need for more intensive behavioral support could explain variations in the effects of CICO (Hawken et al., 2014). Students whose problem behavior is characterized by escape-maintained behaviors may also benefit less from the intervention. One potential source of variation may also arise from variability in intervention delivery, particularly in terms of fidelity and integrity (Filter et al., 2022).
Previous single-case studies have revealed that CICO can have immediate effects on decreased behavior after intervention initiation (Campbell & Anderson, 2011; March & Horner, 2002). CICO contains elements based on behavioral principles that can produce immediate effects, such as systematic and frequent feedback. Immediate effects can be achieved by investing in antecedents (Kern & Clemens, 2007), such as reminding students of behavioral targets and building a clear, predictable feedback system. Overall, goal setting and feedback systems are essential components of interventions for increasing socially appropriate behavior and improving academic learning skills (e.g., Pfiffner et al., 2016). However, the goals should be well formulated—that is, they should be specific, measurable, achievable, relevant, and time-bound (Wade, 2009). Feedback may result in rapid changes in behavior, as it is delivered regularly in situations where target behavior is an essential part of social and situational competence.
Although previous studies have demonstrated the effectiveness of CICO interventions, there remains a lack of studies with a large group of students investigating whether CICO interventions produce immediate effects and how strong these effects are immediately after the start of the interventions. The primary focus of this study was to analyze the aggregated outcomes of multiple individual cases of CICO support in SWPBIS schools. We examined how rapidly the effects of an intervention could be detected, including strength and stability effects. We investigated whether the intervention had an impact on both the target behaviors set for the intervention and the problem behavior.
Present Study
The present study aimed to investigate the timeliness of observing significant and meaningful changes in students’ behavior after the implementation of CICO support. The study assessed the impact of CICO support on both target behavior and problem behavior in various school contexts, including instructional lessons and other school situations. By employing a quasi-experimental time series design and nonlinear growth curve modeling, we addressed the following research questions:
Materials and Method
Participants
Fifty-one students and their teachers participated in this study. The students were from Grades 1 to 6, and they all followed the standard curriculum and were taught in the usual learning environments typical of each school (see Table 1). The 11 participating schools came from three different municipalities in Finland. Ten of the participating schools were situated in urban or suburban areas, and one was situated in a rural area. Inclusion criteria for participating in CICO support were that students had persistent and long-lasting problem behaviors hampering both learning and social coping. Problem behavior had to occur during class. The observed problem behaviors included non-engagement in academic tasks, noncompliance, talking out of turn, and impulsive behaviors (e.g., out-of-seat). The participants were mostly boys (86.5%). Participation in the study was voluntary, and consent was received from the students’ guardians and school personnel. The Ethical Committee of the University of Jyväskylä reviewed the study before the start of the research processes.
Grade-Level and Gender Distribution of Students in CICO Support.
Setting
In Finland, children begin school in the year they turn seven, and compulsory education lasts until the age of 18. From pre-primary education to a higher level, all education is free of charge. Basic education follows the national core curriculum (Finnish National Agency for Education, 2014), which constitutes the objectives and key criteria for local authorities to create local curricula. Finnish primary school teachers have university education and training (a master’s degree) and have pedagogical autonomy (Pesonen et al., 2015). Teachers are involved in creating local curricula and the necessary pedagogical documents for students with special education needs. Several regional areas in Finland are committed to using the SWPBIS approach as the umbrella for behavioral support at all tier levels.
Procedures
Training
All schools participated in CICO implementation training organized by the research team. In six schools, personnel were already familiar with CICO at the beginning of the study and had been using CICO for several years. The school personnel unfamiliar with CICO intervention attended two additional training sessions and were offered coaching in delivering and organizing CICO support in their own schools. Training and additional behavior support (CICO) for students were conducted during the 2019–2020 and 2020–2021 school years. Both training and study were part of a larger Erasmus research project (https://www.pbiseurope.org).
Organization of CICO in Schools
Each school had a CICO manual (Karhu et al., 2017) to guide intervention implementation. All schools formed a team to monitor the selection of students and intervention implementation. Behavioral goals could be set for each pupil one to three at a time. Each school had its own universal-level behavior expectations matrix and CICO teams monitored the alignment of CICO behavioral goals with these universal behavioral expectations. The goals school staff set for supporting students participating in CICO addressed three different areas: supporting learning-oriented behavior, promoting respectful behavior and language, and maintaining a positive classroom climate.
The researchers provided teachers with instructions on how to use the Daily Report Card (DRC) and encouraged teachers to give verbal feedback in addition to the DRC scores. Designated coaches provided daily check-in and check-out meetings. The CICO coaches were usually paraprofessional staff members. Together with the students’ teachers, the teams made decisions about when the CICO period would end. The duration of the CICO support period varied between 5 and 17 weeks. Some CICO processes took longer, mainly because school staff wanted to ensure sustained positive behavioral changes. Procedural fidelity of CICO was followed by a daily checklist that included five questions covering the daily components of CICO: use of DRC during the lessons, record of DRC scores, follow-up of DRC scores, guardians’ involvement (guardian signings of the DRC), and check-in and check-out. For each CICO support period, the proportion of the maximum fidelity score was calculated. On average, the fidelity of support was very high (91.56%, SD = 8.71%), and only four participants had lower than 80% fidelity (range 66%–77%). The fidelity of the universal-level SWPBIS of the 11 participating schools was assessed twice a year using the culturally modified Tiered Fidelity Assessment (Algozzine et al., 2019). The average fidelity of the universal-level SWPBIS was 78.8% (range 30%–96.7%; for three schools, fidelity was lower than 70%).
Measurements
The DRCs were used daily to track and monitor individual behavior and progress in individually set goals. The teachers gave DRC ratings and provided feedback at the end of each lesson. Target behaviors were assessed on a scale of 0–2 (0 = expectations not met, 1 = expectations partially met, and 2 = expectations fully met). The final data, collected with DRC, included daily scores converted into percentages. Repeated daily ratings were conducted at baseline and in the intervention phase. The last three daily baseline measurements and the first five consecutive weekly averages of the DRC of the intervention phase assessments were used for analysis. Behavioral change was assessed by the differences between the baseline and intervention measures. The DRCs are effective in monitoring improvements resulting from interventions (e.g., Jurbergs et al., 2007).
The School Situations Questionnaire (SSQ; Barkley, 1981) was used to collect information about a student’s behavior in various school-related situations. The SSQ was used to assess the change in a student’s problem behavior at baseline and in the intervention phase. The SSQ was not used to assess the achievement of student-specific behavioral goals but rather to assess the prevalence and frequency of problem behavior. The SSQ includes statements or questions about a student’s problem behavior in 12 common school situations; the frequency of the problem behavior in each situation is measured with a 0 to 9 Likert-type scale (0 = no problems; 9 = severe problems). During data collection, the COVID-19 pandemic impacted the organization of field trips and special assemblies in schools. Consequently, data related to these situations were excluded. Further, it is important to note transportation on buses to schools is not a common practice in Finland. As a result, we excluded the original SSQ item “while on the bus,”. The study utilized a total of nine items: five items describing classroom situations and four items addressing other situations within the school environment.
The teachers completed the SSQ assessment at the end of each week. Repeated weekly SSQ ratings were conducted at baseline and in the intervention phase. Three baseline measures and five consecutive weekly measures of the intervention phase were used for analysis. Cronbach’s alpha has demonstrated good internal consistency for the SSQ in school-age samples, with alpha coefficients ranging from 0.84 to 0.91 (Pelletier et al., 2006). In our current study, Cronbach’s alphas were 0.91 for classroom situations and 0.80 for other situations within the school environment.
Statistical Analyses
All analyses were performed using the Mplus statistical program, version 8.2 (Muthén & Muthén, 1998–2017). A latent nonlinear growth model with two latent factors for the outcome measures was employed to characterize the impact of the intervention on DRC and SSQ measures. Each individual’s growth curve during the intervention exhibited specific DRC and SSQ scores, with individual changes occurring in a nonlinear fashion. The presumed nonlinear shape is intended to accommodate all individuals, although the rate of the nonlinear growth is expected to vary between individuals. The estimation of the nonlinear growth curve model yields mean values and variances for both the level and the nonlinear change factors (Grimm et al., 2011).
In the model, the last three daily baseline measurements and the first five consecutive weekly averages of the DRC of intervention phase assessments were used. The daily measurement had to consist of at least two evaluated hours of school. Similarly, for the SSQ, three baseline measures and five consecutive weekly measures of the intervention phase were used. Missing information was random and mainly due to a student’s absence from school. To estimate the change, the factor loadings for the slope were fixed as zero for the first baseline measurement and one for the last measurement. Examination of changes in factor loadings could be carried out between each time point. This allows for the examination of changes between the baseline and intervention phases as well as within each phase.
All loadings of the intercept factor were fixed at 1. The Bayes estimation method with 10,000 iterations was used. The model tests overall fit with a posterior predictive p-value, where p values <.05 indicate that the model does not fit the data. For the parameters, the estimated model produced a one-way p value and a 95% Bayesian confidence interval. Confidence intervals were calculated for the growth parameters to estimate whether they were significantly different from 0 (starting point), and this information was used to determine the time point when the change was statistically significant. In addition, if the 95% confidence intervals of successive estimates do not overlap, this indicates a significant difference between the estimates.
The effect sizes during the process were calculated by comparing the change to the first baseline measure divided by the pooled standard deviation of the three baseline measures. The effect size of 0.20 is small, 0.50 is medium, and 0.80 is large (Cohen, 1992).
Results
The nonlinear growth model (see Table 1) fitted the DRC data (posterior predictive p = .365). No trends or changes in behavior evaluations could be detected during the baseline phase. The estimates and confidence intervals showed that a statistically significant change in DRC evaluations appeared immediately in the first intervention week (Slope Intervention 1; see Table 1). The following intervention weeks’ confidence intervals for the slope factor parameters were significantly different from the baseline but overlapped, indicating that there was no statistically significant change in behavior evaluations between the first and fifth intervention weeks. This result indicates that at the within-group level, the intervention reached an immediate change in positive target behavior and remained stable thereafter.
The nonlinear growth model with SSQ classroom (Posterior Predictive p = .091; see Table 2) and SSQ other situations data (Posterior Predictive p = .384; see Table 3) fit the data equally well. No trend could be observed for either SSQ subscale at baseline. For the SSQ classroom scale, a statistically significant behavior change was detected immediately after the first intervention week (Slope Intervention 1; see Table 2). The parameter estimates for slope increased during the intervention, but this change was not statistically significant. This also means that the immediate change from the baseline was maintained during the intervention.
Nonlinear Growth Model for Daily Report Card: Parameter Estimates, p Value, and 95% Confidence Interval.
Parameters are fixed in the initial nonlinear growth model.
Nonlinear Growth Model for School Situations Questionnaire Classroom: Parameter Estimates, p-Value, and 95% Confidence Interval.
Parameters are fixed in the initial nonlinear growth model.
The results of the SSQ other situations showed similar rapid changes in problem behavior. An examination of the confidence intervals of the factor loadings of the slope showed that a significant change in behavior occurred immediately during the first week of the intervention. Compared with the other two indicators, more fluctuation could be detected between the different measurements of SSQ other situations (see Table 3). The values of the estimates for factor loadings increased during the intervention but decreased in the last measurement. However, confidence intervals for factor loadings overlapped, indicating that the change was not statistically significant in behavior after the first intervention week.
The mean values and effect sizes based on the estimated model showed a large intervention effect, especially in target behavior (DRC; see Tables 4 and 5) and problem behavior in the classroom (SSQ classroom; see Tables 4 and 5). For the DRC, the intervention effect remained stable over the five consecutive measurement points, whereas the effect sizes of the SSQ classroom varied between −0.95 and −1.5. For SSQ other situations, the intervention effect was lower and varied between −0.38 and−0.43 (see Tables 4 and 5).
Nonlinear Growth Model for School Situations Questionnaire Other Situations: Parameter Estimates, p-Value, and 95% Confidence Interval.
Parameters are fixed in the initial nonlinear growth model.
Estimated Mean Values and Within-Group Effect Sizes for Intervention Compared With Baseline Standard Deviation.
Note. SSQ refers to School Situations Questionnaire and DRC to Daily Report card. Baseline 1-3 refers to weeks of baseline assessment. Intervention 1-5 refers to intervention weeks. The pooled standard deviations of the three baseline measures were 18.77 for the DRC. 1.46 for the SSQ classroom and 2.00 for the SSQ other situations.
Discussion
The current research study examined the immediacy of changes in students’ behaviors following the introduction of CICO support in school and how large the effect sizes of the changes in behavior were. The results of the study showed that behavioral changes, both in behavioral goals and in problem behaviors, were observed rapidly after intervention implementation began. The change in behavior during lessons and other situations in school was detectable about a week after the start of CICO support. According to teacher assessments, the positive effects of CICO support were also immediately (after the first week) seen in other school situations. The observed change in the other situations was not as large as in the classroom situations. Given that the behavioral objectives are specifically tied to classroom activities, it is conceivable these findings imply the intervention’s effects may not generalize as robustly to other contexts. It is also possible teachers have fewer chances of observing behavior outside the classroom; therefore, information regarding changes in behavior outside of the classroom reach teachers gradually.
This study confirms the assumption that CICO can result in an immediate reduction in problem behavior (e.g., Campbell & Anderson, 2011). Accordingly, high-fidelity CICO interventions covering the necessary implementation and practice elements should quickly result in behavioral change. Feedback given by teachers and monitoring of the DRC are essential to achieving the impact of interventions. In addition, antecedents and anticipatory measures can serve as elements of intervention and positively influence behavior. Overall, the development of a comprehensive, individualized intervention plan for the student addressing the antecedent and consequence variables maintaining the student’s problem behavior may be critical to achieving immediate behavior change (March & Horner, 2002). Thus, for example, pre-established goals and giving proactive attention while reminding students of their goals can be a factor in behavioral change in an intervention (Kelly et al., 2015; McComas et al., 2003). Further, a clear and predictable structure for support and feedback may increase socially appropriate behaviors (Pfiffner et al., 2016).
In previous research, the effects of CICO support have varied widely; some studies have shown large effects (Drevon et al., 2019), while in other studies, the results have indicated much smaller effects (Hawken et al., 2014; Park & Blair, 2020) or showed no effects at the group level (Maggin et al., 2015). In this study, the effect sizes were large for changes in target behavior and problem behavior in the classroom, indicating that in Finnish SWPBIS schools, CICO is an effective and efficient method for goal-directed modification of student behavior. The effects stayed steadily higher than baseline during the next intervention period, and no trend was detected. It is important to note that the 5-week follow-up period was relatively short. In general, there is insufficient research on the maintenance of behavior change after the intervention is completely removed to conclusively determine the long-term effects of the CICO intervention (Mitchell et al., 2017). While the immediate effects of behavioral support, such as CICO, may manifest quickly, it may require more time for sustainable change and the consolidation of learned skills. The environment plays a crucial role in this process. It is essential to have clear behavioral expectations in place, fostering a supportive school environment (Park & Blair, 2020). Providing positive feedback and praise is imperative for promoting sustainable change in behavior.
As the standard deviation of the slope in the model indicated, there was great variation between students in behavioral change during CICO support. The intervention was beneficial for at least 70% of the students (decrease in problem behavior in the classroom and increase in target behavior). These findings align with the figures provided earlier (Filter et al., 2007; Hawken et al., 2007). Research into this variation in response to intervention has mainly focused on the severity of the difficulties or individual characteristics of the individuals participating in the interventions, but the findings are inconsistent (Drevon et al., 2019; Park & Blair, 2020). Adult-specific quality of interaction, which is independent of the fidelity of the implementation can also contribute to varied intervention effects; therefore, future research on interaction quality in the context of CICO should be conducted.
One source of variation in the intervention effects may result from the different functions of the problem behaviors; CICO may be less effective when applied to students whose problem behavior is perpetuated by avoidance of participation in educational activities (March & Horner, 2002). Some promising results have been achieved with functionally modified CICO, which aims to address behaviors maintained by behavioral functions other than attention (Klingbeil et al., 2019). Future CICO research in Finland should also look more closely at the functions of behavior to see what effect it has on the intervention effect.
In the present study, the baseline assessment showed, on average, a high level of problem behavior in classroom situations (score 5.44 on a scale of 0–9). In some cases, social skill deficits may hinder progress in behavior change and interaction in social settings and may override the effects of positive feedback received (Gresham et al., 2010). Such students may require modified CICO support (e.g., Cheney et al., 2010; Karhu et al., 2020). Among students, additional psychosocial or behavioral parent training may support behavior change and students’ well-being. Future research should include the development and evaluation of a modified version of CICO support for children with more severe behavior problems.
Although a relationship was detected between the commencement of intervention and behavior, and effect sizes were large for classroom situations, a larger number of participants and comparison between the treatment and control groups would have provided a more convincing basis for assessing the actual effects. This is one of the limitations of the study. Second, while the measures used in the study (DRC and SSQ) were deemed reliable and consistent assessment methods, it is worth noting that both assessments were conducted by a teacher who was also involved in implementing the support. Therefore, the data used rely on teacher perceptions and falls short of the rigorous standards upheld by direct observations. We used a self-report of approach because the objective of the larger project from which the data were drawn had the aim of scaling up CICO intervention with schools’ normal resources. Using observations would have been laborious and resource-intensive, given the number of schools and interventions.
In sum, the results of this study indicate large and immediate effects for CICO support at the within-group level. We believe the use of group-level data and nonlinear growth modeling offered a good opportunity to add to the extant evidence of the effectiveness of individual behavior support traditionally studied only with single-case analyses. The major implication of these findings is that CICO interventions can be implemented with schools’ regular resources and scaled up to support students in the Finnish education context.
Footnotes
Authors’ Note
Location of the study: Finland.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
This study was funded by the European Commission (Erasmus+ nos. 606687).
Ethical Approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Informed Consent
Written informed consent was obtained from all legal guardians of the children and from the teachers included in the study.