Abstract
Although evidence-based practices for improving academic engagement for students with emotional and/or behavioral disorders (EBD) have been identified, many teachers do not implement these practices with optimal fidelity. Thus, effective strategies are needed to improve teacher fidelity. Performance feedback is an effective professional development strategy, but it is unclear whether ancillary strategies like goal setting might further improve fidelity. In this study, we evaluated the effects of email performance feedback with and without goal setting on teacher implementation of opportunities to respond and behavior-specific praise using a multiple probe design. Participants were four general and special educators at a U.S. alternative school for students with EBD. Results indicated that a combination of written performance feedback and goal setting was effective, and that performance feedback alone was also effective. We were unable to determine whether the goal-setting component strengthened the performance feedback. Teachers provided positive feedback about the utility and feasibility of performance feedback with goal setting.
Keywords
Teacher implementation of evidence-based practices (EBPs) is critical for student success (Simonsen et al., 2008), yet a gap between research and practice remains. Specifically, although several EBPs promote student engagement and positive behavior, many classroom teachers lack knowledge and skills related to these practices (Stough & Montague, 2015). Examples of these underutilized practices include contingent praise and increased opportunities to respond (OTR; Simonsen et al., 2008). In a review of state policy regarding teacher training in classroom management, Freeman et al. (2014) found that, overall, preservice teachers may not be prepared to manage student behavior effectively. More concerning, although nearly 96% of elementary and secondary education programs include classroom management content, only 66% provide instruction on evidence-based classroom management practices. These problems are only exacerbated by having less experienced or uncertified teachers, which is often the case for students with emotional and/or behavioral disorders (EBD; Gage et al., 2017).
Special educators serving students with EBD were more likely to report intent to leave when their workload was high and resources were low (E. Bettini et al., 2020). In a survey by Cancio and colleagues (2013), EBD educators who remained in the field were more likely to express administrative support for professional development. In addition, special educators in self-contained settings, where approximately 12.3% of students with EBD are placed (U.S. Department of Education et al., 2022), were more likely to report self-efficacy in using effective instructional practices when they reported supportive working conditions (Cumming et al., 2021). Providing learning opportunities, specifically professional development related to implementing EBPs, could help self-contained educators teach more effectively (E. A. Bettini et al., 2016, 2017). Furthermore, State et al. (2019) presented components of effective educator professional development to support students with EBPs. They recommended using a combination of explicit training in EBPs with intensive and ongoing professional development.
Classroom Management for Students With EBD
Classroom management is a significant challenge for most teachers. It is even more challenging for teachers of students with EBD who report managing challenging behavior as the most difficult part of their job (Lambert et al., 2009). Often students with EBD exhibit aggressive behaviors, and unfortunately, the interactions between these students and their teachers are seldom positive. Moreover, researchers found that a higher praise-to-reprimand ratio was needed for students at risk for EBD compared with their peers (Caldarella et al., 2019). However, students with EBD often receive less teacher praise and more teacher reprimands (Rathel et al., 2014).
Teacher attention can be a powerful reinforcer; yet, if attention is used only for inappropriate behavior, the effects can be detrimental to student development and achievement. Two approaches that have been shown to increase appropriate behaviors and decrease inappropriate behaviors are behavior-specific praise (BSP) and providing more OTR. With consistent implementation of these two strategies, teachers can establish a predictable, positive, and engaging environment for students (Simonsen et al., 2020).
Behavior-Specific Praise
Behavior-specific praise is an EBP that helps teachers maintain a positive and productive learning environment (Simonsen et al., 2008) and is a simple yet effective strategy to decrease undesirable behavior and encourage appropriate behavior (Royer et al., 2019). Behavior-specific praise pairs a positive statement (e.g., “great job!”) with a behavior-specific statement (e.g., “following directions”) immediately after the student demonstrates the desired behavior. Although researchers have widely evaluated the effectiveness across many settings and populations (Sweigart et al., 2016), overall rates of BSP in the classroom are low (Reinke et al., 2013). Ennis et al. (2020) identified coaching as an EBP for increasing BSP. In their review of 25 studies, they described coaching as, “any ongoing support to increase or maintain a teacher’s use of BSP” (Ennis et al., 2020, p. 14). In addition, Briere et al. (2015) found that new teachers praised more when they received in-school consultation that consisted of coaching and feedback. Allday et al. (2012) found general education teachers’ rates of BSP increased after training and email feedback were implemented. In addition, as rates of BSP increased, rates of on-task behavior for students with EBD also increased.
Opportunities to Respond
Opportunities to respond is another evidence-based classroom management strategy (Common et al., 2020). Opportunities to respond is used to promote student engagement and fluency in basic skills. Increased rates of OTR have been associated with improved academic behaviors in the classroom, such as higher rates of on-task behavior and participation (Wood et al., 2009) and increased academic engagement time (Adamson & Lewis, 2017). Increasing OTR requires teachers to (a) identify the content or skills to be targeted, (b) prepare an extensive set of questions or prompts that offer students practice with the material, and (c) lead lessons with high rates of questioning, rapid student responding, and immediate teacher feedback (Lane et al., 2015). Although there are recommendations for rates of OTR, optimal rates have not been conclusively identified in the research literature. Some recommendations include 3.5 OTR per minute (Stichter et al., 2009) and 4 to 6 OTR per minute (CEC, 1987, as cited in MacSuga-Gage & Simonsen, 2015). Unfortunately, OTR rates typically fall below these recommended rates (Stichter et al., 2009).
Performance Feedback With Teachers
Performance feedback—an effective way to improve teachers’ skills (e.g., Cavanaugh, 2013; Fallon et al., 2015)—is “monitoring a behavior that is the focus of concern and providing feedback to the individual regarding that behavior” (Noell et al., 2005, p. 88). Performance feedback has been shown to increase teachers’ implementation of a range of EBPs (Cavanaugh, 2013), including specific praise rates (Sweigart et al., 2016).
One barrier to effective coaching and development for teachers is time, as special educators who work with students with EBD need sufficient time to plan effective instruction (E. Bettini et al., 2015). Therefore, school districts and teacher educators need efficient and inexpensive methods for providing performance feedback to teachers on classroom management strategies. Simonsen et al. (2017) found teachers’ rates of BSP increased when they received targeted professional development, including written email feedback, and when teachers employed a self-management strategy. Although this strategy required minimal time, teachers’ BSP rates were not maintained. Furthermore, Gage et al. (2017) used written email feedback to increase teachers’ rates of specific praise. Researchers found that with targeted professional development, including written email feedback delivered by the researcher, teachers were able to increase their rates of specific praise and sustain those increases 3 months after the last written email. Email feedback included a description and graph of the teacher’s progress as well as a reference to a performance goal set during training. Results indicate that with written email performance feedback, teachers can improve and maintain their implementation of classroom management strategies. Although these results are promising, little is known about the effect of written email feedback with goal setting on classroom management of students with EBD in an alternative setting.
Goal Setting With Performance Feedback
Extensive research on the positive effects of goal setting on student achievement exists (e.g., Rowe et al., 2017). However, there is a gap in the literature about the effects of goal setting on teacher performance. In a recent review of research related to goal setting with performance feedback, Criss et al. (2022) found that goal setting with performance feedback is a highly effective combination of training strategies for increasing certain types of behavior, such as BSP and OTR. In a recent meta-analysis of performance feedback in organizational settings, researchers found that the combination of feedback and goal setting produced the largest effect size compared with other feedback combinations, such as feedback alone or feedback with antecedent and consequence combinations (Sleiman et al., 2020).
A goal-setting component has been added to performance feedback treatment packages and resulted in increased teacher performance in the rate of BSP (Briere et al., 2015; Gage et al., 2017; Simonsen et al., 2017), student engagement (Gage et al., 2017), and social interactions and treatment integrity (Artman-Meeker et al., 2014). Several components of goal setting impact its effectiveness, such as the person responsible for setting the goal, the process for measuring progress toward meeting goals, and how and when goals are adjusted during treatment. Criss et al. (2022) found when teachers were responsible for setting the goal, outcomes were better. Similarly, teacher outcomes were better when goals were referenced during performance feedback than when goals were not discussed after the initial goal-setting procedures. These results indicate that (a) teachers should be involved in goal setting, and (b) goals should be discussed throughout the performance feedback treatment.
Purpose
Goal setting has been clearly shown to be effective for student achievement, but less is known about its role in teacher professional development. The purpose of this study was to determine the effects of objective and evaluative feedback with and without goal setting on teacher performance. Specifically, we sought to answer the following research questions:
Method
Setting and Participants
This study took place during the 2020–2021 school year in an alternative school serving students kindergarten through 12th grade. The school, located in a large Midwestern U.S. city, has a diverse student population (42% African American/non-Hispanic, 31% White, 3% Hispanic, and 24% multiracial). Of the 84 students enrolled in the school, nearly 98% of them qualify for free or reduced lunch. Students are placed at this private school either by parent choice or by their local school district due to the student’s behavior. Every student in the school receives specialized services through an Individualized Education Program (IEP). Students in the participants’ classrooms were classified with EBD, autism, or other health impairment (OHI).
To recruit potential participants, we discussed the research project with the building principal, and he presented the research opportunity to all teachers and provided consent forms to any interested teachers. Four teachers—including both special and general education teachers—returned signed consent forms. Participants who received the intervention were teachers who (a) taught at the alternative school, (b) had an interest in receiving performance feedback to improve classroom management skills, (c) currently held a teaching license, and (d) had low or inconsistent rates of BSP and/or OTR during baseline observations (i.e., below a rate of 1.0 per minute for BSP or 3.5 per minute for OTR). The participants did not receive any incentive to participate other than performance feedback on their classroom management practices. All four participants were White teachers with fewer than 12 years of teaching experience.
The first participant, Kendall, was in her 11th year of teaching and 4th year of teaching at this school. Kendall was 41 years old at the time of the study, and she holds a bachelor’s degree in special education and a master’s degree in leadership and administration. At the time of the study, she was pursuing a second master’s degree in instructional design. Kendall’s observations occurred during a middle school English language arts (ELA) class. Three students were enrolled in the class.
The second participant, Walter, was in his second year of teaching with 1 year of experience at the current school and was 23 years old. His undergraduate major was in social studies. Walter’s observations took place during a high school government class with four students.
The third participant, Michelle, was in her fifth year of teaching and her first year at the alternative school. She held an undergraduate degree in special education and was 39 years old. Her observations took place during math instruction in elementary school. The seven students in her class ranged from first to fifth grade.
The final participant, Mallory, was in her fourth year of teaching, and all of her experience was at the alternative school. Her undergraduate and master’s degrees were both in education. Her observations took place during a high school algebra class with five students. Mallory was 26 years old at the time of the study.
Dependent Variables
The two dependent variables measured in this study were rate of BSP and rate of OTR. Both dependent variables were measured using event recording and reported as a rate (total number of BSP or OTR divided by the number of session minutes) and were measured in person and/or by watching a video of the teacher.
Behavior-specific praise was defined as an affirmative statement that referred to and mentioned a specific behavior or performance. Instances of specific praise were counted each time the teacher provided specific positive feedback on the individual child’s correct or appropriate response. Examples of BSP included “Great job following my directions” and “Thank you for cleaning off your desk.” Specific praise was also counted if the participant praised the entire group (e.g., “Wow! I see everyone is ready to begin reading.”). If the participant only provided a general praise statement (e.g., “Great work!”), the statement was not counted as BSP. Moreover, if the teacher said a behavior narration statement such as “I see you are ready to learn” without the praise statement, the statement did not count as BSP. Student behaviors that received specific praise included (a) appropriate social skills (e.g., helping a peer), (b) appropriate classroom behaviors (e.g., following directions), and (c) correct academic responses (e.g., answering a question correctly).
The second dependent variable was rate of OTR, which was defined as the number of chances given to students to respond to instruction divided by the number of minutes in the session. This included opportunities to read aloud, questions directed to the whole group or individuals, and repeating and rephrasing questions. Nonverbal cues such as pointing or gesturing to a student were also counted. The OTR were recorded two ways: (a) whole group opportunities and (b) individual student opportunities. Both types of opportunities were coded as OTR. Observers used a data collection form for both dependent variables. The observers marked the instances of BSP and OTR as well as anecdotal notes about the observation to be used in the performance feedback emails.
Procedures
Baseline
During the initial baseline, teachers were observed during their typical instruction 2 to 4 days a week. During these observations, the observer recorded the teachers’ rates of BSP and OTR. The observer’s only interaction with the teachers was to thank them for allowing them to observe.
Behavior Skills Training
During the training phase, teachers received training on how to deliver BSP and OTR. The training used instructions, modeling, rehearsal, and feedback (Kirkpatrick et al., 2019). The interventionist delivered the training one-on-one using a PowerPoint presentation that explained the evidence supporting the use of BSP and OTR in the classroom, specifically for students with emotional and behavioral challenges. The training also shared grade-level appropriate strategies to increase BSP and OTR in the classroom. The suggested strategies included using response cards, guided notes, the use of technology (e.g., Kahoot), and partner work. Next, the trainer provided instructions and modeled how BSP and OTR are implemented in the classroom. Finally, the teacher practiced the skill, and the researcher provided feedback on their performance. Each training session lasted 20 to 30 min, and teachers practiced until mastery criteria were met. Mastery was defined as three consecutive appropriate uses of BSP and OTR during the rehearsal phase of behavioral skills training. For example, during the role-play, the teacher needed to deliver three correct BSP statements after giving a direction to the interventionist (acting as the student) to meet mastery criteria. If the teacher did not correctly deliver BSP statements, the interventionist would provide feedback and give the teacher another opportunity until mastery was met. All four participants met mastery criteria during the first training session. Once the training was completed, teachers entered the intervention phase. Two of the four trainings were observed by a second observer, and treatment integrity data were collected for training sessions.
Performance Feedback With Goal Setting
During the performance feedback with goal-setting intervention condition, teachers continued to provide instruction to students in their regular classroom. However, feedback was provided on their rates of BSP and OTR by email immediately after their observation. Similar to the format of observations in baseline, the observers recorded BSP and OTR. Some sessions were video recorded for interobserver agreement (IOA). The observations occurred for 15 minutes, and feedback was sent to the teacher before the end of the school day. During the observations, the observer did not interact with the teacher other than confirmation that the previous observation’s email was reviewed. Email feedback included (a) reminder that feedback was for BSP and OTR, including definitions; (b) examples of correct delivery of BSP and OTR and missed opportunities from observation; (c) description and graph of current performance on BSP and OTR; and (d) the teacher’s progress toward meeting goals. Figure 1 shows an example of a sample email. Teachers were instructed to confirm they had read the email either by responding to the email or by confirming in person prior to the following observation. If the teacher responded to the email with a question about the feedback, they were given a copy of the PowerPoint training presentation and referred to the written email feedback. No other information or coaching was given to teachers during the performance feedback phase.
Example of Performance Feedback Email With Goal for OTR.
During the first feedback email after the training, the teachers were instructed to identify a goal for either BSP or OTR. The first participant was randomly chosen to identify a goal for OTR, and every subsequent participant alternated between goal setting for BSP and OTR. Three teachers identified their goals after one observation, and one teacher participant, Kendall, identified her goal after two observations with performance feedback. Teachers identified their own goals but were given training materials that included examples of optimal rates for both BSP and OTR. During written email feedback, a goal line was added to the graph for either BSP or OTR, and progress toward meeting that goal was referenced in the feedback.
Once the teacher’s performance goal was met and maintained for four consecutive observations and data were on an upward trend, the teacher then moved into a maintenance phase.
Performance Feedback Without Goal Setting
Simultaneously with the performance feedback with goal setting condition, teachers also received performance feedback on the second dependent variable. Similar to the goal-setting condition, teachers received email feedback on their performance after each observation; however, the graph and narrative in the email did not include any discussion of or reference to a goal. Instead, the teacher’s progress in relation to the previous observation was included. For example, if the teacher received performance feedback and goal setting for their rate of BSP, in the same email, the teachers would receive performance feedback only for their rate of OTR.
Generalization and Maintenance
Generalization measures were taken during another class period with a different group of students. Generalization measures were taken during intervention and followed the same procedures as baseline with no performance feedback provided. For generalization observations, Kendall was observed teaching a different group of four middle school students for ELA. Walter’s generalization probes occurred during a different high school history class with two students. Michelle’s generalization observations occurred during science instruction with three fifth-grade students, and Mallory’s generalization observations took place during a one-on-one economics class with a high school student. Maintenance data were collected for Kendall at Weeks 1, 2, and 5 after the intervention and for Walter 1 week and 3 weeks after the intervention. Maintenance data were collected for Michelle 3 weeks after intervention. Maintenance data were not taken for Mallory because of schedule conflicts at the end of the school year. During the maintenance phase, the teachers were not given performance feedback on their rates of OTR and BSP, nor were they given feedback relative to their individual goals set during the intervention phase.
Interobserver Agreement
A second observer was present for 30% of sessions across baseline and intervention. A graduate student was trained on IOA by reviewing the operational definition of BSP and OTR, viewing videos, and participating in several practice opportunities that ended with the observer and the primary data collector above 90% IOA on three consecutive observations. Interobserver agreement observations occurred both through in-person observations and by watching video-recorded classroom observations. Total count IOA was used to calculate agreement between observers using the following formula: smaller count divided by larger count multiplied by 100. During baseline, IOA was 96% (range: 67%–100%) for BSP and 88% (78%–100%) for OTR. During intervention, IOA was 96% (80%–100%) for BSP and 93% (76%–100%) for OTR. Maintenance IOA was only collected for Kendall, and results were 92% (range: 90%–93%) for OTR and 100% for BSP.
Procedural Fidelity
Procedural fidelity data were collected on the training and written email performance feedback using a checklist. A graduate student completed all procedural fidelity for training and email feedback. A checklist was created and included every step of the training. A second observer observed Michelle’s training session, and Mallory’s training was recorded and reviewed by the observer after the training was completed. The observer marked on the checklist whether the interventionist followed the training procedure. Procedural fidelity for both trainings was 100%. Another checklist was created for the written email procedure. The graduate student was copied on email feedback for a randomly chosen 30% of emails across all participants. The observer was trained by reviewing examples and nonexamples of email performance feedback and marked whether each component of the email feedback protocol was included. Overall procedural fidelity across all participants was 100% for email feedback.
Social Validity
Teachers completed a questionnaire based on the Intervention Rating Profile-15 (IRP-15; Martens et al., 1985) to evaluate the social validity of the treatment package at the end of the study. The purpose of evaluating the social validity in studies is to determine whether the behavior changed in a positive and meaningful way (Cooper et al., 2020). The IRP-15 was designed to measure treatment acceptability of a student-focused behavior intervention (Martens et al., 1985). Similar to procedures used in Simonsen et al. (2017), researchers adapted each student-focused item on the IRP-15 to reflect the teacher-focused intervention targeting BSP and OTR. Teachers answered 15 questions related to the acceptability of the treatment package on a scale of 1 (strongly disagree) to 4 (strongly agree). See Table 1 for a list of the questions and responses.
Social Validity Results From Teacher Responses to Questions.
Note. Responses were scored on a scale of 1–4. OTR = opportunities to respond; 1 = strongly disagree; 4 = strongly agree.
Experimental Design
A multiple probe design (Cooper et al., 2020) was used to evaluate the effects of performance feedback with and without goal setting on classroom management skills. The two treatments being compared were (a) performance feedback with goal setting and (b) performance feedback without goal setting; these treatments were counterbalanced across participants. For example, the first participant was randomly selected to identify a goal for rate of OTR, and we counterbalanced by having the next participant identify a goal for rate of BSP.
All participants began baseline at the same time. After a minimum of five initial baseline sessions, following What Works Clearinghouse standards (Kratochwill et al., 2010), the first participant with the most stable and lowest response rates for both dependent variables began pre-intervention training. Once a steady increase in responding during intervention was observed, and a minimum of six data points were collected in intervention, all other participants were observed for an additional one to four sessions in baseline. Data were collected until all participants received training. All participants moved into the maintenance phase when at least four consecutive data points were collected at or above the goal, and both dependent variables were on an upward trend. The only exception to this rule was for Walter, who never reached his goal. The class period for observation sessions was kept consistent for each participant, except for generalization probes which occurred during different class periods. Each observation was 15 min and occurred during the first half of class. All interventions were implemented by the first author, a former special education teacher who was an advanced doctoral student in special education.
Data Analysis
Visual analysis was used to assess the level, trend, and variability of the data, overlap across baseline and intervention phases, immediacy of effects, and consistency across participants (Horner et al., 2005). To supplement visual analysis, percentage of nonoverlapping data (PND) was calculated. The PND is a statistic often used in single-case designs to interpret results. It is calculated by counting the number of intervention data points that are higher than the highest point in baseline, then dividing that number by the total intervention points (Scruggs & Mastropieri, 2013). The PNDs greater than 70% are considered large, between 50% and 70% are considered moderate, and below 50% are considered small (Maggin et al., 2019; Scruggs & Mastropieri, 2013; Scruggs et al., 1987). Furthermore, effect size was calculated using Tau-U, an effect size index that evaluates the proportion of data that improved over time between baseline and intervention phases after controlling for any trend in the baseline (Parker et al., 2011). Tau-U values were found using an online calculator (Vannest et al., 2016; http://www.singlecaseresearch.org/calculators/tau-u). Using the calculator, each teacher’s baseline data were contrasted with the intervention data phase to account for the observed overlap of multiple data points between the phases. First, we controlled for baseline trend by calculating the within-phase analysis of baseline only. Next, we calculated the effect size across the baseline and intervention phases, and finally, we calculated an omnibus value to estimate the average size of the effect across participants and phases for the study. Tau-U values greater than 0.90 are considered large, between 0.60 and 0.90 are considered moderate, and below 0.60 are considered small (Maggin et al., 2019; Scruggs & Mastropieri, 2013; Scruggs et al., 1987). Finally, the standard error (SE) for each phase comparison was calculated from the relevant standard deviation values.
Results
Teacher Implementation of BSP and OTR
Data for teacher implementation of BSP and OTR are presented in Figure 2 and Table 2. Below, we describe visual analysis of the data in terms of the effects of performance feedback compared with baseline, performance feedback plus goal setting compared with baseline, and performance feedback compared with performance feedback plus goal setting. Effects were demonstrated and replicated for both performance feedback conditions when contrasted to baseline. However, effects were inconsistent for performance feedback plus goal setting compared with baseline, with two teachers (i.e., Kendall and Mallory) demonstrating more pronounced effects with the combined intervention. Overall differences between baseline and intervention were moderate and statistically significant for both for OTR (Tau-U = 0.64) and BSP (Tau-U = 0.87). Magnitude and significance of participant-level effects are reported below.
Teacher Rate of BSP and OTR With Performance Feedback With and Without Goal Setting.
Summary of Teacher’s Rates of BSP and OTR.
Note. BSP = behavior-specific praise; DV = dependent variable; IV = independent variable; OTR = opportunities to respond; PND = percentage of nonoverlapping data; PF + GS = performance feedback with goal setting; PF only = performance feedback without goal setting.
Value is statistically significantly different than zero at p < .05.
Kendall
During baseline, Kendall’s mean score for OTR was 2.58 (range: 1.86–3.13) with a flat trend (−0.09; Tau-U = −0.03), and she did not give any BSP statements. Visual analysis of Kendall’s baseline data shows steady low rates of both OTR and BSP with some variability for OTR. Immediately after training, Kendall started receiving performance feedback, and her rates of both OTR and BSP remained at the same levels as baseline. After one session of receiving performance feedback, Kendall was randomly chosen to set a goal for OTR, and she set a goal of 3.5 per minute after the second observation the intervention phase. After goal setting, her OTR rate improved to 3.59 (range: 2.53–5.40) and was significantly different from baseline (0.12; Tau-U = 0.68), indicating that performance feedback and goal setting had an effect on Kendall’s levels of responding. Kendall met her goal rate of 3.5 after receiving performance feedback with goal setting for three sessions. During this phase, Kendall received performance feedback but did not set a goal for her BSP rate. Her rate of BSP improved from zero in baseline to 0.38 during intervention (range: 0.13–0.80); this was not a significant difference from baseline (0.03; Tau-U = 0.35). Kendall demonstrated some variability in her rates of BSP and OTR during the feedback condition, and her responding returned to an increased level toward the end of the phase. While performance feedback alone led to higher levels of BSP in the feedback phase compared with baseline, the combination of performance feedback and goal setting led to larger increases in Kendall’s OTR rate. Kendall’s PND for OTR was 62% and BSP was 60%, both indicating a moderate effect. During maintenance, her rate of OTR dropped to baseline levels (M = 2.65; range: 1.73–3.33). During maintenance, her mean rate of BSP decreased to 0.16 (range: 0.06–0.29), which was still increased from baseline levels of BSP. Generalization measures indicated rates of BSP and OTR were lower for other classes than for the class selected for intervention.
Walter
Following the second set of baseline probes, Walter was selected as the second participant to start intervention because his data were the most stable and demonstrated a decreasing trend. Visual inspection of Walter’s baseline data shows low and stable levels of BSP and some variable levels of responding for OTR. During baseline, Walter’s mean rate of BSP was 0.01 (range: 0–0.07), and OTR was 1.87 (range: 0.94–2.67), both of which were flat trends (0.00, Tau-U = 0.11; −0.09, Tau-U = −0.29, respectively). Walter received training and set a goal to increase his BSP. Treatments were counterbalanced across participants. As such, since Kendall was chosen to create a goal for her rate of OTR, Walter created a goal for his BSP rate to increase to 1.5 statements per minute. During intervention, Walter’s mean rate for BSP increased from 0.01 in baseline to 0.85 (range: 0.04–1.47) and was significantly different from baseline (0.03; Tau-U = 1.00). Walter did not meet his goal; his highest rate for BSP (1.47) occurred after 20 performance feedback emails. Walter’s change in levels of BSP rate indicated that goal setting and performance feedback increased his rate of BSP during the feedback condition compared with baseline, even though he did not accomplish his goal. He received only performance feedback for OTR and his rate increased to 3.82 (range: 2.73–5.33) which was also was significantly different from baseline (0.03; Tau-U = 0.99). During intervention, there was variability in both BSP and OTR, but both were at higher levels than during baseline. The positive effects detected for performance feedback were similar when compared with performance feedback and goal setting during intervention. Walter’s PND for OTR was 86% and BSP was 100%, both indicating large effects. During maintenance, Walter’s rate of OTR was similar to his rate during intervention (4.87); however, his rate of BSP was lower (0.27). Walter’s generalization measures were within the same range as his intervention class for OTR (3.07) and slightly lower for BSP (0.53).
Michelle
Visual inspection of Michelle’s baseline data shows low and stable rates of BSP and higher and variable levels of responding for OTR. During baseline, Michelle’s mean rate of BSP was 0.23 (range: 0.06–0.46) and mean rate of OTR was 3.83 (range: 2.40–5.93). During baseline, Michelle maintained a flat trend for BSP (0.02; Tau-U = 0.30) and OTR (0.03; Tau-U = 0.29). As the third participant, Michelle was selected to set a goal for OTR rate, and she made the goal of 4.0 OTR per minute. She achieved her goal on her third session in intervention and consistently stayed above that goal for nine consecutive sessions with a mean rate of 5.09 (range: 3.33–7.13) which shows a moderate difference from baseline (0.04; Tau-U = 0.67), only falling below her goal for one observation during intervention. During intervention, Michelle’s mean rate of BSP improved to 1.54 (range: 0.07–2.50) from the baseline mean of 0.23 with performance feedback which was significantly different from baseline (0.04; Tau-U = 1.00). Performance feedback alone, as well as performance feedback with goal setting, increased the frequency of her delivery of BSP and OTR. Michelle’s PND for OTR was 100%, indicating a large effect, and BSP was 18%, indicating a small effect. During maintenance, Michelle’s rate of BSP was 1.10 and fell within the range during intervention and her rate of OTR decreased to 3.50.
Mallory
Mallory’s baseline data showed overall high rates of OTR with variability and low yet stable rates of BSP. During baseline, Mallory’s mean rate of BSP was 0.09 with a flat trend (range: 0–0.14; 0.00; Tau-U = −0.02), and OTR was 4.82 (range: 3.23–6.73) also with a flat trend (0.19; Tau-U = 0.00). During intervention, Mallory’s OTR rate was 5.66 (range: 4.42–6.86) which demonstrated a small difference from baseline (−0.07; Tau-U = 0.20) with performance feedback. Mallory was selected to set a goal for BSP. She set a goal of 2.0 BSP per minute and met this goal on her third observation during intervention. Mallory’s overall BSP rate increased to 1.72 (range: 1.23–2.33) which was significantly different from baseline (0.07; Tau-U = 1.00) with performance feedback and goal setting. Mallory’s OTR rate during intervention when receiving performance feedback remained at high levels, similar to baseline. Performance feedback with goal setting had greater effects on her BSP rate than did performance feedback alone on her rate of OTR. Mallory’s PND for OTR was 27%, indicating a small effect, and BSP was 100%, indicating a large effect. Generalization measures were within her intervention range of OTR and were lower than baseline levels for BSP. Maintenance data were not collected for Mallory.
Social Validity
Overall, results from the social validity questionnaire were positive, indicating participating teachers were satisfied with the interventions. Mean scores for most questions were 3.5 or higher on a 4-point scale. The only question with responses in the disagree or somewhat disagree range was if teachers regularly used specific praise in the classroom before participating in the study (see Table 1). Furthermore, overall, teachers agreed that receiving feedback by email was an effective way to improve performance. In addition, all teachers stated they agreed or strongly agreed that goal setting was effective for improving their skills (M = 3.66); however, their mean scores were slightly lower (3.33) when asked whether they will use goal setting in their future professional development. Teachers also indicated that increasing their BSP and OTR rates improved student classroom performance.
Discussion
Although performance feedback has been shown to improve teacher performance in the classroom, there is limited research on the role of goal setting in performance management plans. The present study investigated the effects of goal setting and written performance feedback on teachers’ use of BSP and OTR. Results from this study showed that a combination of performance feedback and goal setting was effective, and performance feedback alone was also effective. While descriptively, some teachers made larger improvements when goal setting was included; our design does not allow us to draw causal conclusions regarding whether the combination of goal setting and performance feedback is more effective than performance feedback alone. All four teachers increased their rates of BSP and OTR after performance feedback was delivered with or without goal setting. Two teachers (i.e., Kendall and Mallory) demonstrated more pronounced improvement when goal setting was included alongside performance feedback.
Our results corroborate existing evidence that performance feedback increases frequency of teachers’ implementation of BSP and OTR. Teachers received training similar to participants in Allday et al. (2012); however, teachers in the present study did not consistently show improvements until performance feedback was implemented. These results were similar for two teacher participants, Kendall and Walter, whose rates of BSP were within baseline ranges until written performance feedback was implemented. This reinforces previous research that traditional “sit-and-get” professional development alone is not effective in changing teacher behavior (McLeskey & Waldron, 2002). Similar to Gage et al. (2017), training for BSP followed up with written performance feedback in the form of emails led to increases in teachers’ administration of BSP. Performance feedback alone led to increased levels of BSP and OTR rates for all four participants in the present study. These results align with the findings from Sweigart and colleagues’ (2016) and Cavanaugh’s (2013) reviews of the literature, indicating performance feedback is effective in increasing educators’ use of praise.
In addition, findings from this study show in order for a practice to be sustained over time and generalized to different settings, performance feedback should be provided. Generalization rates of both BSP and OTR for nearly all participants were lower than the rates observed during the class setting identified for intervention. Furthermore, rates during the maintenance phase decreased as time passed. This suggests teachers need regular feedback or may even benefit from a self-monitoring component to maintain performance (Freeman et al., 2018).
Second, this study provides initial evidence that pairing performance feedback with goal setting may further enhance teacher implementation in some situations but might not provide an additive effect in other situations. Although Allday et al. (2012) found teachers’ rates of BSP improved with written email feedback and goal setting, our study extends the research by measuring rates of two classroom management skills and examining the comparative effects of written performance feedback with and without goal setting. Performance feedback plus goal setting may have had better effects for Mallory and Kendall on the skills for which they set goals. Mallory’s rate of BSP was 0.09 before intervention and improved to 1.72 after goal setting and receiving performance feedback. During baseline, Kendall’s mean score for OTR was 2.58 and improved to 3.59 with goal setting and performance feedback. Although these findings are descriptive and do not allow us to draw causal conclusions, they suggest that adding goal setting might impact performance differently for different educators in different situations.
Our findings suggest it might have been more difficult to implement one practice compared with the other. Participants who set goals for OTR performance met their goals more quickly than those who set goals for BSP. Interestingly, one participant, Walter, did not meet his BSP goal, indicating the goal itself may not have impacted performance. Several plausible explanations may account for this. First, the format of Walter’s class was more discussion-based, specifically focused on historical and political issues. Observations took place during a time of extensive political turmoil, and the class had very different political opinions. Walter was careful not to praise students based on their opinions. This may have resulted in the overall lower rates of BSP for Walter. In addition, Walter was less likely to implement the suggested strategies provided in the performance feedback emails than other teachers. The other participant who set a goal for BSP rate, Mallory, followed up an opportunity to respond (e.g., “What is the coefficient?”) with specific praise (e.g., “Great job problem-solving!”). Since Mallory had relatively high levels of OTR throughout the intervention, she could leverage her strength in this classroom management skill to increase her implementation of BSP. This suggests goal setting for BSP can be more impactful when the teacher displays high levels of OTR with other classroom management techniques. Although Michelle did not set a goal for BSP, her rate increased after training and performance feedback. Much of her specific praise was related to behavior management (e.g., “Thank you for using kind words”) as opposed to specific academic behaviors or skills. Effects for Michelle might be related to the focus of elementary teachers on behavior management and could indicate that increasing rate of BSP for elementary teachers might be easier than for secondary teachers.
Teacher feedback for the intervention was positive overall but less enthusiastic about goal setting. Teachers reported they wanted feedback for their performance in the classroom, and all teacher participants indicated the training and feedback they received to deliver BSP and OTR was effective. Moreover, every teacher would recommend the procedures used in this study to increase BSP and OTR to other teachers. Although all teachers indicated goal setting improved their skills, they were less inclined to use goal setting in their future growth. We speculate the following possible reasons. First, teachers were not able to select the classroom management skill for which they received feedback or set goals. This may have impacted their desire to use goal setting in the future. Second, the mean number of days the teacher was observed and received performance feedback was 18 days. The length of the intervention may have exhausted the teachers and, therefore, decreased the investment in goal setting.
In addition, the teachers’ personality differences may have impacted the rate at which teachers met their goals and the participants’ opinions about the intervention. For example, Michelle and Mallory indicated to the researcher they were very motivated to meet their goals. These two participants also indicated their reasons for participating in the study were to improve their practice in the classroom and student engagement. They even asked whether the researcher would be available for more coaching after the study was completed. Participants received suggestions for improvement within the performance feedback email, and Kendall, Michelle, and Mallory quickly implemented the feedback. In contrast, Walter only implemented the feedback if suggested more than once. This suggests Walter was less responsive to the feedback than the other participants.
Anecdotal descriptions add depth to the results, illustrating the impact the intervention had on teachers and students. For example, in response to a performance feedback email, Michelle shared an interaction with a student in another class. The student asked Michelle, “Why are you always telling us what we did well?” Later, the student shared they enjoyed hearing the praise statements. Michelle went on to explain the positive impact the praise statements had on her practice as a teacher across all her classes.
Implications for Practice
Findings from this study have implications for educator training programs, school administrators, and classroom teachers. First, the brief training alone did not increase the teachers’ rates of BSP or OTR; most teachers’ rates of performance did not improve until the email performance feedback was implemented. This further supports the literature stating that quality professional development should include coaching, feedback, and ongoing support (Darling-Hammond et al., 2017). Therefore, school administrators and teacher educators responsible for providing training and professional development should consider including ongoing performance feedback in addition to traditional professional development. Including a self-monitoring component may also increase the long-term maintenance effects.
Although the results of this study support goal setting may be an effective way to increase teacher performance in the classroom, we did not examine how goal setting alone impacts teacher performance. Rates of BSP and OTR were low prior to starting the intervention, and after the intervention, their rates improved. Our findings do not lead to the definitive conclusion that goal setting alone was responsible for the improvement in performance; however, two of the participants shared during the intervention they were very motivated to reach their goals. This further contributes to the literature, indicating adding a goal-setting component to performance feedback plans may improve teacher performance.
Limitations and Directions for Future Research
Several limitations may guide future research. First, this study was well designed to draw conclusions about performance feedback and a combination of performance feedback and goal setting relative to a baseline condition. However, we could not draw strong conclusions about the efficacy of performance feedback and goal setting versus performance feedback alone. A component analysis could be conducted in future studies to determine the relative effects of the package components, such as a multi-element design in which the effects of different interventions or components were compared on the same dependent variable. Second, the dependent variable selection may have limited the type of performance feedback the teachers needed. For instance, Walter may have benefited from performance feedback on strategies to foster a productive classroom debate; however, the email feedback he received was limited to BSP and OTR. Moreover, the number of students in each class may impact the rate of improvement for some teachers. For example, Michelle had seven students, whereas Walter only had four. Michelle’s larger class size may have provided more opportunities to deliver BSP and OTR. Future researchers should consider allowing teachers to select the skills they receive performance feedback and implementing data collection in classrooms of similar class sizes. Third, a multi-element design requires both dependent variables to be independent, functionally similar, and equal in difficulty (Cooper et al., 2020); however, BSP and OTR may not be functionally similar or equal in difficulty. For example, both teachers who identified a goal for rate of OTR met and maintained their goal after just two to four performance feedback emails. In contrast, for teachers who set a goal for BSP, one teacher never met their goal, and the other teacher struggled to maintain performance above the goal consistently. Furthermore, OTR is an antecedent-based intervention for which teachers can plan in advance, whereas BSP is a consequence-based intervention that is delivered contingent on student responding. In future studies, researchers might explore selecting classroom management skills are more equal in difficulty and functionally similar. For example, specific corrective and praise statements could be compared using the same procedures as the present study. In addition, the present study did not include a student impact variable. In future studies, researchers should include a measure of student outcomes, such as students’ rates of on-task behavior. In addition, the first author served as the trainer and interventionist. Researchers may want to consider providing training that is not led by the primary interventionist in the study. Finally, generalization and maintenance measures were not collected for all participants due to schedule conflicts. In future studies, researchers should collect generalization measures during every phase of the study and should collect maintenance measures for all participants.
Future research should continue to examine the different components of performance feedback and goal setting to determine the most effective combination for teacher and student outcomes. Researchers should also explore the effectiveness of performance feedback and goal setting in different settings and with different teacher and student populations. For example, most teacher participants in the present study had less than 6 years of experience in the classroom; therefore, future research needs to explore the effectiveness of email performance feedback with goal setting for more experienced teachers.
Conclusion
With the increasing concern about teachers entering the classroom inadequately trained in classroom management practices (Freeman et al., 2014), researchers, teacher educators, and school districts need to find effective ways to remediate the gaps in teacher knowledge and skills. Findings from this study show teachers of students with EBD can improve classroom management skills with goal setting and written performance feedback. The goal of this study was to evaluate the effectiveness of objective performance feedback with goal setting on teachers’ classroom management skills. The results provide further evidence that training alone is insufficient in changing behavior, and performance feedback combined with goal setting may be an effective practice to improve teacher performance.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
