Going Full Circle With Teacher Feedback

Abstract

Research on the evaluation of early childhood programs focuses mainly on its outcomes rather than its process with often little attention given to the role that feedback to teachers in pre-kindergarten (pre-k) programs plays in the larger cycle of the evaluation process. This article provides a case example of a multiyear evaluation of community-based pre-k programs serving about 360 three- and four-year old children over a 5-year period in the District of Columbia. The Closing the Loop Evaluation Model proposed represents a responsive evaluation approach that illustrates the interconnected interactions between teacher feedback during the evaluation process and two supporting evaluation methodologies that emphasize social justice and utility. Findings from the case example highlight the responsive evaluation approach, feedback process, and ensuing conceptual and instrumental changes that occurred among stakeholders from whole-group feedback to small-group “report card” meetings with add-ons such as technical assistance, teacher-generated action plans, and teacher follow-up and feedback to close the evaluation loop. The authors discuss lessons learned about the evaluation process from the case example around aspects of feedback, including timing, audience, and function. Findings highlight the importance of feedback being timely and prompt, high quality in focus and content, non-punitive, collaborative, concise, and useful. The authors conclude that an evaluation process that includes teacher feedback, couched in social justice and utility, can have positive outcomes for all stakeholders and will likely lead to higher quality early childhood education programs.

Early childhood education in the United States has expanded over the years. In 1965, only 16% of 4-year-olds and 5% of 3-year-olds attended school (Barnett & Yarosz, 2007). By 2010-2011, about 75% of children attended public or private preschool programs at age 4 and about 50% at age 3 (Barnett, Carolan, Fitzgerald, & Squires, 2011). Several factors have contributed to the pre-kindergarten (pre-k) expansion, including (a) concerns about children’s school readiness and subsequent academic achievement, (b) advances in neuroscience research yielding greater insights on the plasticity and learning capacity of the young brain, (c) expanded need for early child care due to the tremendous increase in working mothers, (d) more advocacy for pre-k education, particularly in view of research touting its widespread societal benefits (e.g., reduced dependency costs, increased employment and associated tax revenue, and reduced crime), and development of national goals and standards on education (Goldsmith & Meyer, 2006; Lowenstein, 2011; National Association for the Education of Young Children [NAEYC], 2012).

The goal of pre-k education is to prepare young children for kindergarten and school by ensuring that they have the requisite cognitive, socio-emotional, and behavioral attributes of school readiness. Today, we know much more about how young children develop and how early childhood programs can make a difference (Isaacs & Roessel, 2008; Melhuish, 2011; Weiss, 2004). Evidence indicates that high-quality pre-k programs indeed improve children’s cognitive, social, and emotional skills, decrease the need for grade retention and special education services, and narrow the achievement gap. We know now more than ever before about the importance of early intervention and education services for young children. These programs, when delivered at the appropriate levels of quantity and quality, have meaningful and lasting positive impacts on children and families (Gilliam & Leiter, 2003; Reynolds, Temple, Ou, Arteaga, & White, 2011).

A 2010 analysis of 123 evaluations concluded that pre-k programs provide a real and enduring benefit to children, which persists beyond the early elementary years (Camilli, Vargas, Ryan, & Barnett, 2010). The reality is that too many children do not attend high-quality early childhood programs and children from low-income families are more likely to attend lower quality programs (National Education Association Policy Brief, 2010). Many states have taken steps toward universal pre-k education. However, the states are faced with the challenge of how to best design and implement a statewide program that provides high-quality services to all children being served. Although pre-k enrollment continues to grow, large disparities in access and program quality exist. As a result, there have been greater calls by researchers, policy makers, and practitioners for evidenced-based data on the effectiveness of such programs, particularly for children from disadvantaged backgrounds and their contribution in narrowing the achievement gap (Ferguson, 2007; Rashid, 2013).

Program evaluation is defined as the systematic process of describing the components and outcomes of an organized intervention or service with the aim to improve the quality of services received or to document a program’s beneficial impacts (Gilliam & Leiter, 2003). The primary purpose of pre-k program evaluations is to improve the quality of education and other services provided to young children and their families (NAEYC, 2003). Standardized observational data provide an objective overall picture of the classroom, a basis for teacher reflection and self-assessment, and input into the development and refinement of pre-k programs. Objective feedback from classroom observations helps teachers reflect in productive ways, thus bridging the gap between knowledge about what matters to students and changes in teachers’ actual practice (Stuhlman, Hamre, Downer, & Pianta, n.d.). Program evaluators can have an important impact on professional practice by providing relevant data, which in the context of quality mentoring and support help create high-quality and responsive classroom environments served by high-quality pre-k teachers (Lambart, Rowland, Taylor, & Wheeler, 2010). High-quality evaluations to document the fidelity and effectiveness of pre-k programs, particularly those serving diverse urban communities, are therefore increasing in demand.

The evaluation of pre-k education is both a labor-intensive and complex process, in large part, because program models vary considerably. There is a growing body of literature on the evaluation of pre-k and early childhood programs (Green & McKie, 2007; Green, McKie, & Manswell Butty, 2008, 2009, 2010; Hustedt, Barnett, Jung, & Friedman, 2010; Peisner-Feinberg & Schaaf, 2008). However, much of this work focuses on outcomes evaluation and less on process evaluation for improvement. It is, therefore, our contention that pre-k program evaluations should not simply aim at assessing a program’s merit and worth but should also provide sufficient useful feedback to pre-k staff. The feedback can then be used to improve teaching by providing teachers with strategies to elevate the cognitive, socio-emotional, and behavioral development of children. In other words, evaluators entrusted with examining pre-k programs, particularly those serving urban populations, must do more than simply analyze and report students’ scores on standardized tests. To reveal a more complete picture, evaluators should examine a host of social and other contextual factors (e.g., issues of access, equity, opportunities to learn, and rigor of curricula) that impact the learning of pre-k students. Evaluators of educational programs cannot ignore the fact that they are part of the never-ending struggle to make judgment calls about education, a social activity that creates the conditions or obstacles for social mobility.

This article focuses on process, not outcome, evaluation for improvement and provides a case example of a multiyear evaluation of community-based pre-k programs serving about 360 three- and four-year-old children in the District of Columbia. The responsive evaluation (RE) approach used was not simply a scientific endeavor in search of objective “truths” and “solutions.” We also viewed it as a social justice enterprise, whereby we argued for the use of teacher feedback and pre-k evaluation results to advocate change and restructuring for improvements in the education of young children.

Evaluation Process and the Importance of Feedback

Evaluation Model

Figure 1 highlights the Closing the Loop Evaluation Model we developed for our evaluation process. The model is embedded in a RE approach. During the evaluation, we saw the value of collaboration and participation through feedback to “close the loop” and to steadily improve program quality. Through this process, we ensured that the evaluation process was poised to positively address issues of social justice and utility. The model in Figure 1 considers the primary purpose of evaluation as seeking to improve the quality of education and other services provided to young children and their families. With this in mind, we hoped that our evaluations (a) had a positive impact on teacher quality and consequently on the education of all children, including the underserved and underrepresented, and (b) presented findings and recommendations that were useful to all stakeholders. These two important aspects of our evaluation address issues of social justice and evaluation for utility which are fundamental to the evaluation process and provide teaching staff with the feedback they need to reflect on findings and implement recommendations. The following section addresses the three major components of the model: (a) teacher feedback, (b) evaluation for social justice, and (c) evaluation for utility.

Figure 1.

Closing the Loop Evaluation Model.

Teacher feedback

Feedback can be conceptualized as information provided by an agent (e.g., teacher or peer) regarding aspects of one’s performance or understanding (Hattie & Timperley, 2007). It is a process that involves a two-way non-judgmental communication with the purpose of providing information about the quality of work to enhance one’s ability (Hamid & Mahmood, 2010).

Studies show that providing teacher feedback is still an area of concern and identifies a lack of adequate teacher feedback as a primary problem (Weisberg, Sexton, Mulhern, & Keeling, 2009) in the school setting. Evidence from one study suggests that while nearly three of four teachers may go through an evaluation process, few receive specific feedback about how to improve their practice. Furthermore, even when supervisors do provide feedback, it is often too infrequent to improve performance (Mielke & Frontier, 2012). As the Center for Teaching Excellence (2012) notes, feedback is important to improve individual performance because it can reinforce existing strengths, keep goal-directed behavior on course, clarify the effects of behavior, and increase an individual’s ability to detect and remedy errors independently. According to Williamson (2012), feedback is an important part of the learning process because it allows individuals to become effectively independent, especially when they can be observed putting the feedback into practice. When adult learners are empowered to objectively analyze and understand their own practice and have a clear vision of where they can improve, they are intrinsically motivated to embark on a pathway that leads to expertise (Mielke & Frontier, 2012). This empowerment also increases the individual’s self-esteem.

The literature clearly supports three general conclusions about feedback that should be adopted into practice: (a) feedback is better than no feedback, (b) immediate feedback is better than delayed feedback, and (c) feedback that is immediate, specific, positive, and corrective holds the most promise for bringing about lasting change in teaching behavior (Scheeler, Ruhl, & McAfee, 2004). To evaluate feedback practice, Brookhart (2008) identified different types of feedback strategies (e.g., timing, amount of feedback, mode, audience, focus, content, function, and valence) that research has found to be important for student achievement.

In early learning classrooms, conducting classroom observations along with mentoring or coaching approaches provides ongoing feedback and support about instruction and classroom practices. These approaches are acknowledged as potentially effective and perhaps are a more direct path to producing high-quality teaching in pre-k classrooms (Hamre & Pianta, 2005; Landry, Swank, Smith, Assel, & Gunnewig, 2006). Given the relevance of these approaches, it is not surprising that coaching and mentoring now dominate the teacher-training landscape as forms of feedback.

We argue that it is imperative that professional development models aimed to improve student outcomes include the component of teacher feedback as well as (a) focus on specific teacher–child interactions and implementations as targets; (b) anchor interaction targets in standardized, validated measurement systems; and (c) use consultation procedures that are both justifiable and standardized for training and supporting teachers to improve teaching and classroom quality (Pianta, Mashburn, Downer, Hamre, & Justice, 2008).

Evaluation for social justice

The other component of the model, “evaluation for social justice,” highlights the importance of promoting equity in varying contexts to advance democracy. The term “social justice” has varying definitions. While the definitions vary depending on the source, the common themes that exist include dignity, equality, justice, human rights, and responsibility.

House and Howe (1999), the framers of the Deliberative Democratic Evaluation or the democratic approach to evaluation, address social justice from an evaluation perspective. Their theory examines evaluation from the viewpoint of the inequities of social class and minority culture to advance social justice in a particular context and in the broader society. They posit that evaluation is embedded in the fabric of public decision making and “should be explicitly democratic,” helping to constitute a more democratic society. Their approach ensures that the interests of all stakeholders, specifically those of the powerless and the poor, are respectfully included. They propose that the procedures by which stakeholders articulate, share, and advance their interest in evaluation rest on three inter-related principles: (a) inclusion—that the interests of all legitimate stakeholders are included in the evaluation, (b) dialogue—among stakeholders which are offered as the process through which the real or authentic interests of diverse stakeholders are identified, and (c) deliberation—that is the rational, cognitive process by which varying, even conflicting, stakeholder claims are negotiated through reasoned discussion and with evidence and argument.

House and Howe offer several questions to guide a deliberative democratic evaluation: (a) Whose interests are represented? (b) Are major stakeholders represented? (c) Are there power imbalances? (d) Are there procedures to exclude power imbalances? (e) How do people participate in the evaluation? (f) How authentic is the evaluation? (g) How involved are the interactions? (h) Is there reflective deliberation? and (i) How considered and extensive are the deliberations?

The implications for this approach involve including the perspectives of all members of the group, ensuring that all are represented in conversations and decisions, and employing an array of mixed methods to draw conclusions of the program. Thus, the use of results could be used to advocate change and restructure improvement. This methodology requires the evaluator to promote a democratic sense of justice.

Evaluation for utility

A third component of the model, “evaluation for utility,” highlights the importance of the “use” of evaluation. Patton and Horton (2009) define program evaluation as the systematic collection of information about the activities, characteristics, and outcomes of programs so as to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. Undoubtedly, the use of evaluation results is one of the key assumptions underlying the planning, implementation, and dissemination of any evaluation. It is generally assumed that a good evaluation will yield information that will be used by a decision maker to take future action. It is important to develop and conduct evaluations in a way that increases the usefulness of the findings and the evaluation process to inform decisions and improve performance (Better Evaluation, 2012).

Patton (2013) suggests that evaluators should carefully consider the development of the evaluation design and process in terms of how it will affect use. He notes that the evaluation process is value laden. Thus, he expresses that evaluators are obligated to clearly identify the primary intended users of the evaluation who will in turn be responsible for using the findings and recommendations. Patton also notes that evaluation for utility is situational and personal. This calls for the evaluators to work with the intended users to develop an evaluation that meets their unique needs, that is, an evaluation developed within the confines of defined evaluation standards and principles. He also notes that a utility-focused evaluation can be conducted for any evaluative purpose (formative, summative), any data type (qualitative, quantitative), and any kind of focus (process, outcome, impact). Thus, he asserts that utilization-focused evaluation is a process of making decisions about varied issues in collaboration with an identified group of primary users focusing on their intended use(s) of evaluation.

As we embarked upon planning and implementing a pre-k program evaluation, the concept of using evaluation to improve and guide mid-course corrections for enhancing student outcomes was central to our work. In particular, we focused much of our planning and implementation efforts on facilitating how pre-k staff in their early childhood education settings would experience the evaluation process and how they could apply evaluation findings. As the pre-k programs under study served children of color in primarily low-income urban communities, we were particularly interested in ensuring that our evaluations yielded information that promoted the creation of more equitable and socially just practices in these educational settings. The evaluative framework adopted avoided blaming the victim by offering insights for improving classroom instruction leading to enhanced student developmental outcomes. Equally important to this evaluation was the meaningful use of results disseminated to early learning teachers. Results disseminated to the teachers were often modified to improve their utility. The underlying assumptions and tenets that this evaluation framework relied upon were adopted from other evaluation work discussed elsewhere (see Manswell Butty, Reid, & LaPoint, 2004; Thomas, 2004; Thomas & McKie, 2006).

The Case Example: Pre-Kindergarten Program (PKP) Evaluation

Overview of Evaluation

The District of Columbia’s PKP, formally the Pre-Kindergarten Incentive Program, is a standards-based demonstration program designed to provide high-quality, comprehensive early care and education to 3- and 4-year-olds in the District of Columbia. The demonstration program was implemented in community-based settings in 2005 which comprised of 8 to 10 sites, 21 to 30 classrooms, and 283 to 360 children over the course of the 5-year program. The basis of the program was to serve as a blueprint for creating and ensuring a delivery system of high-quality pre-k classrooms (Kamara, Sykes, & Young, 2004). PKP required (a) a standards-based curriculum, (b) technical assistance and ongoing professional development for all teaching staff, (c) qualified teachers, (d) parental involvement, (e) support for English Language learners, (f) comprehensive health and early diagnostic screenings, (g) small classroom sizes, (h) nutritious meals, and (i) accreditation from a national organization.

During the fourth year of PKP in 2008, the District of Columbia’s City Council unanimously passed the Pre-k Enhancement and Expansion Program Act aimed at creating high-quality universally available pre-k education services in the District of Columbia. The legislation required the development of a 5-year expansion plan to ensure that a minimum of 15% of unserved children are enrolled in pre-k programs each year, and a minimum of 25% of expansion programs are operated by community-based organizations.

PKP offered a full academic year of services to children enrolled and required the provision of at least 6.5 hr of daily instructional services for a total of 180 full days. The community-based centers varied in type and included for-profit, non-profit, and faith-based centers. PKP providers used one of six approved research-based preschool curricula: Creative Curriculum, High/Scope, CORE Knowledge Preschool Program, Open the World to Learning, Scholastic Early Childhood Program, and Scholastic Building Language for Literacy.

The evaluation sought to answer evaluation questions that included the fidelity of the PKP’s delivery systems and students’ developmental gains while in the program. Data were collected from classroom observations, curriculum-based assessments, focus groups, questionnaires, and standardized assessments. During the course of the PKP study, three standardized observation instruments were used to assess all of the pre-k classrooms: Early Language and Literacy Classroom Observation (ELLCO), Early Childhood Environmental Rating Scale–Revised (ECERS-R), and Classroom Assessment Scoring System (CLASS). The purpose of these assessments was to provide feedback that could be used to improve (a) teaching practices relative to children’s emergent language and literacy skills; (b) spatial, programmatic, and interpersonal features of the classroom; and (c) dimensions of instructional practices and quality of the classroom environment.

For the purpose of this article, the following discussion focuses on the evaluation approach and its link to improvements and outcomes. It should be noted that the discussion centers on teacher feedback and the process involved in conducting classroom observations, rather than on student outcomes.

Evaluation Approach

Our approach for the evaluation of PKP was built upon two similar theories: (a) RE theory (Abma & Stake, 2001; Stake, 2004) and (b) contextually responsive evaluation (CRE) framework (Manswell Butty et al., 2004; Thomas, 2004). The first theory (RE) asserts that REs respond to audience requirements for information and take into account the different value perspectives held by stakeholders in reporting a program’s successes and failures (Abma & Stake, 2001; Stake, 2004). This theory is oriented more toward program activities than to program intents. In this type of evaluation, flexibility is important. Evaluation activities and events do not have to be performed in a predetermined or linear order. The responsive evaluator goes where the data and emerging conclusions and opportunities lead. With this theory, it is essential that the responsive evaluator presents findings formally and informally to different stakeholders to increase their input, participation, “buy-in,” and use of the findings. The RE could also be described as a participatory evaluation approach. The objective of responding to the requirements of project staff and orienting to program activities provided a useful framework to foster both long-term and short-term behavior changes in the classrooms because creating quality pre-k classrooms was the long-term outcome for the project.

The second framework (CRE) is an outgrowth of the RE theory. CRE reinforces social justice elements and argues for the use of evaluation results to advocate change and restructuring for improvements, in schools, especially schools serving poor, urban students (Thomas, 2004). In addition, CRE emphasizes both process and outcomes as the intent to:

• et algenerate a more profound understanding of urban education, its students, and its context (knowledge development)

• et alhelp strengthen urban students and schools (developmental evaluation)

• et alprovide information that will enlighten and empower those who have been oppressed by or marginalized in school systems (transformative evaluation)

• et alexamine the merit, worth, productivities, and values of reform efforts (accountability and outcomes evaluations)

The CRE approach demonstrates the evaluators as being involved in decision making in all phases of program development and evaluation at the early learning centers to best meet the needs of all stakeholders. The long-term outcomes of this approach aim to improve classroom quality for all students, especially underserved students, who are considered as assets during evaluation planning, implementation, and dissemination.

Linking Evaluation to Improvements and Outcomes

Use of the results was one of the key assumptions underlying the planning and implementation of the PKP evaluation. We were mindful in documenting how PKP stakeholders experienced the evaluation process and applied the evaluation findings to make improvements. The following sections discuss conceptual and instrumental changes that took place with PKP teachers and the evaluation from year to year during the course of the project, especially as a result of the “report card” meetings.

“Report card” meetings

One goal of the PKP evaluation was to provide information that could be used to improve classroom quality in community-based pre-k sites. To achieve this goal, the evaluation team which included the evaluators, program funders, collateral services providers (training and technical assistance [T&TA] and comprehensive services), and school staff held “report card” meetings twice a year, during fall and spring, at each PKP site. The purpose of the meetings was to inform school staff about their overall performance on classroom quality improvements as measured by three classroom observation instruments: ELLCO, ECERS-R, and CLASS. The evaluators felt that reporting the findings back to school staff, in a timely manner, would allow them to use the feedback to make overall quality improvements in the classroom. The feedback also portrayed their strengths and areas needing improvement. At each report card meeting, the evaluators collected data from two internally developed instruments: (a) the Pre-K Documentation Form, which was used to evaluate the implementation and usefulness of the “report card” meetings, and (b) the Pre-K Assessment Report Card Questionnaire, which solicited information about how staff felt about the meeting.

The following section describes the changes made during 2005 to 2010 to report the findings back to the project staff and to give them feedback (see Figure 2).

Figure 2.

Program feedback and follow-up changes from Year 1 to Year 5 (2005-2010).

During Year 1, at the start of the project, the ELLCO instrument was the only classroom observation tool used to observe the classroom’s language and literacy environment. The observation occurred at the end of the school year and was conducted by T&TA, a collateral service provider, and not by the evaluation team.

Feedback to teachers began in Year 2 (2006-2007). It took the form of a whole-group approach in which feedback of classroom observation results was incorporated in monthly whole-group professional development workshops and trainings attended by project managers. At these meetings, results were disseminated to the project managers who then delivered them to teachers and teacher assistants at their centers. This was followed by site- and classroom-based targeted technical assistance provided by T&TA.

During Year 3 (2007-2008), the feedback approach changed and included both large-group and small-group formats. The evaluators presented the overall program results to project managers at project managers’ meetings in a whole-group format. Feedback of classroom observation results was further given in site-based, small-group “report card” meetings. The meetings were collaborative, with all stakeholders present (site directors, project managers, teachers, teacher assistants, collateral service providers, and evaluators). At these meetings, “report card” documents were issued and discussed. This was followed by targeted technical assistance provided by collateral service providers to individual sites. The “report card” format allowed for more one-on-one targeted support for staff as they learned of their areas of strengths and areas needing improvement.

During Year 4 (2008-2009), small-group feedback continued where program staff received feedback from classroom observations in site-based, small-group report card meetings with all stakeholders present. Large-group feedback was also conducted with project managers. Similar to Year 3, feedback was followed by targeted technical assistance provided by T&TA to individual sites. In addition, teachers developed an “action plan” to address areas needing improvement. The teachers were allowed to discuss the action plan and strategies to implement the plan at the report card meetings. This again allowed for more one-on-one targeted support for school staff as they learned of their areas of strengths and areas needing improvement.

During Year 5 (2009-2010), the small-group feedback approach continued. Similar to program Years 3 and 4, classroom observation feedback during Year 5 was given in site-based, small-group report card meetings. This again allowed for more one-on-one targeted support for program staff as they learned of their areas of strength and areas needing improvement. This was followed by targeted technical assistance provided by collateral service providers to individual sites. Overall program results were also given at project managers’ meetings. During Year 5, teachers were again asked to develop and present an “action plan” around areas needing improvement. New in Year 5 was follow-up to the action plan in which monitors from the funding agency checked in with the teachers to see how they were implementing their action plans.

The action plan and monitoring were added to close the evaluation loop, making the evaluation process go full circle. As a result of the process changes over the 5-year period, the evaluation team was able to document desired changes based on conceptual use and instrumental use.

Conceptual use

Process changes through conceptual use involve conceptual behavior changes observed among teachers based on their awareness of being evaluated and their experience of the evaluation process. The process change occurs when teachers conceptually rethink activities that take place in the classroom. These changes were seen in a number of ways throughout the project cycle.

Conceptual changes were observed in the area of classroom practice. During the “report card” meetings, project staff received feedback about their strengths and weaknesses in the classroom in the areas of language and literacy activities, and teacher–student interactions. Based on the feedback teachers received and their awareness of what constitutes a high-quality classroom, many positive changes were noted from pre- to post-observation. Teachers were allowed to review the findings and have input on action plans that outlined specific activities for improvement. As a result, scores gradually increased across sites and classrooms from pre- to post-observation over the years.

Enhanced discourse among teachers was another area of conceptual change. Initially, during the evaluation and “report card” meetings, teachers had to become more familiar with evaluation and instructional terms and concepts. As a result of professional development training and collaboration among teachers at report card meetings, conceptual changes were observed in the understanding and use of terms and concepts related to issues around early childhood education (e.g., self- and parallel-talk, phonological awareness, scaffolding, and environmental print). Enhanced discourse was observed at trainings and “report card” meetings. This was clearly evident when teachers had to present their action plans and explain the strategies they would use to maintain their strengths and overcome their areas of weaknesses.

Intentional teaching among teachers was another area of conceptual change. To achieve the goals of a quality classroom and a smooth transition to kindergarten, teachers began to focus their attention on their instructional strategies in the classroom. After the “report card” meetings, teachers aimed to improve their scores and classroom quality through intentional teaching. Their strategies were all directed to effectively organize the classroom environment, engage students in language and literacy activities, and sustain cognitive growth in the classroom.

Conceptual change was also seen in teachers becoming more reflective practitioners. Through discussions and actions, teachers often expressed rethinking their teaching methods, approaches, and strategies to improve classroom quality.

Instrumental use

Changes in instrumental use can be defined as changes in the use of instruments and forms used as a result of the report card meetings. Based on the meetings with teachers and other stakeholders, changes occurred in the type and amount of details in “report card” notes given by evaluators to the teachers and other school staff about the classroom observation instruments. These changes occurred to increase the utility of the documents and feedback given to the teachers. Initial feedback from the Pre-K Documentation Form and Pre-K Assessment Report Card Questionnaire highlighted the fact that the notes and figures presented were not always clear. As responsive evaluators, we continued to modify the reports given to teachers to make the documents useful, understandable, and concise. As time progressed, teachers reported that they liked the specific feedback they received and the charts and explanation of findings. They also liked the clear descriptions of the notes and detailed explanations.

The evaluation team also developed an “action plan” to be used by PKP teachers to assist with improving areas of challenge. As the program progressed, the team observed the need to continue to have teachers become more meaningfully integrated in the evaluation process. Consequently, after receiving the results, it was decided that teachers would develop an action plan to describe the strategies they would use to make quality improvements. Completing the action plan came with the support and assistance of team members like T&TA.

Lessons Learned

While evaluating PKP, we learned several things about conducting the evaluation in ways that would benefit teachers and school administrators. We used Brookhart’s (2008) types of feedback (e.g., timing, amount of feedback, mode, audience, focus, content, function, and valence) to reflect on our experiences while conducting the evaluation. Our outcome as evaluators was to make this process useful and meaningful in a way that allowed teachers to develop professionally and improve classroom quality for all children. In the following discussion about lessons learned, we infuse the voices of staff and administrators from documentation forms and questionnaires collected during whole-group and small-group meetings.

Timing and Amount of Feedback

We learned that giving results to stakeholders in a timely manner was critical. It helped teachers to use the results to make improvements. We understood that feedback needed to be provided while teachers were still thinking about the goals of the project and working toward a learning target. There were a few instances where returning the results to the teachers in a delayed manner did not help them develop professionally. When asked about what could be improved, one teacher responded, “The results could be provided earlier.” Another teacher indicated, “The reports and results need to be given back to us in a timely basis.” Yet, other school staff commented on the timing of the pre- and post-results, indicating “We would like to get baseline results soon to implement improvement strategies before second observations.” Similarly, another teacher responded, “The turnaround time between pre- and post-assessments could be better.”

The amount of feedback provided was also very important. In some instances, we gave too much feedback and in other instances we gave too little. Over the years, we found ourselves adjusting our feedback reports to find the right balance of information that would be useful to school staff. Feedback from teachers and administrators about the amount and quality of feedback they received assisted in this process. When asked about the amount of feedback received, one teacher noted, “more information on CLASS.” Another indicated, “more information on the scoring process.” One teacher talked about the level of details in the assessment and stated, “There needs to be more detail in the assessment as to what we need to work on.” For another teacher, the recommendation was, “More detail is always better.” Some teachers also voiced that they needed “examples of what to improve.” After adjusting the amount of feedback, one teacher expressed, “Examples of the behaviors that were observed were given and examples were given in order to improve.”

Mode and Audience

Feedback can be delivered in many modalities. We found that a combination of modalities worked well. We used written and oral feedback to report back to teachers. Written feedback was disseminated in the form of “report cards” and oral feedback in the form of presentations to teachers, both in groups and individually. Feedback to teachers worked best when there was a clear sense of the make-up of the audience (i.e., teacher and teacher assistants, project managers) to convey the right feedback. There were some instances when group feedback worked well (e.g., when giving overall results for the program) and other times when it did not (e.g., when focusing on specific strategies). We had to be aware of these instances and make decisions about how to give effective feedback based on the audience type and approach.

Regarding the mode and audience, one teacher indicated, “The [small group] meeting was interactive, open, and informative.” One administrator stated that in addition to the location of the meeting, she liked “. . . the format of the meeting and conversation style.” Another said, “The group size permitted ‘real’ conversations.” Yet, another teacher indicated, “I like the support provided and the roundtable discussions.”

Feedback Focus and Content

Feedback to teachers and administrators often focused on the evaluation process or the steps used to conduct the classroom observations followed by the results or outcomes. Feedback discussions about the evaluation process showed teachers how the observations were conducted, what was observed, the outcomes, and the next steps. There was evidence presented around areas of strong teacher performance and areas that needed to be improved. From this feedback, teachers were able to develop action plans for quality improvements. During “report card” meetings, teachers received specific feedback aimed to improve their instruction and the general classroom environment. Overall, teachers were pleased at the feedback focus and content. One teacher asserted, “I liked how detailed the assessment report card is. I love the notes.” Another teacher indicated, “[I liked] the explanation and reasoning of the scores and questions. They [the evaluators] took the time to find ways for the teacher to maintain higher scores and understand how the scores are calculated.” Yet another teacher indicated, “The tables/figures were very detailed and helpful, as well as the recommendations.” Another teacher expressed her satisfaction by stating, “The evaluators explained the assessment report card and the details of what happened during the assessment were available.” Similarly, another teacher expressed satisfaction by stating, “The team explained, in detail, the observers’ scores. They [the evaluators] also explained the ‘substantial portions of the day’ part that observers score or tally.” Yet another teacher agreed and expressed, “The presenter explained every aspect of the assessments.”

Function and Valence

The function of the feedback given was to describe what we saw with supporting evidence. We did not evaluate or judge teachers’ practices in any way that would stop them from trying to make improvements. The objective was to express what we observed during classroom observations and identify areas of strength and areas needing improvement. We ensured that the feedback we gave to teachers was positive and non-punitive. We always began the discussion by highlighting the teacher’s strengths, then discussing areas needing improvement. In addition, we offered useful teaching strategies and gave suggestions for improvements as a team. The teachers and administrators often expressed they liked that the meetings were professional and challenged the staff to self-reflect and think about how to implement strategies for improvement. As one teacher said, “It was a stress-free meeting.” Yet another teacher indicated, “The meeting was held in a more relaxed state.” One administrator indicted, “I appreciated that the information was not viewed as punitive and the goal was to work together as a team.” Another administrator responded, “The meeting was presented as a place for all to listen and respond (without penalty) and receive clarity.” A teacher agreed, “It [the meeting] was not punitive and the teachers were given the opportunity to express their concerns.” Some teachers felt that because of the tone of the meeting, “it challenged the staff to self-reflect and think about how to implement strategies for improvement.” Similarly, another teacher expressed, “All attendees were professional and listened to teachers’ concerns.” Another teacher indicated that the meetings allowed for “. . . self-reflection and a plan of action.”

Overall, we learned that there were several things we needed to keep in mind. We had to provide timely and prompt feedback to teachers and other early learning staff so that they could implement recommendations and suggestions in real time. Feedback to teachers also needed to be of high quality in focus and content in all settings (e.g., face-to-face or group). Feedback needed to be professionally presented and non-punitive at all times. We learned that a collaborative and responsive team approach always worked best. In our feedback meetings at the sites, we met as a team with the teacher, teacher assistant, director, collateral service providers, and the funder. As a result, feedback came from different professional perspectives all in an effort to assist and support teachers to improve their instruction and classroom quality. We further learned that information given to teachers needed to be concise and useful. Teachers liked information presented in graphs, figures, visuals, and summaries. Teacher involvement in meetings was integral and teachers needed to use the feedback given to develop, sometimes with support, action plans to close the feedback loop.

Conclusion

From the multiyear PKP case example, we learned that an evaluation that considers teacher feedback, social justice, and utility can have positive outcomes for all stakeholders and will likely lead to higher quality early childhood education programs. Teachers who are provided with ongoing high-quality feedback that is immediate, specific, positive, and corrective, and that include mentoring and coaching, are much more effective in producing high-quality teaching and learning environments. An evaluation that promotes a democratic sense of justice based on the inter-related principles of inclusion, dialogue, and deliberation ensures that there is dignity, equality, justice, and a shared sense of responsibility and collaboration for all stakeholders involved in the early learning environment. A focus on the meaningful use of evaluations could result in positive conceptual and instrumental changes seen in classroom practice, discourse, and intentional and reflective teaching. From our multiyear PKP evaluation process and based on our evaluation model, we have become cognizant of the need to conduct REs that integrate the key notions of teacher feedback, social justice, and utility to enhance classroom quality.