Sage Journals: Discover world-class research

Abstract

This article reports on ways in which one Australian independent school seeks to develop and sustain best practice and academic integrity in its programs through a system of ongoing program evaluation, involving a systematic, cyclical appraisal of the school’s suite of six faculties. A number of different evaluation methods have been and continue to be used, each developed to best suit the particular program under evaluation. In order to gain an understanding of the effectiveness of this process, we conducted a study into participants’ perceptions of the strengths and weaknesses of the four program evaluations undertaken between 2009 and 2011. Drawing on documentary analysis of the evaluation reports and analysis of questionnaire data from the study participants, a number of findings were generated. These findings are provided and discussed, together with suggestions about ways in which the conceptualisation and conduct of school program evaluations might be enhanced.

Keywords

Evaluation methods evaluation utilisation program evaluation school-based evaluation

Introduction

The process of systematic program evaluation through formal strategic planning cycles has been a long established tradition in many schools, both at the school and the system levels (Barber, Chijioke, & Mourshed, 2010). Where there is a purposeful, systematic approach to the management of quality in academic programs, a solid profile of ongoing sustainability and quality has been shown to emerge. For example, through providing a focus on learning and teaching, an effective evaluation process can assist in developing strategies, actions and measurable targets that, in turn, can contribute to new strategic planning cycles. Further, the process can foster the notion of best practice in learning and teaching as a common goal for all members of staff through, for instance, the provision of meaningful feedback to teachers to improve their teaching practice. The literature shows that the provision of such feedback to teachers can have significant impact on student learning (see, e.g. Hattie, 2009; Leigh, 2010).

Designing and conducting effective school program evaluations has, however, been shown to be problematic. In the case of the denominational school in which this study was located, which functions as an independent entity under the jurisdiction of a School Board, a considerable degree of autonomy exists at the school level. As such, it is not constrained by some of the factors impeding program evaluations in heavily centralised schools. Nevertheless, there are a number of external and internal factors that impact upon, and need to be considered, in undertaking any approach to evaluating school program effectiveness. The complexity between these factors also needs to be effectively managed (Wikeley, Stoll, Murillo, & De Jong, 2005). Although contextualised in different ways, key factors include community engagement, leadership, management policies and practices, and the school context and culture (McDavid & Hawthorn, 2006).

In the case of the latter, for example, it has been convincingly demonstrated that strong school cultures enhance student performance (see, e.g. Fullan, 2001; Hoy, Tarter, & Hoy, 2006; MacNeil, Prater, & Busch, 2009) and that improvement initiatives in learning and teaching are ineluctably mediated through the climate and culture of the school (Hallinger & Heck, 2011). Lakomski (2001, p. 68) makes the point that healthy organisational cultures are able to embrace improvement processes and change to extend the organisation ‘beyond its currently held understandings of itself and its ways of dealing both with its internal and external reality’. In schools, this means promoting and embracing high academic standards, strong and effective leadership and sustained collegiality, all of which generate a climate conducive to student success and achievement (MacNeil et al., 2009). Barber et al. (2010) argue that successful school systems are able to both navigate challenges of context and also use context to their advantage, provided the following four contextual factors are taken into account: the desired pace of change; whether the desired change is ‘non-negotiable’; the degree to which there are winners and losers as a result of the change; and the credibility/stability of system leadership and governance.

None of this can occur without the presence of effective school leadership, which has been shown to be crucial to any organisation that aspires to cultivate a culture conducive to change and new approaches (see, e.g. Holmes, Clement, & Albright, 2013; Robinson, 2010). Fullan (2002) argues that reforms leading to sustained improvement in student outcomes can only be achieved in schools where there are leaders adept at handling complex and rapidly changing environments. Durlak and DuPre (2008) agree, claiming that strong leaders who lead and promote change agency ‘can do much to help orchestrate an innovation through the entire diffusion process from adoption to sustainability’ (p. 338).

In the school under study, the leadership team established and, since 2009, has progressively implemented a strong evidence-based improvement agenda, couched in terms of amelioration of measurable student outcomes. Clear and explicit school-wide targets for improvement, with accompanying timelines, have been set and communicated to all in the school community. One of the ways in which school leaders have attempted to meet targets in learning and teaching has been through the introduction of a systematic and ongoing program evaluation process across the school’s six teaching faculties. A number of different evaluation methods have been – and continue to be – used, each one developed to best suit the particular program under evaluation.

In order to gain an understanding of the effectiveness of this process, we conducted a study into participants’ perceptions of the strengths and weaknesses of the four program evaluations that were undertaken between 2009 and 2011. Drawing on documentary analysis of the evaluation reports and analysis of questionnaire data from the study participants, we generated a number of findings that are presented and discussed in this paper, together with suggestions about ways in which the conceptualisation and conduct of school program evaluations might be enhanced.

Context

The school that provided the context for the study is a non-selective, urban Australian denominational day and boarding girls’ school. Its mission is to be a learning community that welcomes diversity in an educational environment that is shaped by Christian values. Founded 120 years ago, the school has a strong academic charter, supported by a diverse range of sporting, cultural and co-curricular programs. Located across three campuses (Junior School: Kindergarten to Year 4, Middle School: Years 5 to 8 and Senior School: Years 9 to 12), the school’s focus is on providing an academic, liberal education with an emphasis on intellectual rigour, personal responsibility and commitment to others. A wide range of subjects is offered from Kindergarten onwards and there are active programs for extension and academic support in all sections of the school. Also within the school is a School of Performing Arts (SPA), which is a selective entry specialist school preparing students academically while focussing on specific performing arts areas such as music, dance or theatre performance.

Of the 160 staff members, approximately 100 are part of the teaching faculty working directly with the 850 students. A leadership team comprised of the heads of the internal mini-schools and the school chaplain are led by the deputy principal who in turn reports to the school principal. The deputy principal also has carriage of the whole of school curriculum team who meet fortnightly to progress the academic agenda of the school. Other leadership groups include the pastoral teams, faculty teams and a sports administration team.

The program evaluation process

Framework

The framework used to conceptualise and design the school’s program evaluation process proved effective in summarising, organising and sequencing the essential stages and features of the process. Presented in Figure 1, the framework is a modification of the public health evaluation framework developed by the Centers for Disease Control and Prevention (CDC) (1999).

Figure 1.

Program evaluation framework (adapted from CDC, 1999).

The framework comprises the steps involved in evaluation practice and the standards for effective evaluation, all of which are imbedded within, inseparable from, and impacted by both the school context and climate and also broader external community and educational factors. The process is further framed in three stages. Although there is some interrelation between the stages, Stage 1 roughly involves the first three steps of engagement, design and focus, Stage 2 the steps of data collection, analysis and interpretation (Steps 4 and 5), and Stage 3 the sharing and dissemination of findings (Step 6).

In this study, the execution of the six interdependent steps differed between the reviews, as did the personnel involved in decision making about the evaluation design and conduct. This was one of the reasons that the process was deemed ‘rich turf’ for empirical study. The four standards of utility, feasibility, propriety and accuracy were used to underpin and inform all steps of the process in order to ensure the effectiveness of each evaluation. Table 1 provides a brief overview of how the stages, steps and standards framed the broad evaluation process.

Table 1.

Stages, steps & standards of the evaluation framework (adapted from CDC, 1999).

Stages & Steps
Stage 1	1. Engage the stakeholders	Those involved in, served or affected by the program
	2. Describe the program	What the program entails; who it is served by & for; the context in which it exists
	3. Focus the evaluation design	To assess issues of greatest concern to stakeholders while cognisant of economies of scale & resources
Stage 2	4. Gather credible evidence	To indicate how the program has performed & how it can be enhanced
	5. Justify conclusions	By comparing the evidence gathered with the agreed-upon values/standards set by stakeholders
Stage 3	6. Ensure use and share lessons learned	Through feedback, follow-up & dissemination
Standards
Utility		Serves the needs of intended users
Feasibility		Is realistic, judicious, diplomatic & cost-effective
Propriety		Conducted legally, ethically & with due regard for those involved in the program & affected by the evaluation
Accuracy		Reveals and conveys technically accurate & defensible information

As we now describe, the four program evaluations contextualising this study were couched within this framework.

Design

Although articulated in slightly different ways, the overall aim of each evaluation was to gauge identified stakeholders’ (a) overall satisfaction with, and (b) perceptions of the strengths and weaknesses of the current programming model in order to enhance the program. The design approaches of each particular evaluation were overseen by the principal and deputy principal and are elaborated in Table 2.

Table 2.

Design approaches of the four evaluation.

Program	Evaluation context
School of Performing Arts (SPA)	New programming model with newly appointed director who initiated the evaluation
Maths & Science (M&S)	Scheduled evaluation
LOTE	Principal perceived a need for change in the programming model; awaiting appointment of new Head of Faculty (HoF)
Expressive Arts (EA)	Principal wished to investigate staffing & structural issues
	Evaluator/s
SPA	SPA director & Senior Curriculum Adviser
M&S	External consultant with Maths & Science expertise
LOTE	External consultant with LOTE expertise
EA	Senior Curriculum Adviser
	Method
SPA	• online survey for all SPA teachers, students & their parents/guardians
SPA	• Focus groups for (a) SPA students and (b) SPA parents/guardians
M&S	• full-day workshop for Year (Yr) K-12 M&S teachers, who assisted in the development of a student survey
	• online survey for students in Yrs 3, 4 & 7
	• focus groups for Yr 7 students
	• individual semi-structured interviews with sample of M&S teachers & senior administrators
LOTE	• online survey for students in Yrs 7 & 8 and those in Yr 10 who did not select a LOTE
	• semi-structured interviews with acting HoF, staff member/s from each language & 6 key leadership & senior staff (selected by Deputy Principal)
	• focus group for Yr K-12 parents
EA	• EA teachers invited to provide written input into the evaluation process
	• half-day workshop with all EA teachers
	• unstructured interviews with 7 EA teachers
	• focus groups with EA students
	Dissemination of findings
SPA	All findings from the final report were discussed with SPA teachers
M&S	Commendations & some recommendations of the final report discussed with staff
LOTE	Modified version of the final report circulated to/discussed with LOTE staff
EA	Final report was not discussed with staff

The different approaches listed above were tailored to suit each particular program and its context, while meeting the school’s program evaluation standards. It is beyond the scope of this paper to discuss the reasoning behind each set of approaches. Rather, we focus on how teachers responded to the evaluation/s in which they were involved.

The study method

The aim of our study was to gain an understanding of the strengths and limitations of the different school program evaluation methods, from the perspectives of the staff members involved in those evaluations. Thus, the central framing research question was: In the view of participating staff members, what were the strengths and weaknesses of the four program evaluations undertaken between 2009 and 2011? While there was arguably scope to broaden the study to include other stakeholders, such as students and their families, our particular interest was in understanding the views of the teachers who would be predominantly responsible for implementing any program changes recommended in the evaluations. Thus, we limited our study to teaching staff.

The two researchers responsible for designing and conducting the study both played roles in the program evaluations, albeit in different ways. Specifically, one researcher, a key curriculum leader in the school, acted as an (internally based) facilitator in one evaluation (SPA) and as an evaluator in one other (Expressive Arts). She also takes co-responsibility for the oversight of the school’s embedded evaluation process. The other researcher, a university academic, was employed in a consultancy role as an ‘external’ evaluator of the languages other than English (LOTE) program. (A third external evaluator, responsible for the Mathematics and Science evaluation, did not play a role in the study.) Once the school principal had provided consent and the relevant Social Sciences HREC had granted ethics approval (University of Tasmania, 2010), data were collected across two phases, spanning 2009 to 2012.

Phase 1: Documentary analysis

A key principle of the school’s program evaluation design is that it incorporates participation from across the range of school and community stakeholders. As such, and as outlined above, students, their families, school board members, leadership team members, and teaching and administrative staff were invited to participate. From each evaluation, a number of documents, including survey data, field notes taken during workshops, interview transcripts, and final reports were generated. The first phase of our study involved (a) extracting from these documents data provided by teaching staff that related to the evaluation process (as distinct from the program per se), and (b) analysing these data within the frame of our research question.

Following Rapley’s (2007) interpretive approach to documentary analysis, we each examined the documents and document extracts through a series of iterative readings to establish ways in which teachers socially constructed their experiences of engaging in the evaluation process/es. Working collaboratively, we then compared and contrasted our margin notes, applied codes (single words and phrases) to the views and interpretations reported in the data, and finally generated three of the five themes reported in the findings of this paper. These themes included: understanding the evaluation aim and purpose, staff contribution to the evaluation, and timing, scope and sequencing of the evaluation.

Phase 2: Online questionnaire

In late 2011 and early 2012, when the four program evaluations – and associated documentary analysis – had been completed, we designed and generated an online questionnaire (SurveyMonkey Inc., 2013) with the purpose of further gauging staff members’ views of the evaluation processes. The questionnaire included five open-ended questions that were purposefully designed to collect as broad a range of responses as possible, namely:

From your perspective, what were the strengths of the evaluation process and how it was conducted?

From your perspective, what were the limitations of the evaluation process and how it was conducted?

In what ways, if any, do you believe the evaluation process could have been improved?

Do you believe the findings of the evaluation (e.g. Commendations, Recommendations) will be acted on? Why//Why not?

In your view, whose responsibility is it to ensure the findings (e.g. Commendations, Recommendations) are acted on? Please explain why.

Purposive sampling (Cohen, Manion, & Morrison, 2011) was used to select staff members who had participated in one or more of the program evaluations. This method generated a sample of 67 teachers, including two who taught in two programs. All those in the sample were invited via email to participate with informed consent by following a URL survey link. In order to avoid any perception of researcher coercion (Babbie, 2013), emails were sent from a generic school email account and the questionnaire was designed to ensure that all response data were unidentifiable. The overall response rate was just under 30%, indicating a reasonable level of interest and engagement in this evaluation appraisal. Respondent numbers and response rates can be found in Table 3, and Table 4 provides comparative information for sampled teachers and achieved respondents.

Table 3.

Questionnaire respondent numbers & response rates.

Program & Year of evaluation	Teachers	Respondents	Response rate
SPA 2009	15	3	20%
Science & Maths 2011	24	8	33%
LOTE 2011	8	6	75%
Expressive Arts 2011	20	3	15%
Totals	67	20	36%

Note: The column headed “Teachers” refers to the number of teachers sampled as they had been involved in at least one program; the total number of teachers at the school was 100.

Table 4.

Comparative information for sampled teachers and achieved respondents.

	Characteristic
Program	Gender: sampled teachers/respondents	Teaching Experience (mean): sampled teachers/respondents	Years at the school (mean): sampled teachers/respondents
SPA	100% F/100% F	10 yrs/10 yrs	5 yrs / 7yrs
Science & Maths	75% F & 25% M/38% F & 62% M	15 yrs/20 yrs	7 yrs/13 yrs
LOTE	88% F & 12% M/83% F & 17% M	25 yrs/25 yrs	15 yrs/18 yrs
Expressive Arts	85% F & 15% M/100% F	15 yrs/20 yrs	5 yrs/8 yrs

Questionnaire data were analysed using cluster analysis (Miles & Huberman, 1994), which, in a similar method used in the documentary analysis, involved the two researchers individually coding and categorising the data, comparing codes and categories, and modifying the final categories. We then conducted a cross analysis with the codes and categories derived in Phase 1 in order to generate the themes that represent the findings of this study.

Findings

Five key findings emerged from data analysis conducted to yield a response to the research question: In the view of participating staff members, what were the strengths and weaknesses of the four program evaluations undertaken between 2009 and 2011? The five themes and a brief extrapolation of each are presented in Table 5. Our data analysis draws attention to some key areas, many of which need to be considered in other school program evaluation processes.

Typically, the analysis of data collected in this study has resulted in support or corroboration of the extant literature. However, there are a number of issues raised by participants that add more detail or nuance to what has already been reported. For instance, although each evaluation was designed to suit the particular program and the context within which it was located, a number of teachers reported questioning this design process on the grounds that, for example, there were inequities inherent in the differentiated approaches across the evaluations. This articulated concern points to the lack of understanding that some participants reported having about the evaluation aim and purpose, and to the questions raised about the selection of evaluators. Thus, in many ways, as this example demonstrates, each of the themes is interrelated and interdependent in terms of an overall picture of an effective school program evaluation.

Discussion

Understanding the evaluation aim and purpose

Not surprisingly, there was consensus among teachers that the process and intent of the evaluation need to be preferably shared by, and at the very least clearly articulated to, all staff before it is undertaken. This type of collaborative communication has been shown to be key to effective evaluations (see, e.g. Scheerens & Demeuse, 2005). In regards to the actual conduct of the evaluations, quite disparate views were reported about how well teachers understood the aim and purpose of the process.

Each of the program evaluators had different agendas in terms of their review’s terms of reference and different degrees of control over their ability to communicate the evaluation aim and purpose to participants. Even when the aims of the review are articulated to individuals, and in some cases collaboratively created, there is still room for suspicions and ill-feeling to emerge. Feelings of a lack of trust about any possible underlying issues or an alternative agenda can easily emerge especially where there is a lack of transparency in the processes and where the evaluation process may appear to be ‘top-driven’. Respondents’ views reflect these different perceptions; the first where time was spent in an initial whole group sessions are positive:

It was an open and transparent review … [and] we knew why we there and what to expect.

The initial meeting gave a good contextual background for the evaluation.

Where the aim was not clearly articulated, the respondents’ perceptions of the review were coloured by this and it seemed to create an undercurrent of distrust and lack of faith in the process:

I arrived at the meeting not knowing what it was really about and what my role might be.

It would have been good to have more time as a group with the facilitator, so that the aims of the evaluation were clear to everyone.

I would like to have known more about the timeline of the process.

It should be stated that the program evaluators did not have full control over this communication process. In one case, the review aim was co-created by the evaluator and the head of faculty but this was not articulated to the teacher participants. Rather, judging by a number of statements made by participants, there seemed to be a deliberate undermining of the process characterised as a ‘them versus us’ approach and, for some, this tainted the initial stage of the process. Despite good intentions from senior management and the evaluators about transparency and openness in the processes, where the school climate does not reflect this openness or where individual personalities and personal histories are not positive, then the outcomes will be less than ideal. These are the types of realities of workplaces and the challenges of context that have to be taken into account, as noted by Barber et al. (2010) and others.

Staff contribution to the evaluation

The conceptualisation and design of the four evaluations involved different levels of engagement by staff, varying from significant input by Mathematics and Science teachers to none at all by LOTE staff. Although staff involvement in these preparatory stages was not initially afforded much consideration by facilitators, it proved quite powerful for those who participated. For example, on the one hand, all Mathematics and Science teachers had the opportunity during a whole-day workshop to discuss the aim, purpose and anticipated outcomes of the evaluation, as well as to contribute to the development of the student questionnaire and student focus group questions. Participants reported this level of staff engagement as one of the overriding strengths of the evaluation, as exemplified in these comments:

All teachers of Mathematics and Science were included right from the start in how the evaluation would work. This was refreshing and proved productive.

[The workshop] allowed for an interesting discussion, with all teachers being called upon to contribute. Through this discussion, it was very evident to us that there was work to be done to bring the teaching of science … to the required standard.

On the other hand, LOTE staff members were not invited to play a role in any of the early stages of the evaluation. The external evaluation facilitator, in collaboration with the deputy principal and senior curriculum adviser, took full responsibility for the conceptualisation and design of the evaluation. Although this represents quite a traditional and commonly used approach to evaluation, it was not well received by participants, particularly in light of the involvement that they knew Mathematics and Science teachers had had in their evaluation. Further, it exacerbated the levels of anxiety and scepticism already held by some in the Faculty about both the future of languages in the School curriculum and the purpose of the LOTE evaluation. During individual interviews, several teachers asked whether there was a ‘hidden agenda’ at play and to what extent the recommendations of the evaluation had been predetermined. Questionnaire comments such as the following indicate that staff would have welcomed much more involvement in the evaluation process:

The terms of reference were very open and generic. … As LOTE teachers, we could have developed more explicit terms of reference instead of wondering what was being reviewed. The way LOTE is taught? The efficiency of LOTE teachers? The scope for LOTE teaching at school?

Even though a portion of the School community was included, maybe not the widest section. Not sure why … it wasn’t our decision to make apparently.

Durlak and DuPre (2008) make the point that effective initiatives are more likely to be better implemented in situations where collaborative methods have been used to determine how the initiative should be conceived and conducted in the first place. Shared decision making through the entire evaluation process from conceptualisation to diffusion can empower individuals to exercise a degree of control over what occurs and can cultivate champions who exude a sense of optimism about the evaluation process (Haigh, 2012). Data in this study would suggest that collaborative methods play an important role in the perceived success of the evaluation process. Given that all evaluations were intended to explore ways to either create a new school-appropriate program (SPA) or to enhance the existing program (Mathematics/Science, LOTE, EA) the importance of staff involvement in the early preparatory evaluation stages takes on even greater significance.

Further, there was evidence that those involved in the evaluation design process benefited from collaborative engagement and discussion that transcended the level of involvement in planning the procedural requirements of the evaluation:

The single session at the start [allowed for] an interesting discussion with all teachers being called upon to contribute.

The way it was set up made it an open and transparent review.

In having the opportunity to provide answers to the questions: ‘What aspects of the program are to be evaluated, and why?’ – precursory questions to ‘How will the program be evaluated?’ – program staff were able to share understandings about such issues as curriculum expectations and design, student engagement and outcomes, and teacher professional learning and support, all key in systematic curriculum delivery (Masters, 2010).

Choice of evaluators

Two of the four evaluations were undertaken by external consultants and two were undertaken by an internal evaluator. There are benefits and limitations in the two differing approaches (Conley-Tyler, 2005). An internal evaluator can have a deep understanding of the school context and of the project; there is familiarity also with the staff and the community groups involved through being part of the organisational structure of the school. They are undoubtedly less costly. On the downside there may be perceptions of lack of objectivity, lack of skills or time to devote to the work, and there may be perceptions that the evaluation and its outcomes are less important to school management because of the lack of external evaluation, and that the recommendations made will hold less weight and credibility. All of these elements were evident in the participants’ responses to the internal evaluator’s work as evidence in these comments:

[The evaluator] understood the issues we were grappling with.

Some people were concerned about a lack of anonymity.

I’m not sure why we had an internal reviewer. Was it to do with the importance of the review of our program? Finances?

Using external evaluators has the potential of bringing an outsider’s perspective to provide a more independent evaluation backed by personal expertise and experience from other evaluations. As the external evaluator is not part of the normal organisational structure, they may have less knowledge of the school context and political environment. Again these elements were picked up by survey respondents who commented:

There was no feeling of bias in the way the information was collected and presented.

It is arguable that the facilitator had a view formed before data was readily available.

The review did not occur in the context of the school organisation with all that [entails].

Irrespective of whether the reviewer is an internal or external evaluator, the most important consideration is the type of approach that is used, which, according to Durlak and DuPre (2008) needs to be collaborative and ‘characterized by non-hierarchical relationships among participants, mutual trust and open communication, shared responsibilities for completing tasks, and efforts to reach consensus when disagreements or stalemates arise’ (p. 338). Judging by documentary and survey data, these characteristics were, in the views of participants, present to varying degrees across the evaluations. Conley-Tyler (2005) makes the important observation that it is the evaluator’s roles characterised as the ‘consultant’, ‘facilitator’ or ‘director’ of evaluation that may have more influence than whether the evaluator is internal or external.

Timing, scope and sequencing of the evaluation

Despite planning that takes into account the time needed to undertake a review, it has to be recognised that schools are busy places where many teachers work under time pressures and in an unpredictable environment. This busyness invariably leads to issues around the timing, scope and sequence of evaluations.

The overall impact of the evaluation is dependent on the features of the evaluation process such as its timeliness, relevance, quality, and responsiveness to the school environment. There were both positive and negative comments on the processes used and the timing of the processes as indicated in the following indicative statements:

It was terribly disappointing that there were no further meetings or discussions to follow the first meeting.

I would have preferred a written questionnaire where I had time to consider my answers more fully.

It was a very short timeframe.

One inference that can be drawn from these concerns is the importance of explaining the time, resources and method relationships to stakeholders in order to help to manage their expectations and negotiate their requirements (Nunns, 2009). Where these explicit needs are not met, the process is diminished in its potency.

Communication of outcomes

School program evaluation comprises an accountability component and a formative component in order to inform future decision making. One of the key assumptions and expectations of teachers involved in program reviews is that they would be able to use the findings of the program evaluation to improve implementations. When this does not occur or the dissemination process is delayed, there is a sense of frustration and lack of faith in the process. In one program evaluation the evaluator had an opportunity to meet with participants who noted such things as:

It was very good that there was a final meeting with the reviewer and the faculty so that participants could clear up any misunderstandings.

In the case of the LOTE review, the faculty also had an opportunity to meet with the evaluator to discuss the report findings. This process allowed participants to have further input and to clarify areas of concern. These two faculty groups showed the greatest degree of satisfaction with the communication of outcomes.

The Expressive Arts evaluation report had a lengthy gestation period where it sat on the principal’s desk for some months before it was considered. The principal then talked the findings over with the head of faculty but made no further overt moves to release the findings more broadly. This inevitably led to dissatisfaction in the process and uncertainty about the review process. Participants commented, for example:

I would hope that the findings will be made available to the participants and recommendations brought forward for further discussion.

The review findings have not been discussed with any of the faculty. To date, nothing has been done. The recommendations … have been ignored.

Who has seen what we wrote? What points were acknowledged as worthy of real consideration?

As noted earlier, all program reviews are situated in the organisational context and culture of the school. Bolman and Deal (2008) and Schein (2004) provide extensive documentation that helps examine and consider some of the issues related to organisational cultures. From these discussions it is reasonable to conclude that the way organisations do business influences the form and impact of program evaluation. The relationships and structures of the organisation are the lived realities that can affect the evaluation processes and determine why some program evaluation reports are not acted on. Weiss’ views (1997) that evaluation is a political act and that decision making is not always a rational enterprise, serve to highlight the need for evaluators to consider how they might exert influence to ensure that participants’ expectations in relation to having access to the final phase of the process are met. Where significant findings and or policy implications are withheld or not utilised, for whatever reason, the program evaluation processes and credibility are inevitably damaged.

Conclusion

This paper provides insight into participants’ perspectives of the strengths and limitations of four differently constructed program evaluation methods in an independent school. The five key findings that were generated through documentary and questionnaire data analysis related to staff members’ understanding of the evaluation aim and purpose and their contribution to the process, the selection of internal and external evaluators, the timing, scope and sequencing of the evaluation, and the communication of outcomes. Interestingly, none of the evaluation methods emerged per se as being perceived as more effective or appropriate than any other. Rather, the study showed that participants overwhelmingly valued being provided with a clear understanding of the nature of and rationale for the evaluation and ‘having a say’ in its conceptualisation, development and conduct.

The importance of shared decision making in organisational practice has been widely acknowledged (see. e.g. Durlak & DuPre, 2008; Wikeley et al., 2005). Durlak and DuPre (2008) note, for example, that situations in which there is collaborative decision making among key stakeholders have consistently led to better implementation of organisational programs. Our study shows that, in the view of participants, shared decision making should extend to the conceptualisation and creation of program evaluation processes, which indicates that they would have welcomed playing a role in all three stages of the evaluation cycle (see Figure 1). Works by Bryson, Patton, and Bowman (2011) and Weiss (1997) highlight the potential value of involving primary users in the evaluation process. Despite the additional work that this entailed for Mathematics and Science teachers in our study, all valued the opportunity to provide input into the design and implementation of the evaluation of their programs. Participants also stressed the importance of receiving timely, ongoing and detailed feedback about the evaluation process; and knowing that the outcomes of the evaluation would be acted on. In cases where staff considered the above factors to be absent, scepticism, mistrust and a lack of engagement prevailed.

The present research study reinforces a number of issues identified in the literature in relation to program evaluation and points to ways in which the conceptualisation and conduct of other school program evaluations might be enhanced. In particular, the study supports the professional worth of involving the primary users in program evaluations. Noteworthy is the relationship between the degree of feedback about the evaluation process and the value placed on the evaluation outcomes and utility. Where evaluations are seen to be valued and the findings of the evaluation acted on in a timely manner, the higher the utilisation and utility of the practice is deemed to be.

Table 5.

Summary of key findings.

Theme	Elaboration
1. Understanding the evaluation process, aim and purpose	The process, purpose and intent of the evaluation need to be discussed with all staff before the evaluation is undertaken.
2. Staff contribution to the evaluation	Staff involvement in the conceptualisation and design of the evaluation process was very powerful for those involved and enhanced the evaluation.
3. Choice of evaluators	External program evaluators engage without preconceived ideas and biases but don’t necessarily understand the school context. Internal program evaluators understand the context but may have preconceived ideas.
4. Timing, scope and sequencing of the evaluation	How, when and where data were collected impacted on the success of the evaluation.
5. Communication of outcomes	Evaluation outcomes should be communicated with staff in a timely and collaborative manner.

Footnotes

Declaration of conflicting interests

None declared.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

References

Babbie

(2013) The practice of social research (13th ed.), Belmont, CA: Wadsworth.

Barber

Chijioke

Mourshed

(2010) How the world’s most improved school systems keep getting better, London: McKinsey & Company.

Bolman

L. G.

Deal

T. E.

(2008) Reframing organisations: Artistry, choice and leadership, (4th ed.). San Francisco, CA: Jossey-Bass.

Bryson

J. M.

Patton

M. Q.

Bowman

R. A.

(2011) Working with evaluation stakeholders: A rationale, step-wise approach and toolkit. Evaluation and Program Planning 34(1): 1–12.

Centers for Disease Control and Prevention (CDC). (1999). Framework for program evaluation in public health. MMWR; 48(No. RR-11). Retrieved from ftp://ftp.cdc.gov/pub/Publications/mmwr/rr/rr4811.pdf.

Cohen

Manion

Morrison

(2011) Research methods in education, (7th ed.). New York, NY: Routledge.

Conley-Tyler

(2005) A fundamental choice: Internal or external evaluation? Evaluation Journal of Australasia 4(1&2): 3–11.

Durlak

J. A.

DuPre

E. P.

(2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology 41: 327–350.

Fullan

(2001) Leading in a culture of change, San Francisco, CA: Jossey Bass.

10.

Fullan

(2002) The change leader. Educational Leadership 59(8): 16–21.

11.

Haigh

(2012) Sustaining and spreading the positive outcomes of SoTL projects: Issues, insights and strategies. International Journal for Academic Development 17(1): 19–31.

12.

Hallinger

Heck

R. H.

(2011) Exploring the journey of school improvement: Classifying and analyzing patterns of change in school improvement processes and learning outcomes. School Effectiveness and School Improvement an International Journal of Research, Policy and Practice 22(1): 1–27.

13.

Hattie

(2009) Visible learning: A synthesis of over 800 meta-analyses relating to achievement, Milton Park, UK: Routledge.

14.

Holmes

Clement

Albright

(2013) The complex task of leading educational change in schools. School Leadership & Management 33: 270–283. doi: 10.1080/13632434.2013.800477.

15.

Hoy

Tarter

J. C.

Hoy

(2006) Academic optimism of schools: A force for student achievement. American Educational Research Journal 43(3): 425–446.

16.

Lakomski

(2001) Organizational change, leadership and learning: Culture as cognitive process. The International Journal of Educational Management 15(2): 68–77.

17.

Leigh

(2010) Estimating teacher effectiveness from two-year changes in students’ test scores. Economics of Education Review 29(3): 480–488.

18.

MacNeil

A. J.

Prater

D. L.

Busch

(2009) The effects of school culture and climate on student achievement. International Journal of Leadership in Education: Theory and Practice 12(1): 73–84.

19.

Masters

G. N.

(2010) Teaching and learning school improvement framework, Camberwell, VIC: Australian Council for Educational Research.

20.

McDavid

J. C.

Hawthorn

L. R.

(2006) Program evaluation and performance measurement: An introduction to practice, Thousand Oaks, CA: Sage.

21.

Miles

M. B.

Huberman

A. M.

(1994) Qualitative data analysis: An expanded sourcebook, (2nd ed.). Thousand Oaks, CA: Sage.

22.

Nunns

(2009) Responding to the demand for quicker evaluation findings. Social Policy Journal of New Zealand 34: 89–99.

23.

Rapley

(2007) Doing conversation discourse and document analysis, London, UK: Sage.

24.

Robinson

V. M.

(2010) From instructional leadership to leadership capabilities: Empirical findings and methodological challenges. Leadership and Policy in Schools 9(1): 1–26.

25.

Scheerens

Demeuse

(2005) The theoretical basis of the effective school improvement model. School Effectiveness and School Improvement: An International Journal of Research, Policy and Practice 16(4): 373–385.

26.

Schein

E. H.

(2004) Organizational culture and leadership, (3rd ed.). San Francisco, CA: Jossey-Bass.

27.

SurveyMonkey Inc. (2013). Survey monkey. Palo Alto, CA: Author. Retrieved from www.surveymonkey.com.

28.

University of Tasmania (2010) Social Sciences HREC, Hobart, Tasmania: Author.

29.

Weiss

C. H.

(1997) Evaluation: Methods for studying programs and policies, (2nd ed.). Upper River Saddle, NJ: Prentice-Hall.

30.

Wikeley

Stoll

Murillo

De Jong

(2005) Evaluating effective school improvement: Case studies of programmes in eight European countries and their contributing to the effective school improvement model. School Effectiveness and School Improvement: An International Journal of Research, Policy and Practice 16(4): 387–405.