Assessing Impact Amongst Chaplains in a University Setting: Phase Two of an Action Research Project

Abstract

This article, the result of an Action Research project, describes the process of creating and testing a resource for assessing the contribution of chaplaincy in a British university setting, and the resultant insights and outcomes: organisational and individual learning, changes in chaplains’ attitudes to monitoring and evaluation, and a resource which is perceived as having benefits and limitations. This article considers the evaluation process as applied to chaplaincy and offers a model for further testing.

Introduction

The requirement for chaplains to give an account of themselves, to demonstrate ‘value’, is felt across all the sectors in which chaplaincy operates (Slater, 2016) and the perceived need for it within higher education (HE) (Aune et al., 2019) provided the impetus for the action research project from which this article arises. The growth in the call for outcomes-based approaches in chaplaincy has been discussed widely in the literature with some examples of different evaluation and research practices (Peery, 2021; Schmid & Sheikhzagedan, 2020; Walton & Körver, 2017) but also with some expressed reservations. A university chaplaincy team's anxieties and the difficulties they perceived around identifying contribution were uncovered in a first phase of this action research project, described in a paper by the same authors, co-published in this issue. ¹ That first phase also helped chaplains to recognise the value of an evaluative process and their desire to engage in one. Such a process, they believed, would help them meet institutional requirements for accountability, enable the development of their practice, increase individual chaplains’ awareness of the diverse contributions of their colleagues, satisfy their sense of personal responsibility, and provide an impetus for exploring the theological understandings and imperatives which inform their work.

The processes involved in demonstrating value and usefulness, in identifying outcomes and ‘impact’, are complex, whatever the sector or function – with processes, metrics, and indicators matched to the particular context. This applies in relation to chaplaincy, with methods used to tease out ‘impact’ varying across the sectors and according to specific context. Healthcare chaplaincy has been more likely to employ quantitative designs and the measurement of observable outcomes to assess the effectiveness of any intervention, and such approaches have become more prevalent (Fitchett et al., 2014; Weaver et al., 2008), even if there is some suggestion that individuals have struggled to carry out such evidence-based research (Handzo et al., 2014; O'Connor, 2002). The perceived requirement for verifiable evidence and scientific approaches in healthcare chaplaincy, however, reflects its integration into the healthcare system (Handzo et al., 2014; O'Connor, 2002). Where there is more separation between chaplaincy and its host institution there are arguably more possibilities for developing bespoke systems and methods. Whatever the sector, it is suggested that approaches to outcome evaluation in chaplaincy should be careful to take account of the particularity of the nature of chaplaincy in order to ‘preserve its integrity’ (Damen et al., 2020). The use of case studies and of qualitative data, of observation and client views, are amongst approaches advocated in the literature and used in some examples of outcomes-based approaches (Schmid & Sheikhzagedan, 2020; Todd & Tipton, 2011) – while both Aune et al. (2019) and Ryan (2015) recommend Most Significant Change (MSC) as a methodology (Davies & Dart, 2005) for identifying chaplaincy's impact.

Outcomes-based approaches specifically in HE chaplaincy are not commonly reported. Barton et al. (2020), exploring the contribution of campus chaplains in the United States, for example, claim that little is known about ‘how chaplains conceptualise their contributions’. The current article gives an account of action research to develop and test a ‘resource’ for assessing chaplaincy's contribution within a British university and describes the outputs and outcomes of this process. ² The principles and methods employed to enable the production of an effective resource acceptable to and useable by the chaplains themselves involved dialogue, leading to the development of individual and group knowledge; the exploration of values and purposes; discussion and debate about the meaning of gathered data; and an explicit commitment to reflection and learning. Effectively, the process for designing and testing a mechanism and tool for the evaluation of the chaplains’ contribution itself became an evaluation exercise, as much about improvement as accountability, akin to the action evaluation (AE) described by Friedman and Rothman in Bradbury (2015). Further, the approach shared features in common with developing practices in evaluation which recognise the complexity of achieving change, the value of continuous inquiry, and ‘collaborative meaning-making’ (Sharp, 2018).

Methodology: Shaping an Approach

This action research started with an explicit commitment to listening and dialogue, to sharing understandings, and to exploring assumptions: principles which were followed throughout the project. The early stages of the project used MSC (Davies & Dart, 2005) as a ‘way in’ to considering questions of impact. MSC was not, however, applied in its complete form, with stories about change being provided by chaplains themselves rather than collected from those whom they serve, and without a process of story selection involving wider stakeholders. MSC did, however, inform aspects of the thinking and approach throughout, as in the time devoted to conversation about what constitutes desirable change and how to attribute meaning, and the focus on group dialogue and reflection to reveal values and beliefs (Willetts & Crawford, 2007).

Following early focus groups looking at attitudes to and perceptions of impact assessment, and meetings to share stories, conversations were facilitated to explore further the work of chaplaincy in this HE context and to enable chaplains to express and share their perspectives and understandings with one another. Transcription of meetings meant that learning could be captured and key findings, concerns, and areas of interest together with unresolved questions could be fed back and discussed at the next meeting in an iterative process, with the purpose of helping to build a shared understanding of the work and a basis for an evaluation process.

The findings of the initial work – phase one of the project – about attitudes to, and the possibilities for, outcome evaluation have been published separately in this same journal issue (JPC&C 78:1). Chaplains identified a number of factors which caused them to be hesitant about engaging in impact measurement and outcomes-based evaluation, both in relation to chaplaincy more generally and specifically in their HE context. These focused predominantly on beliefs that standard methods could or should not apply to chaplaincy work, but also on the practicalities of evaluating as chaplains. Despite these objections, the first phase of the research project also revealed a common belief amongst chaplains that evaluation of impact was nonetheless an important exercise for them to pursue, both to speak into their institution and also for moral-theological reasons around the need for accountability.

Methodology: Creating a Resource

This next phase of the project therefore sought to develop and test a means for evaluating chaplaincy work in HE which could enable chaplains to demonstrate accountability in their work, both to a university and to their sending faith institutions, but which also offered the possibility of mitigating the various problems with aspects of impact assessment and evaluation which the chaplains had identified.

The team's recognition of, and interest in, the potential of evaluation for both accountability and improvement informed the methods and approach used in this process of development. The various findings about the work and roles of chaplaincy in this specific HE context provided foundational principles and some reference points for the resultant two-part resource, an output of the action research process, which constituted a ‘map’ of chaplaincy activity and an evaluation form.

A ‘Map’ of Chaplaincy

The first part of the resource was a list of the areas of activity in which the chaplaincy team – either individually or corporately – engaged on a regular basis (Figure 1). Lack of a clear, articulated understanding of the purposes and roles of chaplaincy and of the ‘domains’ in which it works to achieve change was recognised by the chaplains participating in phase one of this project as a barrier to attempts to assess impact. More broadly, the importance of being clear about purpose, and particularly about the specific purposes within the context in which the chaplaincy is operating, is stressed by Ryan (2015) whose report on chaplaincy in the UK focuses on impact assessment.

Figure 1.

‘Map’ of chaplaincy activity.

Accordingly, a list was created to offer chaplains a quick method of identifying how each aspect of their work, the impact of which they might choose to evaluate, fits into the wider picture of the responsibilities shared across the team. At the request of the chaplains involved in the study, this list was developed into a visual form – similar to a mind-map – and became quickly known by them as the ‘map’ of activity.

The content of the map emerged through intentional listening on the part of the research facilitators, noting the types and examples of work which chaplains themselves mentioned across all of the phase one discussions, subsequently ordered and categorised into a two-tier hierarchy: broader areas of work and specific repeating activities.

In line with the action research methodology of this project, the map was tested in a focus group and adjusted in light of feedback, leading to changes in content and also the addition of a very brief indication against each activity of how that work might be considered a potentially impactful contribution and be thought about in relation to the measurement of success. These indications were akin to the ‘domains of change’ used in MSC to guide the collection of stories, while not limiting the possibility of other unanticipated areas of impact (Davies & Dart, 2005).

An Evaluation Form

The second aspect of the resource was a form to guide the chaplains through an impact evaluation process for their work. It was developed in response to the themes which emerged in the facilitated conversations during phase one of this project. Broadly, these themes spoke of the need for a flexibility in approach and language, to enable a diversity of types of activity and of impacts to be recognised, and for a diversity of methods to be used in measuring them, as deemed appropriate to the chaplains themselves. The form was therefore structured to allow for that broad diversity to be included, while also taking chaplains through a sequence of steps in defining what might be appropriate measures of impact in each case.

The form asked the chaplain completing it first to set out the basic details and broad nature of the activity and to describe its category (with reference to the chaplaincy map). It then posed two open questions:

What is the intended purpose/s of the activity?

What difference do you want the activity to make?

No requirements were placed on chaplains in filling out each of these sections, and no further restrictions were made around articulating the size or scope of the activity. This open quality allowed ownership of purpose and intended impact, including the scale of that impact and the language used to describe it, to be held completely by each chaplain.

Having articulated the nature of the activity and the intended purpose, subsequent questions were posed which encouraged a focus on equipping the chaplain to find the means of describing the impact of their activity, using an evidence base, but without dictating the nature of that evidence:

How will you know a positive difference has been made?

How will you go about noticing and recording the difference which has been/is being made?

What other information will help to record the reach and impact of the activity? How will you gather and store this?

The form therefore allowed chaplains themselves to offer the language and the measures of success – qualitative, quantitative, or mixed – that they believed appropriate to the work, while ensuring that the necessary thinking and planning went into the gathering of those data.

Two final, summary, questions were posed in order to allow chaplains to take a broader, reflective approach to the activity after its completion or at particular milestones in its delivery. These asked about the extent to which their activity achieved the intended purposes and whether, if repeated, it could go further in achieving those purposes:

How successful do you consider the activity to have been?

Reflecting on the successes and limitations of the activity, what do you consider might help achieve greater impact in the future?

The form was designed so that it could be used both individually and jointly. In order to address concerns raised around the time this evaluation work might take, the form was kept to a small number of questions. Whilst the principle of not imposing indicators which might fail to capture the changes which chaplains felt to be most significant or illustrative of their work was upheld, some guidance notes were offered in two areas:

• hyperlinks to key institutional strategic documents so that activity purposes could be considered in light of these, amongst other drivers;

• a long (but not exhaustive) list of suggestions about how change might be noticed during or after an activity.

These suggestions spanned a range of types (e.g., direct feedback by email or in conversation, quality of conversation, increased engagement, changes to policy and practice, development of skills or theological understanding), responding to the chaplains’ desire – which emerged clearly in phase one of this research – not to focus primarily on engagement numbers.

Methodology: Testing the Resource

Between October 2021 and April 2022 chaplains were provided with this two-part resource and asked to test it with a self-selected sample of work areas, events or programmes. Chaplains were offered the opportunity to meet with the research facilitators, both individually (by request) and also in scheduled group review sessions, to discuss their use of the resource, to raise queries, and to share experiences and good practice with one another.

In total, seven meetings took place. During the review sessions, chaplains were invited to comment on the quality and value of the resource as a means to articulate impact, and to make suggestions for developing the resource that they believed would improve it for their purposes. As a result of this, changes were made, and the revised resource was then shared for use by the chaplains in advance of the next review session.

At the end of this testing period, the chaplains were invited to meet as a whole team to discuss the resource, to gather for a semi-structured focus group, and then to meet again to discuss how the work might be used to speak into the university's own reporting structures. In these meetings, chaplains were asked to reflect on the resource in its final form, on their responses to using it, and on how their use of it had either supported or changed their views on the viability of evaluation in HE chaplaincy.

Outcomes of the Action Research Process

Participating in this research led to a range of responses from amongst the group of chaplains about the value of the work. Across the research exercise, perceptions evolved and changed even in relation to each individual. However, there was agreement that the process of discussing the assessment of impact and the piloting of the resource resulted in various outcomes connected to the chaplaincy's work, including the building of team understanding, the articulation of purpose and contribution, the identification of a range of relevant measures, and the implementation of evaluation for improvement. The conversational and participatory nature of the process was significant in relation to the achievement of these outcomes, as is evident below.

Understanding One Another

One benefit of the project, raised on a number of occasions, from the early stages of work up to the end point of the research, was the positive impact on the chaplains themselves as a group of individuals working with a diversity of roles, job descriptions, and lines of accountability, but asked to form one chaplaincy team within one institutional structure.

Chaplains acknowledged that at times they can fail to notice or to understand what and why other chaplains work in the ways that they do, or to misinterpret those things.

you can assume things when you’re working with people and do not really have the opportunity to really uncover what's going on underneath ³

The impact of this, chaplains recognised, could be damaging to the team.

when we don’t understand what motivations people are bringing, or what priorities within the chaplaincy's role are more important to one person than another, I think it's very easy in a team that has a lot of pressure on it … to become irritated when people don’t respond in the way that we would with our tunnel vision set of priorities and notions about what chaplaincy is. And what's destructive to that is when we push that down rather than bring it out, and bring it into the light, and relate in a positive way.

The approach taken in this project, which encouraged chaplains to speak with one another about their work and to describe together in-depth examples of where they felt their work had achieved change and impact, brought greater understanding and appreciation of one another's contributions.

having that guided conversation … helped us to get to know each other's work in a very different way. And actually, that was really positive

that has to be a good thing in terms of us communicating with each other about what it is that we’re doing. It has to be a good thing that even, at least to us, it's quite visible what it is that we’re doing and the impacts that we’re having

Allowing one another to talk about why each chaplain had chosen their particular examples when discussing the impact of their work also proved a starting point for conversations about theology, calling, motivation, and vocation, allowing chaplains to gain a deeper understanding of how and why they work in the ways they do and in the areas they do.

I valued very much the conversations. And actually having the chance to talk around why we do what we do has been really valuable within a team.

[It is] important to clarify the various motivations and priorities that we all, as people of faith, bring to the roles of this team

The benefits of this for team dynamics were widely anticipated.

It has also helped me grow and have more of an appreciation that there are a lot of similarities between chaplains of different faiths as well. Why we do what we do, we've probably got far more in common than is different … It's quite helpful, actually.

These improved dynamics were seen as having implications for the work, with more effective collaboration, particularly between those from different faith traditions, and with shared intentions and a clearer rationale for what they can offer as a team:

just giving over that time to have a probing conversation will help us work better in the future

I think, possibly within the team, there's a greater sense of intentionality … there's a growing confidence within the team around articulating why it is we're doing something as opposed to doing something else.

it helps us to be able to move in a direction where we might possibly look at whether we need to do as much as we do, because actually, we are providing that sense of change or that sense of experience for the community. And if we're doing it through that one activity, do we have to do it through a different activity just because we're of a different faith?

Speaking With Purpose

The research process, by calling on chaplains to find ways to explain to others the purpose and the value of their work and the measures by which its impact might be understood in meaningful ways, challenged each chaplain to interrogate whether they had fully themselves taken time to discern purpose and intended impact, and to understand why it mattered that they did.

But the people are really struggling. And actually, I've not actually sussed out why I'm doing what I'm doing and why I’m not doing what I'm supposed to be doing, whatever. It's a really important exercise and I think gives me a greater appreciation for this work

By providing the means for them to think on these matters, the project then allowed chaplains to be more focused, more intentional, and more articulate about their work.

this exercise has actually helped me really think why I'm doing it and the purpose behind it. I found that quite useful, even though I haven't got very far.

it's been quite liberating to complete the form because it allows me to enrich my own knowledge as to why I do what I do and then also be able to share that with others.

It did make me reflect on what I was trying to achieve by running the sessions. And that's one positive that has come out of it.

As well as for individuals, this focus would be seen also in the work of the chaplains as a team.

I think one of the things we might learn from this process, as a team, is not just to be gung-ho about what we do, but to have a sense of a shared understanding of what the purpose of it is, and who it is for, and what does success here look like. And then that might enable us to be a bit bolder about saying, this isn’t working, let's try something else. And it might help us be more insightful about what kind of things we could adapt it to that might work better

it's enabling purpose to come to the fore

This would allow chaplains to think through what they do – to assist planning and prioritisation in ways informed by meaningful data, with a focus on those people which chaplaincy should be serving.

I just feel that when if you are assessing the impact that you’re making, in the back of your mind you are thinking about the audience for whom you’re collating this information. How might I do this differently next time as a result? So that we're not just getting into the habit of doing the same old thing because we think that that's what people want

The benefit to the team of sustaining this thinking and dialogue was therefore recognised.

That conversation within the team has been helpful and needs to continue.

Building a New Narrative

The many and varied concerns which chaplains expressed during phase one of this project, when discussing their perceptions of evaluation and impact measurement, were focused in large part on fears that quantitative measures would offer understandings of their work which missed the real point of it. Quality, not quantity; relationships, not numbers of interactions: these emphases were believed to sit outside the mainstream concerns of departments in a university context and would therefore require a different kind of impact assessment to the standard ones.

Chaplains widely recognised that the qualitative approach taken to evaluation in this project addressed those concerns and offered a new model – one which would allow chaplains to speak more meaningfully about their work.

I hope that one of the things that comes out of this process that we can value is that sometimes our work isn’t about doing work with the most number of students. But it is about reaching and supporting some of our most vulnerable students

This whole experience, really, I found it really helpful. I found it, in some ways, quite exhilarating actually. Because it's been quite enjoyable to see how we can move towards something that may allow people to better understand what we do

By having the tools properly to articulate chaplains’ contributions, the process of evaluation would become one which they believed would be worth pursuing.

you can find ways to be able to do those measurements as well and I think it's important to reflect on that as opposed to saying that it's not possible to measure impact.

Achieving this began by providing a process and the resource for chaplains to think for themselves in new ways about their work – to provide the space and the means to develop their own language for understanding and articulating contribution.

I think it's impinged in the way I've articulated what I want to do and putting in writing, which maybe it might have just been simple thoughts in my head, but not as detailed as I would have without the resources that we have now.

We were speaking about language earlier and I suppose for me, it's helped me to be able to just diversify how I can … articulate. That is why … And to myself as well. Why I'm doing what I'm doing. It's been helpful to me

Enabling chaplains to do this thinking for themselves, the process and the resource which emerged from the project also allowed them to articulate to others the purpose and the value of their work more effectively.

I think there is something to be said for articulating things, which is really what this whole process has become, about articulating things that probably there aren’t opportunities to articulate or it's felt that perhaps it would be too indulgent to articulate

I think in terms of what, certainly this exercise, this project, can help chaplaincy to do, which is telling the story, isn’t it, about what chaplaincy does and how it does it? And how that influences university in a whole range of ways

Many of the chaplains spoke negatively about the norms of language in evaluation processes. This extended to broader criticisms of language used within organisations, including universities, to describe their purposes and goals.

I am one of those people who struggles with being able to articulate well what we do in places where management speak is a norm.

Chaplains were open to seeing their role as one which was called to challenge this, and saw this project as offering the opportunity to do so – both participating in the processes of evaluation and also providing a new way by which to approach it, using a new language.

part of this whole process is that we’re gaining the language that will help us articulate that interface, that hopefully will give some hope and comfort to those on the other side of it

it was helpful for me in terms of reflecting on an activity, why it was spiritually important. But also trying to straddle the need for the institution to also have some understanding of the impact that my activity or chaplaincy activities were making.

it's helped articulate. Because when you sit down to do it, it forces you to sit down and really think about it.

Participating in this project provided chaplains themselves with ownership of the language they used to demonstrate their contribution and the purpose of that contribution. This licence to use language that was more familiar and more appropriate to the work and ministry of people of faith was fundamental to the chaplains’ engagement in the process and their positive reactions to it.

To say, first of all, why am I doing this thing at all? What's the purpose, the value of this activity in my own language? Language of chaplaincy or the language of my denomination or whatever that might be. And then building on that to then think through how you therefore will be able to say, that actually achieved what we set out to achieve or it didn't achieve

Using Indicators and Measures

The process and form gave permission and encouragement to use a range of different measures in order to understand contribution – gathering a diversity of data and bringing them together to understand the whole. This was found to be particularly helpful in some aspects of chaplaincy work.

That demonstration might come in terms of a quotation from an email saying, ‘that was a really helpful encounter for me, thank you.’ Plus, some data about the numbers of people who come to the event on average or over the course of a term. Plus some photographs of people having a wonderful time … during community lunch … to have that breadth of demonstration of the kinds of things which are being achieved in that context allow us to reflect on the significance of something to the community

Where chaplains were in a supporting role at an event, or working collaboratively with others outside of the chaplaincy team, but not leading the work, quantitative measures might offer little understanding of how such a role added value, but qualitative assessments could do more.

But it's not having to claim it all for ourselves as a department. I think that's perfectly valid and a different model, but one which fits.

A further area of complexity the project enabled chaplains to address was where the impact of chaplaincy work might need to be interpreted according to context. The capacity to take a complex mixed-methods approach to evaluation allowed chaplains to develop ways of articulating the contribution of their work without needing to homogenise all their stakeholders, offering the opportunity to recognise the unique quality of impact even when activities might themselves look similar.

Yes, I suppose one of the key questions to telling those stories is to identify what success looks like. And that's one of the things we work with students on because what success looks like for one person is not what success looks like for another person.

Evaluating for Improvement and Improving Evaluation

Chaplains agreed that this project allowed them to think in more depth about the effectiveness of some of their work. This could provide challenge, when the success of an activity did not appear to match the measures of impact discerned by chaplains in planning that work.

But actually, there still seems to be a real disconnect between what the student needs were and what we were offering

The process of forming the ‘map’ helped chaplains to identify gaps in their work and reminded them about activities in the past which might be renewed or resumed, with some benefit to the community.

[It] triggered for me memories of other things that we've done which as well were really great and for one reason or another, have fell out of being done. And I think my problem is that it's expanded what the opportunities or areas are … And highlighted the areas where we're not really doing much at this stage where we definitely could be engaging more.

… it helped identify the things perhaps we weren't doing rather than just to focus on the things that we are doing. It's just where the gaps are perhaps as well. The map helped me do that … the map really helped me look at other areas and think, what other things could I possibly be doing rather than trying things out?

As well as providing chaplains with the opportunity to shape the language – and therefore the measures – by which their work was described, the approach to evaluation enabled by this resource offered chaplains the freedom to evaluate their work over a period of time and to monitor and evaluate for improvement. Moving away from simple, quantitative measures allowed the chaplains to consider how their impact might be seen across a significant period, or across a repeated series of events, activities or interventions. This, they believed, provided the means both for a more meaningful evaluation of their work and for a more complex consideration of how chaplains could ensure their work achieved the greatest benefit.

I did a … Student Welcome in September and I did a form afterwards for that reflecting on it. And then the same … Student Welcome in January, when planning it, it was helpful having oversight of the previous one.

And it is certainly a challenge to think, if the student turns up to future community lunches, how can I engage? How can I get that student engaged more with other students? I actually found it quite useful in that example of thinking, how can we create greater impact on future occasions

Looking at the resources now and the planning side of it, I found it really helpful because it made me sit down and really think what I wanted my participants to get and what I wanted to see in them. Maybe not at the end of one session, but later on.

A Viable Resource

Using a resource shaped by their own understandings and practices enabled the chaplaincy team to engage with an evaluation of their work and its impact. The open qualities of the evaluation form offered chaplains the ability to describe contribution in ways they saw fit for that work and its goals, using language which they themselves chose. Holding this openness alongside clear guidance and the ‘map’ of chaplaincy's areas of activity ensured some focus to the work and helped in the articulation of purpose and impact.

Assessments from the team offered broad approval of this approach.

The paperwork that's come out of this, the resource, I will find helpful

I think we've ended up with something that's valuable to help us reflect as a team.

it will help me think about not just what I put on, but to be able to understand the quality of my offering as well

Reflecting on both the challenges and opportunities, chaplains were clear that that the form and the discussions around it had encouraged them to think more intentionally about the contribution of their work. The resource would allow them to make more strategic and informed decisions, and specifically consider how to increase the impact of their work. This, they recognised, was already visible in their activity.

I found the resources quite helpful for myself to reflect on what I felt I was offering and what I wanted to offer and whether it was received and whether it made an impact

I think it was completing the form that made me think about, have we progressed this?… And it was quite helpful to be challenged to think, how could this have a greater impact?

There was a recognition, too, that the resource might not only enable chaplains to improve their offering but also demonstrate to others the reach of their work.

This could then be a useful tool for us to append to reports we might offer to the university, or just a helpful thing for us when we’re doing work around anything that involves reporting or business planning, or setting up corporate or individual workplans

Whilst the evaluation resource was found to be helpful in identifying gaps in provision and potential improvements to their offerings, however, there were some limitations in its potential to identify how chaplains might prioritise their activities.

I haven’t gone through one of those processes or looked at what we've done and said, actually, I don't think X doesn't fit here. What it doesn't help me do is work out, I guess, the priority side of it

had we now stopped doing some things because of doing the forms? And for me, the form doesn't do that, I think. Maybe because it is broad

The time demanded by the complex considerations involved in the qualitative approach was also perceived as a limitation.

I think for me, it's just time to sit down and do it and reflect on it.

Further, some questions did remain about how far impact could be identified and described in some of the complex and nuanced situations in which chaplains work, reasserting some of the early reservations expressed by the group around impact assessment in particular chaplaincy contexts.

For those on the edges, those are the ones that we often end up connecting to. I don’t know yet if what we’ve got helps us celebrate that actually

But it's like something's moved, not just in the person but in yourself as well. And that encounter, it's really hard to put into words. So I don’t know if others can relate, but there are stories like that, but I don’t know how to share them actually.

Measuring Chaplaincy's Impact: Giving an Account

The impetus for this project in the first instance was the desire to demonstrate the difference made by chaplains within a British HE institution and to find a way to document the chaplaincy's impact, in line with the approach advocated by Aune et al. (2019): ‘an account of their contribution that others can appreciate, yet that does not contradict the very values that make them distinctive and of worth.’ (p. 118). The first phase of the project, exploring the chaplains’ perspectives on measuring impact and outcomes-based approaches revealed a desire to engage in evaluation of their work and to demonstrate accountability to their various stakeholders. The conversational, participatory process which built on these findings led to the development of the resource described in this article.

The resource provides a mechanism which enables the chaplains to reflect on their work, to identify their goals, to make adjustments and improvements, to be more intentional in aspects of their work, and also to find a means to articulate their offering in ways which can be understood by others outside chaplaincy – the latter being the originating aim of the project and an outcome which is also sought in the literature, as in Aune's comment above (Aune et al., 2019; also Kevern & McSherry, 2016). However, there remains some complexity around this.

An improved capacity to articulate impact and purpose to an external audience (and the benefit of being able to do so), is evident in the experience of one chaplain, who, in putting forward a bid for funding before the end of the project, was able to engage more effectively with the processes and expectations in place in a different organisation.

…And actually, I went into the form. I did that to say, this is why we’re doing it … Because I’ve actually sat down and I've vocalised why I'm doing it and it's not just because I think gardening's good. It was good to be able to actually say, I think there's a whole spiritual practice around coming to terms with life and death. And there's stuff around how it's community building that helps bring people together.

In relation to the university, the host institution where the chaplains are chiefly seeking to give an account of themselves, there is evidence of increased engagement with the university's strategic planning documentation, with some attempt to align with its content.

I suppose the benefit around the process of this has been the primacy of the strategic document that the university has produced. Questions around, how does what you're doing speak to that, as well as speak to the other documents we have? And I think that has been helpful, because we’d all read it, but it's helpful in embedding in the day-to-day what that is looking for.

Nevertheless, there remains a lack of confidence, related in part to a concern that the open approach to establishing measures of impact, and the freedom of language enabled by the evaluation resource, may limit the capacity of chaplains to speak effectively into the organisation (with its own more specified models and measures and its different lexicon and understandings), however helpful the evaluation resource might be for the chaplains themselves.

we're not always brilliant at using the institutional language. And I think I'm still left in that place of uncertainty.

it depends who we're evaluating for as well. When I'm doing the evaluation, if I'm doing it for my own self, then it's very helpful to expand on that list and put in more. But actually, it might not be … if this process of evaluating it is for someone else. When I've completed it, I've used it as a form for me as much as it is for anybody else, if that makes sense

for it to be meaningful in achieving its purpose, it's not just that the forms are written, it's that the forms are shared and understood and we have extracted out of them that which will be helpful.

The chaplains also struggled to identify existing opportunities and forums for disseminating what they had uncovered via use of the resource.

I think I’d just like to be clear about how we end up using the tool and what its best function is

I think one of the question marks I still have, and I'm recognising it because I'm still struggling with it, I think, is how we communicate to those within the structures who have interest in what chaplaincy do … And finding the right way to communicate what we do and where the value of it is, I do struggle.

Specifically, chaplains articulated the need for a process whereby those stakeholders would have the opportunity to look at the chaplains’ work in the evaluation of impact and to reflect back to the chaplains their views on it. Inviting that kind of involvement from stakeholders and different levels of authority with the institution is a feature of the MSC methodology which chaplains recalled as being attractive and which they believed could be integrated into their own processes of evaluation to some benefit.

In the original process, there was something that actually we received back. And I think it was in the receiving back, that's that missing gap

…And it is because of the pandemic and everything else. It was obviously that part of the process was inviting that other layer of people to come and look at those stories and feedback. And I think because we haven't been able to connect what we've been doing into the systems and the structures of the institution … do we know what we're saying is helpful to them to help them understand what we're doing, if that makes sense? … It's that connect. I don't think we have the connection or the relationships now maybe sometimes with people who are asking or have expectations.

Concluding Discussion

The project and the resource which was developed through this action research process achieved success in enabling a chaplaincy team to engage with outcome evaluation. It also highlighted the value of a systematic process for identifying different types of work and activity and the goals of that work as a precursor to measuring impact. The chaplains’ desire to talk about the work, to reflect and learn together, and to refine the chaplaincy offering, also demonstrated their capacity for, and commitment to, evaluation for improvement. The chaplains were comfortable in, and wiling to give their energy to, conversation and dialogue as a route to this and the project demonstrated something of the value and efficacy of such conversational methods, as asserted, too, by MSC as well as other participatory approaches (Willetts & Crawford, 2007; also Ryan, 2015; Sharp, 2018).

This attention to process has particular resonance in the context of chaplaincy where presence and relationships are paramount. Nevertheless the project also demonstrated an interest in outcomes – the chaplains were invested in achieving positive change but also in being able to identify and articulate this impact to the institution. The project enabled the development of a language and potential methods for enabling them to do so (even if there was no straightforward process in relation to those elements of their ministry with a longer-term and spiritual perspective). These methods were not imposed, and the chaplains had agency in determining how to deploy them, according to the nature of the particular task or work activity.

In terms of aligning with institutional priorities and stakeholder expectations, the chaplains showed an interest in this, and there was evidence of increased engagement with the university's ‘enabling strategies’, but there was also some difficulty. Although the resource had some success in enabling chaplains to engage more closely with institutional priorities through the embedding of key documents in the evaluation process, chaplains continued to remain somewhat unclear on what was expected of them. This was matched by a sense that their work was not necessarily understood by the institution, because of a lack of conversation and communication and insufficient connection between chaplains and the institution's ‘management’. In that context, chaplains were aware that not only the standard metrics but also the language used by the institution to express ‘value’ did not always match their own articulations, and this would continue to be problematic without the means to enable meaningful conversation. How to bring together the language of chaplaincy and the language of the institution, and to identify areas of common interest about what is deemed desirable change in the context, continued to present some complex questions.

For chaplains, this did not diminish the value of the resource as a means to enable them confidently and competently to articulate the contributions of their work using their own language and methods, influenced, too, by their own knowledge of institutional priorities. However, it did indicate the likely value of a further stage in the process, during which chaplains and senior leaders in the institution could be brought together in dialogue.

The approach advocated by the MSC methodology involves just this kind of dialogue between different ‘levels of authority’ and brings together different stakeholders to discuss accounts of change (Davies & Dart, 2005). When the research was initiated it was envisaged that this would be part of the process, but it proved impossible to incorporate it during the lifetime of the project. A continuing desire for such a dialogue did, however, emerge in the chaplains’ reflections on the project and the process. A meeting with the institution's management is now planned, in which chaplains will bring their narratives around chaplaincy's impact (as uncovered by the evaluation resource) and where different perspectives will be presented and explored. It is hoped that this additional stage in the evaluation process, to be undertaken annually (a commitment made following the project), will improve mutual understanding and bring further institutional value to the evaluation resource. The first iteration of this next stage of the evaluation process will constitute phase three of the action research project and provide new data.

Nonetheless, findings in the literature and from this project suggest that some discrepancies will remain. The language of universities and the measures used in the context of HE will only capture those aspects of change which accord with the prevailing norms about what universities are for and societal expectations around HE, along with particular notions of ‘impact’ and how it happens. Chaplaincy's engagement and ambitions will be less influenced by these current and immediate concerns and priorities (“Faith in Higher Education”, 2020). There is also some suggestion from this project's findings that the chaplaincy may choose sometimes to highlight these differences to the university, modelling a different way and articulating a narrative which does not conform to that of the institution.

So we return to the complex relationship between chaplaincy and outcome-oriented approaches, between chaplaincy and measures delineated elsewhere, which do not fully reflect chaplaincy's concerns because those concerns are driven by spiritual interests. Some discrepancy between the chaplaincy and its institution in terms of language and how it articulates its contribution reflects the nature of chaplaincy, the impetus and rationale for it, its values and virtues: the need to retain chaplaincy's ‘integrity’ will always impinge on attempts at measuring its impact (Damen et al., 2020; also, Aune et al., 2019).