Abstract
Community-based programs involve participants whose perspectives are shaped by the social and geographical context in which they operate. Evaluations of these programs should capture this context and incorporate the diverse perspectives of participants in data collection and findings. This is demonstrated in an evaluation of a community-based disaster resilience program in New South Wales, Australia, in which six community case studies were developed from geo-social statistics, site visit observations and in-depth interviews with key community informants. The evaluation used the case studies to identify commonalities and differences in how the program was understood by participants in different communities and by external stakeholders to the program. The findings showed how context informs a more comprehensive understanding of a community-based program’s impact and how this may vary according to the context. This evaluation reinforces the relevance of Realist Evaluation principles which emphasise the importance of context in evaluating programs.
What We Already Know
Context is a factor for consideration in evaluations using Realist Evaluation methods. Social factors apply in community-based programs. Mixed-methods approaches to evaluation have the benefit of drawing data from multiple sources to provide a range of perspectives to the evaluation questions.
The Original Contribution the Article Makes to Theory and/or Practice
Context includes geo-social characteristics when a program is location or place based. Variation in these geo-social characteristics may result in variations in how the program is understood and applied, and the outcomes it generates within a particular community. Case studies, which use observation and phenomenological analysis as social research methods, can obtain data to consider geo-social characteristics impacting on a program.
Introduction
The social, cultural, economic, political and physical/geographical conditions that surround a program form its context. Programs that are community-based can be especially impacted by context because they are place based, that is, they occur in communities. Accordingly, contextual factors – social, cultural, economic, political and physical/geographical factors – influence their implementation and achievement of intended outcomes. Realist Evaluation principles recognise the importance of context in understanding what works and why in a program. These principles are therefore highly relevant to the evaluation of community-based programs.
The evaluation of the Community Resilience Teams facilitated by the Australian Red Cross, in line with realist evaluation principles, examined how local context affected the implementation and outcomes of these Teams across six communities on Wonnarua Country, Gumbainggir Country and Bundjalung Country in New South Wales, Australia. The evaluation sought to answer questions regarding the design, relevance and effectiveness of Community Resilience Team activities, considering the diverse experiences of community members in different locations. The evaluation also examined stakeholder outlooks on the Community Resilience Teams and compared them to those of community members.
The findings highlighted the importance of moving beyond one-size-fits-all evaluation approaches by accounting for community-specific contexts. In doing so, this evaluation critically examined the assumptions about the program design and delivery contained in its theoretical frameworks. This enhanced the utility of the evaluation for future Community Resilience Team program development.
The Importance of Context-Sensitive Evaluation
Context-sensitive evaluation recognises that the impacts of programs are not merely the products of the programs themselves but are determined by the environment in which they are implemented. Variability in outcomes often occurs because of differences in social, cultural, economic and environmental conditions rather than the characteristics or implementation of the programs alone (Alderman, 2015; Bamberger et al., 2012; Roche, 1999). Context sensitivity recognises that programs do not operate in a vacuum.
Social development programs, educational interventions, health initiatives and community development projects are embedded within complex systems that affect and are affected by a myriad of contextual factors (Kelly & Rogers, 2022). These include the social and cultural norms of a group or community, the prevailing resource and economic conditions, the political dynamics and the physical/geographical environment. These factors alter the pathways through which programs exert their effects, modify the behaviour of participants and achieve their intended outcomes (Patton, 2011).
Realist Evaluation, as developed by Pawson and Tilley (1997), views the interactions between context, mechanism and outcome as essential for determining what works, for whom and under what conditions. Evaluations should explore how the interconnections between specific mechanisms and contexts lead to observable outcomes. This paradigm shift urges evaluators to consider not just the presence or absence of outcomes but the reasons for their occurrence in a particular context, providing deeper insights into program effectiveness (Mukova-Moses, 2017; Pawson & Tilley, 1997).
Moreover, context-sensitive evaluations are valuable to inform the adaption of programs in different settings or for scaling up. Understanding how and why programs succeed or fail in differing contexts enables policymakers and managers to design programs that are more likely to be effective in a range of settings. This insight is critical in today’s globalised world, where programs developed in one locale are frequently adopted in others, each with its own unique features.
Methodologically, context sensitivity in evaluation benefits from a mixed-methods approach, drawing on quantitative methods to measure outcomes and qualitative methods to capture the nuances of how contextual factors affect those outcomes (Elahi et al., 2015; Greene, 2007). Fieldwork, including site visits and observation, can inform a deeper level of contextual understanding and nuance that is not possible through more distant methods of data collection (Kelly, 2022). Such approaches enhance the evaluators’ ability to provide recommendations that are informed by a comprehensive understanding of the contextual landscape.
Certainly, there are limits to the contribution of context sensitivity in program evaluation. Scriven (2008) offers a warning against the potential overemphasis on context that could obscure the identification of generalisable principles. However, the inclusion of context sensitivity ensures that evaluations account for the complex realities within which programs operate, leading to more effective, adaptable and sustainable outcomes. Drilling down on what works well for whom and under what conditions supports good program design and adaptation, tailored to the specific needs and conditions of each setting. By embracing this, evaluators can produce findings that are more accurate and applicable to stakeholders, enhancing the practical utility of evaluation research.
Background
Between 2019 and 2022, devastating bushfires and floods severely impacted many communities in New South Wales. In the northern parts of the State, in the Hunter Valley to the west of Newcastle and around the regional centre of Lismore, floods occurred in relatively short time after bushfires, compounding the impacts of these disasters on affected communities. Following these disasters, Australian Red Cross identified the need to enhance its suite of community-based emergency service programs by focussing on disaster resilience. An internal report, Beyond Vulnerability, outlined the capacities linked to an individual’s disaster resilience: knowledge, connection, security and wellbeing (Richardson, 2014; Richardson et al., 2023).
Definitions of resilience are contested and complex, but a simple definition is the ability of individuals, communities and systems to absorb, adapt to and recover from shocks or stresses without significant long-term disruption. Community resilience is the collective capacity of communities to use resources and knowledge to cope with and recover from disaster impacts, particularly when these occur at the scale of local systems (Norris et al., 2008). This is reinforced by systemic understandings of resilience as not only about bouncing back but also about transforming systems to better withstand future challenges (Ungar, 2018).
Australian Red Cross has increasingly adopted a community-led approach to disaster resilience and recovery, supporting the establishment of Community Resilience Teams as a primary initiative. These Teams aim to improve people’s preparedness for a range of hazards, including floods, fires and pandemics. The Teams are established with the guidance and support of Australian Red Cross. They also interact with emergency management organisations, which include New South Wales State Emergency Services (SES), New South Wales Rural Fire Services (RFS) and Local Emergency Management Officers at local councils.
Community Resilience Teams are presented as ‘a way for community members to support each other and improve their capacity for disaster resilience’ (Australian Red Cross, 2022, p. 3). The foundation for the Community Resilience Team is a local communication network that distributes essential emergency information to residents, while also gathering local intelligence on how a bushfire or flood is unfolding. By establishing a network of informed and connected residents, Teams encourage people to take a coordinated local response to disasters. Teams are typically structured around a team leader, a deputy team leader and street or area coordinators who engage with and contact residents. These volunteers draw on local knowledge and external information and tools offered by specialist agencies including Australian Red Cross.
Little is asked of residents regarding their involvement in a Community Resilience Team beyond the provision of contact details so they can receive information and updates through the Team network. The leaders and street coordinators are provided with training to improve their understanding of risks and safety actions on disaster response and preparedness. People selected for these roles have skills and experience in group leadership and coordination of activities. If the community wishes, the Team can extend its activities towards the creation of plans to coordinate actions in preparation for a disaster or on the identification of resources that may be needed post-disaster.
The flexible nature of the Community Resilience Team model allows it to be adapted to the needs and characteristics of communities, whether defined geographically or by common bonds, such as cultural or linguistic ties. The model exemplifies a shift towards a more decentralised and community-centric approach in emergency management, to empower local communities to take a lead role in their own disaster resilience initiatives (Kelly et al., 2024; Rawsthorne et al., 2022, 2023; Sampson et al., 2021).
Community Resilience Teams equip people in a local area to thoughtfully enact immediate safety and survival plans that are not reliant on outside assistance from emergency services or others; the recent experiences of the bushfires and floods in northern New South Wales highlighted that local areas may not be accessible or able to receive outside assistance in the period immediately following the impact of a disaster.
Community Resilience Teams are designed to harness the capacities and local knowledge of community members, enabling them to support each other before, during and after disaster events. The Teams operate under the premise that individuals and households in local communities are best positioned to understand their unique risks, resources and capacities. This model is intended to empower people and communities to take an active role in their own disaster preparedness and recovery processes.
Evaluation Methods
External evaluation of the Community Resilience Teams was commissioned in 2022 after the experiences of bushfires and floods in northern New South Wales saw a rapid increase in interest in Community Resilience Teams. The first author (an independent evaluator with no prior connection to Red Cross) was contracted by the second author (the National Evaluation Advisor at Australian Red Cross) to conduct the evaluation through a competitive tendering process. The evaluation examined existing and newly established Community Resilience Teams in six communities, to assess their value and impact. The purpose of the evaluation was to inform decision-making around a long-term commitment to support Community Resilience Teams and the potential for the concept to be adopted more widely across New South Wales and Australia.
There were three evaluation questions: Were the Community Resilience Teams well designed and supported? Were the Community Resilience Team activities relevant and appropriate? To what extent were intended outcomes for Community Resilience Teams achieved?
The evaluation analysed qualitative interview data, field observations, documentary materials, operational monitoring data and descriptive socio-demographic statistics across six community case studies. The design of the evaluation applied Realist Evaluation principles and practices to examine what works for whom, to what extent and in what contexts (Pawson & Tilley, 1997). Figure 1 summarises the processes adopted in the evaluation. Context-sensitive evaluation framework for disaster resilience and recovery, adapted from Pawson and Tilley’s (1997) realist evaluation framework
Context mechanism outcome (CMO) statements were developed for the evaluation from the theoretical frameworks relevant to the development of the Community Resilience Teams. First, the Australian Red Cross Emergency Services Monitoring and Evaluation Framework (Kelly, Goodall, & Lombardi, 2022; Kelly & Lombardi, 2022), which includes the theory of change for all emergency services programs, was used to select outcome statements that aligned to the Teams. In particular, the outcome statements on encouraging residents to collaborate and build horizontal bonding/binding and vertical linking social capital (Putnam, 2000) were selected as relevant to the Community Resilience Teams. Second, the framework developed by John Richardson (2014) in his report Beyond Vulnerability was referenced for the recommended four adaptive capacities for building people’s disaster resilience: knowledge, connection, security and wellbeing. These capacities were also included in the outcome statements for Community Resilience Teams because of their focus on building participants practical knowledge and skills.
Data collection and analysis, accordingly, was based on the criteria and data measures that related to the theory of change outcomes and the four adaptive capacities that were selected for the CMO statements. Evaluation findings were formulated and presented against these outcomes and capacities to allow the examination of the context using the case studies.
The evaluation adopted a mixed-methods approach, and data was collected from several sources to examine in detail the characteristics of Community Resilience Teams, the intended purpose of their activities and the extent to which these activities have generated outcomes relevant to the evaluation.
Initially, a review of documentation on Community Resilience Teams and their operation was undertaken, including examination of the Australian Red Cross approach to community disaster resilience, the Community Resilience Team guidance document and the associated training materials. This was used to confirm the intended design and operation of the Teams.
Case study research as a data collection and analysis tool enabled investigation of localised adaptations of Community Resilience Teams. Six communities in which Community Resilience Teams had been established were selected as case studies with the involvement of Australian Red Cross personnel who were well placed to provide information and insights about the communities. While their involvement raised a risk of selection bias, this risk was offset through the involvement of the independent evaluators. Moreover, it was assessed that the risks of inappropriate selection and engagement with communities were greater and justified the involvement of Australian Red Cross personnel who knew the communities.
The process of selection involved several steps: first, the objectives for the selection were defined as having communities that had active Community Resilience Teams and communities that did not; having communities with sufficiently different characteristics to support comparisons and contrasts; and having communities that could be visited for the evaluation with minimal risk of harm to residents. These objectives were translated into the following questions: (a) Has the community been impacted by a disaster in the past 12 months? (b) Does the community have a mature Community Resilience Team in place? Or does the community operate without a Community Resilience Team? (c) What are the characteristics of the community with regard to: • Nature of the disaster impacts, • local socio-economic profile and • indicators of community strength, such as shared identity, cohesion and social trust/collaboration.
Two exclusion criteria were set to meet the objective of minimal risk of harm: Communities that are currently pre-occupied with immediate disaster response activities such as safety and repairs and will not have time or practical capability to contribute to the evaluation. Communities in which people are highly distressed to a point where engagement on the evaluation could add to this distress.
The communities that were selected captured examples of established and emerging Community Resilience Teams from across the affected regions. Accordingly, the case studies covered various degrees of maturity in Community Resilience Teams. All case studies were in the northern regions of New South Wales, and all but one had been impacted by fire and/or flood emergencies in relatively recent times. Two indigenous communities were selected but not included in the evaluation activities as circumstances in those communities changed during the early stages of the study and became no longer appropriate to visit them.
Communication with the selected case study communities was managed through the team leaders for each Community Resilience Team. This was the most workable arrangement as these leaders already had the acceptance in the communities to provide liaison with external organisations. They were well placed to represent the people in their communities. After gaining consent from each of the team leaders, the evaluator visited the communities on agreed days/times. This field activity was important for the collection of data because of the place-based nature of Community Resilience Teams. The suitability of this approach has been identified for studies that defy simple quantification and require the recognition of nuances of attitude and behaviour: ‘By going directly to the social phenomenon under study and observing it as completely as possible, you can develop a deeper and fuller understanding of it’ (Babbie, 1991, p. 285). The visits allowed for observation of the geographic and built features of the community, the social norms and outlooks and the characteristics of those who were coordinating the Teams.
Visits to the case study communities occurred over a 2-month period during the spring season. These visits were primarily conducted by the evaluator with Australian Red Cross program officers for Community Resilience Teams accompanying the evaluator on four out of six of the visits. During the visits, semi-structured interviews were conducted with the Team leaders and street coordinators (between two and four people per community; a total of 20 people). Information on the establishment and operation of the Teams along with feedback was collected through these interviews. Opportunities and barriers to the Team’s development were also identified.
To supplement the observations and interviews, materials, documentation and visual resources on Community Resilience Team activities were collected during the visits. Photographs were taken of local features or assets relevant to disaster response and resilience.
Demographic and socio-economic profiles of the communities were developed drawing on data from multiple sources including Australian Bureau of Statistics data and local government planning data.
Another source of data for the evaluation was the Australian Red Cross monitoring data on community engagement activities, which included notes or comments entered by Red Cross personnel on the issues, sentiments and feedback raised in interactions with people (Kelly et al., 2022). The non-identifiable quantitative and qualitative data were stored securely in a proprietary database. Red Cross personnel extracted the relevant data into Excel spreadsheets, which were shared with the evaluator under the data access and use conditions of the evaluation. Descriptive statistics were generated from the quantitative data while thematic summaries were drawn from the qualitative data.
Five semi-structured interviews were held with stakeholder representatives from emergency management organisations and local government. The interviews were conducted a month after the field activities were completed and obtained information and insights about these stakeholder’s understandings of Community Resilience Teams, their interaction with Community Resilience Teams and their views about the relevance and effectiveness of Community Resilience Teams for disaster response and emergency management. The limited availability of emergency service staff for interview restricted the number of organisational representatives for these interviews. However, a representative of each emergency service organisation was identified and interviewed.
For each case study community, the quantitative and qualitative data collected from all sources were compiled and examined in entirety. Content analysis was undertaken on documents and materials. Thematic analysis was undertaken on the notes and quotations gathered from the interviews. This involved taking a deductive coding approach initially, using the evaluation outcome measures as the framework, but also shifted to inductive coding as unique or recurring information from the interviews emerged. The coding was undertaken entirely by the evaluator (the lead author of this article), providing an assurance of independence, while recognising that there were limits imposed in the resources available for the task. Observation data was subjected to analysis against the emerging themes from the interview data. Where observations challenged the thematic findings, the coding was checked a second time. Bringing all data together for each case study allowed the processes of checking and triangulation to identify traits, similarities and differences in the data. Finally, the emerging findings for each case study were considered against the other case studies to identify similarities and differences.
All evaluation activities were undertaken by the independent evaluator who is experienced in social research methods, and a Member of the Australian Evaluation Society, that is, bound by the Society’s Code of Ethics. The evaluator’s work was overseen by the Australian Red Cross Head of Research and Evaluation to ensure ethics were addressed and protocols observed throughout the evaluation.
Ethics issues surrounding participation in the evaluation data collection were addressed in several ways: Australian Red Cross distributed a written description (plain language statement) of the purpose and conduct of the evaluation ahead of any evaluation activities, giving people the option for informed participation. This included mention that the de-identified findings may be reported in external publications. Local Australian Red Cross personnel facilitated introductions between the evaluator and the key community representatives in each of the case study communities. Stakeholder consent for interviews was obtained in advance with clear opportunities for withdrawal of consent before, during and after engagement. Ethical considerations were discussed with the evaluation working group, which was composed of the evaluator and relevant Australian Red Cross personnel. The organisation and evaluator accepted responsibility for adhering to the agreed mitigation strategies and maintaining risk to a maximum level of slight discomfort.
To illustrate the caution required on ethics issues, initial plans for the evaluation included a survey of residents in communities where a Community Resilience Team was operating on their engagement with the Team and feedback on its impact. However, the survey did not proceed on the basis that residents had provided their personal contact details to a Community Resilience Team for emergency contact purposes only; using these details to issue a survey would have breached this agreement. Further, most of the case study communities had recently experienced disasters, so sensitivity to residents’ emotional distress and wellbeing was required. A survey could have been distressing for respondents. Australian Red Cross’ previous experiences with surveys in post-disaster settings also suggested that surveys are less effective in obtaining responses than engaging directly with groups and individuals on a personal level (Newnham et al., 2023).
There are limitations to this study that should be discussed. The selection of case study communities enabled the evaluation to examine Community Resilience Teams in varied contexts and stages of maturity, but the findings should not be seen as representative of all Community Resilience Teams. The use of qualitative interviews and observations allowed data to be collected on local dynamics, although this data may be influenced by perspectives of those who are most personally associated with the Community Resilience Team activities. Data was not collected directly or widely from residents, and this reduced the ability to adjust for any bias held by Community Resilience Team Leaders in the case study communities. The data from emergency management services representatives was limited by people’s availability for interviews, narrowing the range of institutional viewpoints. Reliance on existing Australian Red Cross operational monitoring data ensured the use of established records but meant some activities may not have been documented in full detail. As the evaluation took place during the early stages of Community Resilience Team development in many communities, the findings capture formative outcomes and perceptions, offering a valuable baseline rather than a definitive assessment of long-term impact.
Findings
Were Community Resilience Teams Accepted and Supported?
There was wide agreement within Community Resilience Team leaders and external stakeholders that the Community Resilience Team concept being promoted by Australian Red Cross meets people’s needs and complements the existing array of emergency management services. Moreover, it was possible to identify from the six community case studies and the interviews with emergency services representatives those specific features of the Community Resilience Teams that were widely regarded as the key to their acceptance.
From the perspective of Team leaders, it was the ‘community-led’ characteristic of Community Resilience Teams that made the concept responsive to local resident’s preferences and priorities. They liked the focus of the Teams on providing reliable information that was relevant to local circumstances and the needs of residents who were making decisions on their response to a pending disaster. They saw the benefit of having a communications network and identifiable key people in the local community who would maintain this network. They pointed to the ‘light touch’ approach of the Teams with little being required of residents beyond the provision of contact details for communication flows. They appreciated the flexibility in the Community Resilience Teams operations so the local community could determine which other activities the Team would get take on.
The elements of the Community Resilience Teams that Team leaders identified as gaining community acceptance are presented in Figure 2. Key elements of Community Resilience Teams that were appreciated by team leaders
Similarly, the Community Resilience Teams were regarded favourably by external stakeholders in emergency services organisations, although for different reasons. These stakeholders valued having a single local point of communication, with protocols, and a structured means by which communication with a locality could occur. They also saw value in the Teams facilitating a boost in local capabilities and viewed them as a support to their own services and plans. For these stakeholders, the concept of ‘community led’ meant that residents were equipped for greater self-reliance, and with realistic understanding about what external assistance could be provided post-disaster. They liked the way the Community Resilience Team could reinforce actions contained in emergency management plans and communication these actions to residents in a systematic way, for example, how road closures would be determined, announced and operate.
The elements of Community Resilience Teams that won support from emergency service stakeholders are presented in Figure 3. Key elements of Community Resilience Teams supported by emergency services’ stakeholders
Were Community Resilience Team Activities Seen as Relevant and Appropriate?
The evaluation considered the question of Community Resilience Team’s relevancy and appropriateness using the disaster resilience framework which relates a person’s resilience to four adaptive capacities: knowledge, connection, security and wellbeing (Richardson, 2014). Shifts in these capacities for residents were examined using the data gained from the six community case studies, noting that data was not obtained directly from these residents via survey or similar instrument for reasons already stated.
The Teams generated knowledge for individuals and throughout the communities in several ways. First, they raised people’s awareness of risks to safety, property and local environment from disasters and increased their understanding of the potential for disasters and the likely scales of impact by providing locally tailored information and drawing on local experiences, including recent disasters. They provided people with information and tools to undertake personal and household planning and educated people towards more realistic appraisals of what advance preparations, resources and assistance they may need to organise.
The Teams improved the connection with residents in each community and enabled more rapid transmitting of information between the community as a whole and the emergency services such as the fire service, state emergency service (SES) and police. This was achieved through the register of contacts in a locality so communication could occur in ways that would reach everyone, even in times of conventional communication systems failure using phone trees, or street coordinators who would visit each property. The CRTs also improved people’s access to authoritative data on weather and disaster occurrences by providing structured and reliable points of contact and communication.
The Teams had an impact on individual and household security through the emphasis they gave to personal choice and informed decision-making in disaster preparedness. In equipping and encouraging individuals to prepare themselves and their households for a disaster, people’s sense of security and self-efficacy was being strengthened – they knew what they would do and how they would do it, if required. It was likely, but not possible to measure in this evaluation, that individual and household coping capability and therefore mental wellbeing were strengthened.
The four adaptive capacities from the Beyond Vulnerability report, knowledge, connection, security and wellbeing (Richardson, 2014; Richardson et al., 2023), aligned closely with the descriptions of Community Resilience Team purpose and benefit that both the local community representatives and the emergency services representatives put forward. The evaluation found the relevance of Community Resilience Teams in disaster response lay mainly in their ability to build individual and household self-awareness and their capacities for self-reliance.
To What Extent Were Intended Outcomes for Community Resilience Teams Achieved?
Overall, the evaluation found it was those outcomes from the theory of change in the Australian Red Cross Emergency Services Monitoring and Evaluation Framework (Kelly, Goodall & Lombardi, 2022; Kelly & Lombardi, 2022) related to individual and household impacts that were more clearly met through Community Resilience Teams than outcomes on psychosocial impact and social capital.
Notably, Community Resilience Team leaders did not generally identify social capital generation as an outcome of the Team as they did not see the Teams influencing local community dynamics, nor did they wish the Team to attempt do so. Team leaders liked the ‘light touch’ of the Community Resilience Team concept that upheld individual self-determination in disaster responses.
Community Resilience Teams are an effective vehicle for equipping individuals and households to take practical steps toward preparation, planning, capability building and actual responses to disasters. They harness local knowledge on risks and hazards and combine this with information from emergency management agencies to provide people with the knowledge they required for these activities. In this regard, Community Resilience Teams are working toward the first outcome in the theory of change whereby communities are taking localised, community-driven actions to build resilience and recover from disasters.
The second outcome in the theory of change was people collaborate to share information, strengthen community connection and support resilience and recovery. Community Resilience Teams enabled the sharing of information by individuals and households in communities so common knowledge on risks and hazards was established in all the case studies. This was especially noticeable in those communities with relatively large numbers of new arrivals, meaning local and historic knowledge is not universally held in the community. It remained less clear, however, whether the Teams foster increased collaboration or support among people, although there were some specific examples in the case studies of this – such as a coordinated plan to evacuate animals in a community where many properties had horses.
Crucially, Community Resilience Teams are effective conduits for the distribution of reliable and authoritative disaster information – contributing to the theory of change outcomes that people have information needed to make decisions and that people feel empowered and in control of decision-making. For example: in all the case studies, the most noticeable difference between a community with a Community Resilience Team and a community without was access to this information. In one case study community, the whole impetus to form a Community Resilience Team was the realisation in a recent flood that people lacked access to the authoritative information on weather and emergency responses that the neighbouring community which had a Team was able to draw on. This reaffirms the role of Community Resilience Teams in enhancing people’s self-reliance by providing them with essential information and tools, thereby relating to their intrinsic desire for independence and self-determination. However, it is important to note that the extent to which all community members received this information and the benefits of empowerment in their decision-making was a point identified in the evaluation that required further examination.
The theory of change outcome that people understand the psychosocial impacts of disasters was found to be less likely directly addressed by Community Resilience Teams. The identification of psychosocial impacts and needs surrounding a disaster were less obvious in the Community Resilience Team activities, although there were some examples where this occurred, such as efforts to address children’s safety and security in one community. This suggests the Team’s focus is more likely to be on immediate physical safety than the broader psychosocial health of the community.
An extension of this finding is that the emotional and psychological safety of community members, which relates to the theory of change outcome that people feel safe, calm and hopeful, is less likely to be associated with Community Resilience Teams.
Another outcome on psychological wellbeing from the theory of change – that people advocate for the psychosocial needs of the community – was found to be addressed through Community Resilience Teams, but not necessarily by the activities of the Teams. Examples were identified from the case studies in which Team leader had advocated for additional psychosocial support for people in their communities; this advocacy was often facilitated and advanced by the Australian Red Cross personnel working with the Team leaders. This finding highlighted the need for more structured efforts in identifying and addressing psychosocial needs in communities.
Context-Sensitive Variations
Summary of Activity Variations Across Case Study Communities
Discussion
It was a somewhat unexpected finding for a community-based program that individual and household outcomes were more prominently and more consistently identified across the case studies than psychosocial and social capital outcomes. The four adaptive capacities from the Beyond Vulnerability report, knowledge, connection, security and wellbeing (Richardson, 2014; Richardson et al., 2023), were more easily identified as the outcomes of Community Resilience Team activities. This finding was to some extent at odds with the theory of change in the Australian Red Cross Emergency Services Monitoring and Evaluation Framework (Kelly, Goodall & Lombardi, 2022; Kelly & Lombardi, 2022) that informed the development and evaluation of Community Resilience Teams. The design of community-based programs under the Australian Red Cross community disaster resilience model tends to draw on theories surrounding social development and capacity building that assume social benefits will generate greater and more widely held impacts across a community (Putnam, 2000).
From a realist perspective, this pattern indicates that in contexts where autonomy and self-reliance are strongly valued, Community Resilience Teams primarily activated mechanisms related to information access, confidence and self-efficacy, producing outcomes at the individual and household level rather than in the form of collective social capital. This finding can be explained by the influence of the context in which the selected Community Resilience Teams operated. Many people living in small rural and remote communities in northern New South Wales have a strong preference and capability for self-reliance. Their choice to live in isolated areas and provide for themselves reflects this. This evaluation observed these residents were not inclined to relate much or rely on others beyond what is essential to preserve life and safety during a disaster response. A common attitude was that each property owner/household should make their own decisions and manage their own affairs – the Community Resilience Teams were regarded as providing the information and communication tools to equip individuals to make these individualised responses.
This contextual factor in the evaluation showed how a more nuanced application of Community Resilience Teams is required for each local community. This context will affect the extent to which social connection and social capital are outcomes that will be generated for every community. Indeed, a subsequent evaluation of Community Resilience Teams in different towns found that social connection was strengthened by their activities (Horn et al, forthcoming). This may be due to contextual variation or other unknown factors. Either way, it is useful to identify that differences can occur between communities and not assume that all participants in Community Resilience Teams will be attracted to or embrace the concept of social capital. They may be more concerned with the capacities of individuals and linkages to emergency services in disaster preparedness and response.
The overarching advantage of Community Resilience Teams seems to lie primarily in their being a low-resource, community-led, light touch approach. They are not designed nor resourced to be much more than this. Yet, this was their main attraction to many people in smaller communities, and to emergency service stakeholders: the CRTs added something without interfering in people’s own decision-making and without crossing specialist services roles and responsibilities. This evaluation presented findings that moderated expectations for Community Resilience Teams in the future.
The findings from this evaluation demonstrated that many people in smaller, rural, communities are more interested in building their own capacity for self-determination in response to disasters than generating a community or collective capacity for those responses. That said, a blanket approach is inappropriate. It may not, or will not, be the same for other communities. For example, the special disaster communication needs of marginalised and minority groups in each setting should always be considered (Howard et al., 2017; Kelly et al., 2024).
This evaluation’s findings raise implications for those involved in disaster response and resilience in local government, state emergency services and civil society organisations such as Australian Red Cross. Community-based programs will be more readily received and generate greater impact if they are attuned to the beliefs, values and life preferences of those to whom they are offered. There will be differences between various communities, and these form contextual factors that need to be identified and understood in the planning and implementation of community-based programs. The limitations, as well as the potential, of the scope for community-based programs can be addressed effectively if there is well-developed knowledge of the community context. Accordingly, generalised community programs that do not adjust to contextual factors are less likely to be fully effective.
Conclusion
This evaluation shows the importance of a Realist Evaluation approach to community-based programs, that addresses context and variations, to draw out what works, for whom and why. It demonstrates that community-based programs such as the Community Resilience Teams by their nature are likely to be understood, experienced and assessed by different participants and stakeholders in different ways.
The evaluation found that people living in rural and isolated communities are often drawn to these locations because they value self-determination in all aspects of their lives. This orientation towards autonomy was an important contextual factor shaping the Community Resilience Teams, as a community-led initiative.
Context affected how the Community Resilience Teams were understood by participants and how they were implemented in the case study communities. The evaluation has highlighted an aspect of disaster resilience in rural Australia: self-determination for safety and survival can be recognised and strengthened through low-cost, practical, community action, such as Community Resilience Teams. This evaluation also shows that while there will be some areas of common ground across all program participants, which are useful to record, it is often the areas of difference that provide the greatest insight for decisions about the future development and ongoing operation of community-based programs. Making a wide range of findings available to decision makers ensures a more complete picture, and any surprises that emerge only reinforce the value of testing program theory and delivery for context.
For evaluations of community-based programs generally, this evaluation highlights the importance of addressing context sensitivity more systematically in evaluations. Contextual factors should be embedded at every stage of the process, from design through to interpretation and reporting. This means working with stakeholders to identify the socio-cultural, political, economic and environmental factors that shape program delivery; designing data collection to reflect diverse and marginalised perspectives; and interpreting results considering these dynamics.
Social research has a contribution to make in assisting evaluators examine and understand contextual factors for a community-based program. The use of case study research methods is possibly an under-utilised approach to generating the data required. Allowing for the risks surrounding case study bias, or non-representational nature, and the difficulties including ethical constraints in approaching people for data collection, the benefit of case study research is that it enables a deep exploration of individual reactions to a program in the context of social, economic, geographic and cultural influences that surround them. This can be applied to the evaluation of the program’s effectiveness using Realist Evaluation tools in analysis of data and the reporting on findings.
Reporting on findings for a community-based program should make explicit how context has shaped both implementation and outcomes, in part to show how the program has operated with the people and communities involved, but also to enable conclusions to be drawn on the extent to which the program can be adapted appropriately to other settings. Embedding a context-explicit approach ensures that evaluations produce findings that are grounded in the realities of place, local characteristics and stakeholder perspectives, thereby generating insights that are genuinely useful for effective and sustainable program design.
Footnotes
Funding
The authors received no financial support for the authorship, and/or publication of this article. Dr Woodward was commissioned by Australian Red Cross to conduct the original evaluative research upon which this article is based.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.