Abstract
Demand is growing in the public, private, and philanthropic sectors for economic and quantifiable outcome data to guide decisions regarding the allocation of resources. This paper explores the challenges and opportunities in economically evaluating an Aboriginal Social and Emotional Wellbeing intervention, emphasising the value of utilising existing data. Driven by local and national Indigenous research priorities, the study involved a pilot Social Return on Investment analysis of the Aboriginal Family Wellbeing empowerment program. Underpinned by a collaborative mixed-methods approach, designed to mitigate data limitations and bolster trustworthiness, the analysis was undertaken across four stages, including a literature review to identify impacts and create a theory of change; co-design of an impacts framework; a targeted literature review to inform impact quantification and attribution; and calculation of the Social Return on Investment results. Results indicate that for every Australian dollar of investment in delivering Family Wellbeing between 2001 and 2021 in the Yarrabah Aboriginal community, A$4.60 of benefits were produced for participants and the community. The return is expected to be higher if important community and cultural impacts of the program were included in the monetisation. Data limitations and implications for future mixed methods economic evaluation of social interventions are explored.
Keywords
• Communities wishing to advocate for support for local programs require a robust foundation of evidence to convince policy makers and funding bodies. • Researchers are caught in a bind between the call for measurable evidence and the difficulties associated with producing this form of evidence for complex social health programs. • Concerns continue to grow that research outputs such as reports and publications are not used to facilitate better and timely decisions regarding improving health and wellbeing for Aboriginal and Torres Strait Islander people.
• This article presents an innovative mixed-methods approach to a pilot economic analysis of a complex social intervention, emphasising the value of using extrapolated data. • The study’s timeliness and relevance are evident in its prompt contribution to decision-making, resonating with local leaders, and reinforcing commitment to a data-driven approach to decision-making through local ownership and control. • Whilst there are limitations to this form of economic analysis, it is sufficiently robust to guide decision-making and resource allocation, particularly when compared to using no economic evidence at all.What we already know
The original contribution the article makes to theory and/or practice
Introduction
For over 20 years, the Family Wellbeing program has been a cornerstone, providing a trauma-informed approach to healing for over 80 First Nations communities across Australia. Co-led and co-designed by Aboriginal community organisations and researchers, Family Wellbeing has evolved into a national network, collaborating with First Nations service providers to integrate the program into core services such as child protection, family support, and pre-employment and tertiary access. The program focuses on enhancing individuals’ capacity to exert control over factors influencing their social and emotional wellbeing, yielding tangible outcomes and positive transformations for participants and increased community capacity (Innovative Research Universities, 2023; Perera et al., 2022).
Yarrabah, located 56 km south of Cairns in north Queensland and Australia’s largest discrete Aboriginal community, provides a notable example of the community-level results through its involvement with Family Wellbeing. Aboriginal and/or Torres Strait Islander people make up 95.9 % of the estimated population of 3000. Yarrabah is relatively disadvantaged by many social indicators such as employment, year 12 completion, and overcrowding of housing (Queensland Government, 2024). When the Yarrabah men’s group was first introduced to Family Wellbeing in 2000, they immediately saw the relevance of the program both to themselves and their community. In the words of community leader and author Father Leslie Baird, ‘Much of our sense of disempowerment, individually and as a community, came from not being able to make critical decisions about our own lives. A course designed by people like us was what we needed. It spoke our language, it understood how we experienced empowerment and wellbeing’ (Baird, 2019, np.).
This article delves into the strategic utilisation of extensive existing research and experiential knowledge spanning two decades to conduct a pilot Social Return on Investment analysis of the Indigenous-developed Family Wellbeing empowerment program, with a particular focus on Yarrabah. A Social Return on Investment methodology was chosen for its ability to incorporate social and non-market benefits into the economic evaluation, as well as the focus on participant experience and articulations of value, allowing for the best utilisation of both qualitative and quantitative data. Our objective is to employ the pilot results as a foundation for comprehensive economic evaluations of Family Wellbeing and analogous First Nations social and emotional wellbeing programs in the future.
The decision to use existing data
The decision to utilise existing Family Wellbeing research and experiential knowledge for a pilot Social Return on Investment study was driven by two Indigenous research initiatives – one local and the other national. The local initiative, the Yarrabah Leaders Forum, aimed to improve the health and wellbeing of a rural Aboriginal community by developing a comprehensive strategy encompassing six key pillars: 1. Safe Community: Making our community safe; 2. Employed Community: Helping people to be financially secure and off welfare; 3. Smart Community: Ensuring everyone has the capacity for lifelong learning; 4. Sustainable Community: Developing infrastructure that meets the community’s needs; 5. Healthy Community: Ensuring everyone is healthy, and the community is free from preventable disease and illness; 6. Supportive Foundations: Building values, principles, standards, family supports, and funding to help construct the desired community (Yarrabah Shire Council, 2022).
One crucial aspect of the 6-pillar strategy involves projecting potential returns on investments for each component pillar and its associated intervention strategies. The objective is to produce relevant economic and quantifiable outcome data, supplementing predominantly narrative information. This approach aims to bolster the community’s advocacy efforts with the government by providing a more robust foundation of evidence. The study responds to growing demand in the public, private, and philanthropic sector for economic data to guide resource allocation.
Different organisations in Yarrabah have taken responsibility for addressing each pillar; for example, the local health service, Gurriny Yealamucka, has taken responsibility for pillar 5, and the employment agency Wugu Nyambil has taken responsibility for pillar 2. The Family Wellbeing program was identified as foundational to all six pillars, offering empowerment and capability building appropriate to the community’s diverse needs.
Family Wellbeing workshops have been intermittently delivered within the community for nearly 20 years, so a large proportion of the community has had some exposure. By 2011, 16% of the Yarrabah community, approximately 463 people, had attended Family Wellbeing sessions, demonstrating a significant reach (Baird, 2019; Perera et al., 2022). In addition to improving participants’ social and emotional wellbeing, Family Wellbeing workshops brought people together – men, women, and other community groups – to work collectively towards achieving important outcomes for the community, including building a comprehensive community-controlled health service, improving the local housing stock, and preventing suicide. In keeping with the Yarrabah Leadership Forum’s interest in economic data to strengthen community advocacy, the proposed implementation of Family Wellbeing as a foundational empowerment and capability program across the 6 pillars became an opportunity to conduct a pilot Family Wellbeing program Social Return on Investment study.
A national transformative funding model
The second and simultaneous Indigenous research initiative involved a call for research funding issued by the Lowitja Institute, Australia’s national Aboriginal and Torres Strait Islander community-controlled health research organisation. This call aimed to shift the research paradigm by directly funding Aboriginal and Torres Strait Islander organisations as holders of research funds, placing them at the forefront of decision-making processes (Australian Institute of Aboriginal and Torres Strait Islander Studies & The Lowitja Institute, 2017; Lowitja Institute, 2023b; Tsey et al., 2016). Launched in the 1990s, the Lowitja Institute, through its precursor the Cooperative Research Centre for Aboriginal and Tropical Health, addressed long-held concerns by Aboriginal and Torres Strait Islander Australians that they have been ‘researched to death’ with few tangible outcomes. Despite contributions to research reform in Australia, concerns remain that research outputs such as reports and publications are not necessarily translating or being deliberately used to inform better decisions and services, leading to improved health and wellbeing outcomes for Aboriginal and Torres Strait Islander people (Bainbridge et al., 2015; Tsey et al., 2016).
In this context, the new funding call sought to flip or turn traditional researcher-led processes upside down. Instead of awarding research funds directly to researchers to, in turn, build partnerships with community organisations, as has been the norm, Lowitja would now fund service organisations directly so they, in turn, could invite researchers to partner with them (Lowitja Institute, 2023b). Additionally, before embarking on new research in the conventional sense, research teams would be first required to determine whether existing knowledge (research and experiential) was enough to meet the identified research need and the benefits versus costs of using the existing knowledge and/or doing more research to inform smarter decisions. The emphasis was on transparently assessing whether existing knowledge sufficed to meet research needs, minimising research waste, and facilitating rapid translation of research into practice (Tsey et al., 2016).
In response to this call, a partnership led by Gurriny Yealamucka and the Yarrabah Leaders Forum involving James Cook University (JCU), Australia National University (ANU), Batchelor Institute of Indigenous Tertiary Education, and Deloitte Access Economics secured funding to support the implementation and evaluation of Family Wellbeing (Lowitja Institute, nd). The proposed project included a pivotal Social Return on Investment analysis, aligning with the commitment to Aboriginal and Torres Strait Islander-led research and ensuring that research outputs directly contribute to better decisions and outcomes for their communities.
This paper will provide a detailed exploration of the rationale behind choosing existing data for the pilot Social Return on Investment study, elucidating the innovative framework employed, and highlight the value of maximising existing knowledge for meaningful community impact.
Background
Before discussing the technical details of the pilot Social Return on Investment analysis, it is essential to present an overview of the Family Wellbeing program, along with the nature and scope of the foundational knowledge. This analysis aims to explore the adequacy of the available Family Wellbeing evaluation data for the specific Yarrabah study and argues for the appropriateness of extrapolating data from other relevant Family Wellbeing evaluations for the pilot study.
The Family Wellbeing empowerment program
In Australia, Aboriginal and Torres Strait Islander communities face significant, yet preventable, social inequities resulting from historical factors such as colonisation and dispossession, leading to family, community, and cultural disruption (Perera et al., 2022). Despite waves of social policies since colonisation, these inequities persist, making empowerment crucial in addressing relative disadvantages (Fredericks, 2009). Empowerment education programs, however, remain inadequately researched (Bainbridge et al., 2018; Williamson et al., 2023).
Family Wellbeing, developed in 1992, serves as an empowerment and transformational education program for Aboriginal and Torres Strait Islander Australians. Originating from the Aboriginal Employment Development Branch of the South Australian Department of Education, it addresses the complex challenges faced by these communities, emphasising self-empowerment (Tsey & Every, 2000; Perera et al., 2022).
Initially designed as a 150-hour accredited vocational education and training (VET) sector certificate course, Family Wellbeing evolved in response to participant feedback, leading to the development of a non-accredited short-course by the JCU-led research team. This increased accessibility, allowing the program to be run in two or three-day blocks, or over a longer period with weekly sessions (Perera et al., 2022). Covering topics such as group agreement, human qualities, basic human needs, life journey, relationships, conflict resolution, managing emotions, crisis, grief and loss, beliefs, and attitudes, Family Wellbeing utilises a narrative group process fostering transformative human qualities like empathy, critical reflection, and problem-solving skills. While initially tailored for Aboriginal and Torres Strait Islander Australians, its universal themes render it adaptable for all cultures and social groups (Perera et al., 2022).
Existing body of Family Wellbeing knowledge
To evaluate the adequacy of existing Family Wellbeing knowledge for conducting a return-on-investment analysis in Yarrabah, we organised the available information into two categories: Yarrabah-specific data and other Family Wellbeing data relevant to Yarrabah. The primary motivation for Yarrabah Leadership Forum leaders to delve into the economic value of Family Wellbeing within the 6-pillar strategy arises from compelling Yarrabah-specific evidence. This evidence was drawn from the firsthand experiences of Forum leaders who had been Family Wellbeing participants, trainers, or community leaders themselves.
Over a span of two decades, local Family Wellbeing participants and community leaders have consistently shared insights through local evaluation reports, videos, and conference presentations, highlighting the transformative impact of Family Wellbeing on both individual and community life (Baird, 2019; Daly et al., 2004; Tsey et al., 2019), as evident in the following quotations: Now I can love my wife and children because I love myself. I used to drink all the money, but now I got money instead of alcohol. we identified poor housing…or not poor housing but lack of houses and tried to do something. We’re not going to stop until we are satisfied. (Daly et al., 2004)
Some notable participant stories are presented below in the Box, illustrating the profound benefits of Family Wellbeing, not only for individual participants but also for the community at large.
Box 1: Notable Yarrabah stories of transformation and change
1 A community leader, who initially left school at Year 10 with the belief that education was not for him, underwent a transformative experience with Family Wellbeing. The Life Journey exercise inspired him to set a goal of attending university and writing a book about his community development work in Yarrabah. He successfully completed a Masters of Indigenous Health at Sydney University in 2014 and is now a lecturer, using Family Wellbeing to develop drug and alcohol and suicide prevention courses for Indigenous students. 2 An unemployed single mother, residing in a tin hut outside Yarrabah, found empowerment through the Basic Needs component of her Family Wellbeing course. She established a Community Housing Action Group, influencing local council decisions on housing and infrastructure. Progressing from a participant to becoming a Family Wellbeing and women’s worker, she continues to deliver Family Wellbeing to women’s groups in Yarrabah, as well as students in and beyond Yarrabah schools. 3 An Alcohol Worker at Gindaja Rehabilitation Centre recognised the holistic Family Wellbeing approach’s alignment with human qualities and basic physical, emotional, mental, and spiritual needs. After attending Family Wellbeing Facilitator training provided by JCU researchers, she integrated Family Wellbeing topics into Gindaja’s residential treatment service, impacting an estimated 80 clients annually for over a decade. For more comprehensive oral accounts of people’s stories of change through Family Wellbeing, visit the following links: https://www.youtube.com/watch?v=x1BYxXADjMk; https://www.youtube.com/watch?v=AR50rxAOnyU&t=13s
Additionally, independent research commissioned by the Healing Foundation on First Nations suicide prevention revealed Yarrabah as one of only two Indigenous communities in Australia, alongside the Tiwi Islands, to have successfully reduced high suicide rates over the past 20 years (Prince et al., 2018). The evaluation attributed substantial contributions to Family Wellbeing and the local Men’s Group in Yarrabah. Given the profound impact of Family Wellbeing on the community, the next logical step is to identify relevant Family Wellbeing data beyond Yarrabah that could complement and strengthen these narratives.
Other data relevant to Yarrabah
Data from numerous Family Wellbeing programs across Australia consistently revealed positive outcomes. Small-scale qualitative studies in various communities (totalling 11), spanning New South Wales, the Northern Territory, North Queensland (including Yarrabah), Victoria, and even international studies in Papua New Guinea and China, consistently highlighted narratives of personal growth, heightened confidence, increased resilience, self-care, and improved management of alcohol and gambling issues. Participant testimonials further underscored the positive impact on relationships, education, and employment (Perera et al., 2022).
Similarly, small-scale quantitative studies (totalling 5), including three from distinct Indigenous communities in New South Wales, North Queensland, and Victoria, as well as others involving non-Indigenous Family Wellbeing participant groups in Australia and internationally, corroborate qualitative evidence. They report significant reductions in psychological distress, positive changes in empowerment and wellbeing measures, and increased life satisfaction and inner peace (Perera et al., 2022).
Relevant cost data encompassed a 2017 analysis of the economic cost required to deliver the Family Wellbeing program as workforce training across five remote North Queensland communities. The study provides valuable insights into resource utilisation, suggesting potential cost mitigation if offered on-site in the community (Kinchin et al., 2017).
A notable impact of the largely small-scale, community-by-community Family Wellbeing research to date has been the continued incremental uptake of the program across Australia and internationally. The dissemination of research publications and knowledge translation activities and resources, often with funding and other support from the Lowitja Institute, has helped spread the message about Family Wellbeing program outcomes. These activities and resources include community reports, knowledge-sharing forums, brochures, short videos, and ‘word of mouth’ conversations. They have led to an increasing number of Aboriginal and Torres Strait Islander primary health care services across Australia adopting the program as a preferred model for preventative care, with over 5000 people participating in either the full 5-stage course or the shorter version in 20 years (Perera et al., 2022).
A further source of relevant data involved the Mayi Kuwayu National Study of Aboriginal and Torres Strait Islander Wellbeing. Over the past five years, analysis of data obtained through this study reinforced the small-scale research findings by indicating robust associations between Family Wellbeing exposure and empowerment outcomes (Williamson et al., 2023). The study uses a culturally congruent survey designed in collaboration with Aboriginal and Torres Strait Islander communities across Australia. The survey asks questions about empowerment, wellbeing, identity, culture, language, and community, and how the various measures relate to each other. Participants are asked to indicate whether they have participated in known Aboriginal and Torres Strait Islander social and emotional wellbeing programs such as Family Wellbeing. To date, the Mayi Kuwayu survey has been completed by over 12,000 Aboriginal and Torres Strait Islander people Australia-wide.
The inclusion of Family Wellbeing in the survey questions enabled a national cross-sectional study of Family Wellbeing participant outcomes across several health and empowerment outcome areas, including personal control, general health, and community empowerment. The analysis concluded that there are strong associations between Family Wellbeing exposure and empowerment outcomes at the family and community levels. For example, compared to non-participants, Aboriginal and Torres Strait Islander participants in Family Wellbeing reported: • a 13% higher rate of family functionality • a 74% higher rate in higher cultural wellbeing • a 21% higher rate in higher levels of local decision-making in community (Williamson et al., 2023).
The findings of the Mayi Kuwayu survey do not establish causality, but they do strongly corroborate and strengthen 20 years of consistent research and experiential knowledge that the Family Wellbeing approach has the capacity to motivate and support people to take greater control and responsibility for their lives, even when many aspects of their social environment remain difficult.
Overall, while data specific to Yarrabah alone may not suffice for a comprehensive Family Wellbeing return on investment analysis, the wealth of experiences shared by participants and community leaders, along with extrapolated data from various Family Wellbeing program studies, forms a sturdy foundation for a pilot Social Return on Investment study. This strategy is justified as a deliberate approach to initiating the Social Return on Investment analysis and advancing the understanding of Family Wellbeing program impact. The evidence consistently indicates positive outcomes of Family Wellbeing across diverse communities, reinforcing its effectiveness. This preliminary assessment not only expedites the translation of evidence into practice but also guides strategic and targeted decisions regarding future data collection.
Methods
The longer-term objective of the economic evaluation was to create policy-relevant evidence to demonstrate the estimated economic impact of Family Wellbeing. A related objective was to create a robust framework for future economic evaluations. The mixed-methods approach recognises and values participant and community voices and has been undertaken to be of benefit to them. The study received ethical approval from the Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS) Research Ethics Committee (EO296-20210927) and James Cook University (JCU HREC H8724).
This paper aimed to examine the challenges and opportunities associated with the economic evaluation of Family Wellbeing empowerment program, emphasising the value of utilising existing data. The study’s value should be evaluated based on its effectiveness in assisting local decision-makers address pivotal questions crucial to Yarrabah’s comprehensive development plan.
With its emphasis on valuing social as well as economic impacts, Social Return on Investment analysis was chosen as the economic impact methodology. This approach compares the cost of a program or intervention with its estimated social impact, by monetising outcomes where possible. Social Return on Investment methodology is comparable to other value for money evaluative techniques such as Return on Investment analysis and Cost–Benefit Analysis. However, Social Return on Investment analysis was seen as the most appropriate framework for this evaluation due to its broader conceptualisation of value (Corvo et al., 2022; Gargani, 2017) and focus on participant experience.
Social Return on Investment methodology is underpinned by seven key principles (Social Value UK, 2016): 1. Involve stakeholders. 2. Understand what changes. 3. Value the things that matter. 4. Only include what is material. 5. Do not over-claim. 6. Be transparent. 7. Verify the result.
A mixed method and participatory approach to mitigate data limitations and bolster trustworthiness
From the outset, mindful of potential pitfalls of using extrapolated data to conduct a Social Return on Investment analysis, a collaborative mixed-methods approach, designed to mitigate data limitations and bolster trustworthiness, was implemented. Key components of this approach included leveraging the extensive qualitative datasets to construct a preliminary Social Return on Investment analysis framework. The draft Social Return on Investment framework underwent refinement through interactive workshops with information-rich Family Wellbeing facilitators and researchers, whose feedback overwhelmingly affirmed its alignment with local experiential knowledge. The synergy between different datasets not only complemented and strengthened each other but also facilitated the incorporation of valuable experiential knowledge from frontline Family Wellbeing facilitators and researchers at critical stages of the analytical process. This collaborative approach significantly enhanced the integrity of the pilot study and confidence in the framework. It elicited enthusiastic endorsement from experienced Yarrabah Family Wellbeing facilitators and leaders who found the pilot Social Return on Investment results resonant with their individual and collective experiences. Incorporating these voices directly in subsequent research phases will further bolster confidence in the benefits framework.
Ultimately, the mixed-method approach sought to best utilise all available qualitative and quantitative data, while also iteratively testing and refining the framework and findings with participants to ensure data interpretation and extrapolation was contextually appropriate for the experience of Family Wellbeing within Yarrabah. While conscious of data limitations, which have been noted where relevant, this innovative and practical approach allowed for the wealth of existing data and evidence to be optimised and contribute to the Social Return on Investment narrative.
Data limitations for the pilot Family Wellbeing Social Return on Investment analysis for Yarrabah included a scarcity of quantitative program data from Yarrabah Family Wellbeing deliveries and challenges associated with relying on conventional datasets for economic evaluations in the context of suicide prevalence and the unique demographics of the Aboriginal and Torres Strait Islander population in Yarrabah. Acknowledging these limitations, the economic evaluation drew on findings from Williamson et al. (2023) to quantify the empowerment impacts of Family Wellbeing for participants in Yarrabah, despite the findings not being specific to Yarrabah. Recognising the uncertainty surrounding participant characteristics, dosage, and program stages, the study assumed broad consistency over 20 years based on qualitative evaluations and experiential knowledge. Williamson and colleagues’ findings, indicating correlation rather than causation between empowerment outcomes and Family Wellbeing, added a layer of complexity.
Despite these limitations, the application of overarching study results to the Yarrabah context is viewed as conservative, likely underestimating program impact. Where uncertainty existed in the data, conservative assumptions or estimates were adopted. A conservative attribution factor (20% for most benefits) was also applied, considering the vulnerability of the community and the high program exposure likely to result in greater benefits. However, it is crucial to interpret this pilot economic evaluation as an indicative estimate, subject to further research and refinement.
Social return on investment analytic framework
The actual Social Return on Investment analysis involved 4 key stages to identify, quantify and, where possible, monetise the benefits of Family Wellbeing. Stage 1: A desktop review of existing qualitative research on the impacts of Family Wellbeing identified 14 qualitative studies relating to implementation and impacts for Aboriginal and Torres Strait Islander peoples, five studies adopting quantitative approaches, and two reviews of Family Wellbeing literature. Stage 2: Deloitte Access Economics drafted an initial theory of change for Family Wellbeing, based on a review of the existing, largely qualitative, research. Hence, the Theory of Change was informed by the voices of Family Wellbeing participants. It documented the intervention inputs, activities and outputs followed by the immediate and short to medium empowerment outcomes and the program impacts for participants. The benefits for participants in the program, as well as their families, and the broader community were considered. As noted, a team of Family Wellbeing researchers and facilitators reviewed the draft theory of change which resonated with their experiences (see Figure 1. Family Wellbeing Theory of Change). Stage 3: A more targeted desktop review focussed on the pre-post intervention quantitative studies in Aboriginal and Torres Strait Islander and non-Indigenous settings to verify and estimate the social impacts of Family Wellbeing. This review, which drew on existing quantitative studies and data collections to estimate impact, included an analysis of evidence for Family Wellbeing impact of exposure based on findings from the Mayi Kuwayu National Study of Aboriginal and Torres Strait Islander Wellbeing involving all responses received nationally between October 2018 and December 2020 (n = 9,843) (Williamson et al., 2023). Four overarching themes of impact for Family Wellbeing: health, wellbeing, and culture; education and employment; families and connectedness; and self-determination and empowerment were identified. Indicators for measurement were listed within each theme along with the data source, and the approach to measurement (e.g. monetise, qualitative, or quantify). Stage 4: Calculation of the Social Return on Investment results. This involved monetising the quantifiable impacts based on existing data or quantified or qualified based on existing research, including the existing Mayi Kuwayu survey data, on the impacts of Family Wellbeing and comparing the monetised impacts to program costs to estimate the Social Return on Investment ratio. Outcomes data was drawn from the outputs of Stage 3, while cost data was informed by previous research on implementation and delivery of Family Wellbeing in Cape York (Kinchen et al., 2017). Family Wellbeing program Theory of Change. Source: (Deloitte Access Economics, 2022). Produced for the Gurriny Yealamucka Health Service to inform the economic evaluation of the Family Wellbeing Program.
Monetising the benefits
This analysis only captures those impacts which could currently be monetised, which collectively represent only a portion of the overarching benefits of Family Wellbeing in Yarrabah. Many of the important impacts of the program could not be quantified or monetised in this analysis with available data.
Attribution
An attribution factor estimates the extent to which outcomes can be attributed to an intervention. As previously noted, an attribution rate of 20% was applied to each benefit (except for healing benefits), to reflect the uncertainty in the identification of the impact of Family Wellbeing program in Yarrabah, in isolation of other social and emotional wellbeing programs implemented in Yarrabah between 2001 and 2021.
The rate of 20% was selected considering that Family Wellbeing was introduced to Yarrabah as one of three targeted suicide prevention initiatives introduced to address the spate of suicides in the mid-1990s – alongside the establishment of Gurriny Yealamucka Health Service, and the Yaba Bimbie Men’s Group. The rate of 20% attribution reflects the assumption that each of these initiatives contributed equally to health and empowerment outcomes in Yarrabah over that period, while also allowing for an additional combined impact of the three interventions alongside other community initiatives and responses. The attribution factor of 20% is relatively conservative based on best practice guidance (Social Value International, 2021) and like evaluations, indicating that the Family Wellbeing program played a supporting – rather than critical role, in producing the measured outcomes. The qualitative evidence produced in previous evaluations confirms that the Family Wellbeing program played, at a minimum, this supporting role.
Given the unique focus of the Family Wellbeing program – compared to the other interventions in Yarrabah, on incorporating a trauma-informed approach to healing and reconciliation an attribution rate of 50% was adopted for healing impacts. In Yarrabah, it is estimated that 80% of the community have connections with the Stolen Generations (Prince et al., 2018) and this history along with entrenched structures of disadvantage remain major challenges to improving health and wellbeing.
Sensitivity analysis
The attribution rate was a central assumption underpinning the results in this analysis.
To understand the impact of the selection of this assumption on the results, a reduced attribution rate of 10%, and an increased attribution rate of 30% were tested.
Despite the low proportion of overall benefits able to be monetised at this time, all Social Return on Investment ratios were positive.
Costs
In the absence of comprehensive costs of delivery of Family Wellbeing in Yarrabah, due to inconsistent and short-term funding, cost estimates were informed by estimates from an analysis of the costs of delivering the program in a remote Cape York community. This most likely overstates the costs of delivering Family Wellbeing in other sites due to the additional costs associated with delivery such as the cost of flights for program facilitators. Other costs not captured in this analysis include the costs to Family Wellbeing program facilitators in the time required to apply for short-term grant funding to continue delivering the program. This funding insecurity has impacts for program facilitators. Costs also consider the research and evaluation activities undertaken at Yarrabah.
Results
Summary of social return on investment results (present value, A$2021).
The largest benefit of the Family Wellbeing program is through the reduced prevalence of suicide in the community, representing half (A$8.67 million) of monetised benefits captured in the period as shown in Table 1.
The Results were reported within the four impact themes of health, wellbeing, and culture; education and employment; families and connectedness; and self-determination and empowerment. Monetised impacts from across these four areas were then summed, and considered alongside program costs, to estimate the social return of the Family Wellbeing program between 2001 and 2021.
Reduced prevalence of chronic disease – reduced alcohol consumption and smoking
In Yarrabah, many Family Wellbeing deliveries were focussed on alcohol rehabilitation, and are delivered by Gindaja, an Alcohol and Other Drug service in the community. Williamson et al. (2023) found that one quarter of Family Wellbeing participants (26.4%) report quitting alcohol, compared to 20.4% of non-participants (n = 9,843). This economic evaluation applied this 6% change to the annual number of Family Wellbeing participants in Yarrabah, with a 20% attribution factor to Family Wellbeing. To estimate the average number of additional healthy years of life gained attributable to quitting drinking, this analysis adopted evidence of the total burden of disease which is attributable to alcohol for Aboriginal and Torres Strait Islander people. In 2011, the total burden of disease attributable to alcohol use disorders among Aboriginal and Torres Strait Islander people was estimated at 14.6 Quality Adjusted Life Years (QALYs) for every 1,000 people (Australian Institute of Health Welfare, 2016). The value of these years of life gained is monetised by applying the value of a statistical life year (VSLY) estimated by the then Office of Best Practice Regulations, now the Office of Impact Analysis. In 2021, the VSLY, was A$220,000 in 2021. The analysis applied a 7% discounted value to account for time. The estimated benefits of Family Wellbeing program in reduced alcohol consumption was A$326,773.
A similar mechanism was applied to benefits gained from participants quitting smoking. The Williamson et al. (2023) study found that 33.4% of Family Wellbeing participants from the Mayi Kuwayu National Study reported quitting smoking, compared to 31.9% of non-participants. This percentage change was applied to the annual number of Family Wellbeing participants in Yarrabah estimated to have been smokers at the time of participation. Approximately half (54.5%) of Aboriginal and Torres Strait Islander people aged 18 and over were current smokers in 1994, which declined to 43.4% in 2018–19. The latter rate was adopted to estimate the share of participants in Yarrabah who were current smokers at the time of participation, and therefore may be a conservative estimate. The discounted value of average number of quality-adjusted life years (QALYs) gained because of quitting smoking were applied, which estimates the average years of life lost for the major causes of death attributable to tobacco smoking (including lung cancer, chronic obstructive pulmonary disease, coronary heart disease, and other cancers and strokes). Based on the average of years of life lost due to these diseases, the average years gained from quitting tobacco smoking were estimated at 12.75 years. This includes deaths from these diseases at all ages, including in old age. The estimated benefits of Family Wellbeing program in quitting smoking were A$1.01m.
Reduced prevalence of suicide in community
The largest benefit of Family Wellbeing in Yarrabah is through the reduced prevalence of suicide in the community, representing approximately half (A$8.67 million) of monetised benefits captured in the period. Family Wellbeing was introduced to Yarrabah following a spate of suicides in the 1980s and 1990s. It was part of a community movement to address this local issue, which included changes to the alcohol canteen, a life promotion program, and the forming of other groups (many of which use Family Wellbeing principles).
Due to a lack of reliable data on suicides in Yarrabah between 2001 and 2021, this evaluation relied on a number of conservative assumptions about the reduced prevalence of suicide in Yarrabah since the introduction of the Family Wellbeing Program. It compared historic prevalence of suicide in Yarrabah to current rates of suicide in Yarrabah and found that 27.4 suicides could be estimated to have been prevented in Yarrabah over 2001 to 2021. It applied a 31% reduction to reflect the national reduction of suicide over this time. It then applied the value of a statistical life and a 20% attribution factor to Family Wellbeing. The total estimated value for benefits arising from reduced prevalence of suicide in the community was A$8.67 m. This figure represents the estimated value of reduced completed suicides and does not account for reduced prevalence of attempted suicides, or broader wellbeing impacts on the rest of the community. It is therefore likely to represent an underestimate of the value of the reduced prevalence of suicide attributable to Family Wellbeing in Yarrabah.
Improved education and employment outcomes for participants
Analysis by Williamson et al. (2023) found a significantly higher level of educational attainment for Family Wellbeing participants (with 57.8 of participants completing Year 12 or above compared to 52% of non-participants, and 40.0% compared to 45.2% completing Year 10). Based on Census data, approximately half of the population aged 15 years and over in Yarrabah had completed a highest level of education at Year 10 or below. Given the uncertainty in the ages and education levels of Family Wellbeing program participants in Yarrabah over the evaluation period, the average benefit to expected lifetime earnings associated with completing Family Wellbeing could be estimated based on the average change in lifetime earnings attributable to Year 12 attainment. Based on the average benefit in net present value (NPV) terms adopted in the Deloitte Access Economics cost-benefit analysis of the Murri School Healing Program (2017), this resulted in an average benefit of A$3,118 in 2016 dollars. This evaluation applied the 55% figure of Yarrabah inhabitants not expected to progress past year 10 to Family Wellbeing participants in Yarrabah, with an attribution factor of 20%. Therefore, the final estimated benefit through education and employment outcomes was A$4.8 m.
Improved sense of healing for participants
The Family Wellbeing Program was developed by, and for, Aboriginal and Torres Strait Islander people, to deal with the legacy of the Stolen Generations and the lasting impacts of colonisation on physical, emotional, mental, and spiritual aspects of life. Although enduring everyday discrimination and entrenched structures of disadvantage continue to contribute to psychological distress among Aboriginal and Torres Strait Islander Australians (Thurber et al., 2021), Family Wellbeing worked on the ‘micro’ level to support healing and empower participants to take control over the decisions they have, improve family relationships and reconciliation within cross-cultural relationships. Some use Family Wellbeing as a ‘stepping stone’ for seeking professional help. Resolving trauma was also hypothesised to protect future generations by breaking the cyclical transmission of violence (Atkinson, 2002).
There has been little attempt to quantify or monetise these healing benefits for Stolen Generations and other Aboriginal and Torres Strait Islander peoples, and it is beyond the scope of this evaluation to do so. Further research into the effects of trauma-informed programs and empowerment programs is needed.
However, as part of reparations and redress packages across Australia, financial compensation has been offered by governments to provide healing assistance and trauma-informed counselling. While these costs do not directly map onto the benefits of healing, and are individual rather than community focussed, they provide some indication of a financial proxy, in that they have similar benefits to the Family Wellbeing program.
In seeking to monetise this benefit, the evaluators looked at Australian and international jurisdictions that offered reparations payments to compensate for the trauma or suffering caused by colonisation, and to support healing by survivors and their families. The Victorian Government has introduced the Stolen Generations Reparations Package, offering a lump sum payment of A$100,000 for persons removed in Victoria, as well as access to supports such as up to 35 hours of trauma-informed counselling (at up to A$180 per hour, to a total of A$6,300). The Territories Stolen Generations Redress Scheme offers a redress payment of up to A$75,000 and a healing assistance payment of A$7,000. This analysis adopted the lower healing assistance payment of A$6,300 as a financial proxy for the value of healing supported by participation in the Family Wellbeing program. This A$6,300 was applied to the framework to a proportion of the Yarrabah Family Wellbeing participants equal to the proportion of Yarrabah estimated to be descended from the Stolen Generations (80%). A factor of 32% was applied to reflect the share of participants in the Williamson et al. (2023) study, reporting moderate or high cultural wellbeing, compared to non-participants. This led to finding an estimated benefit of A$1.15 m from an improved sense of healing for participants. It should be noted that while this methodology gives an illustrative example of value, more research is needed here to accurately capture these benefits, and this is likely a conservative estimate.
Other benefits identified in the social return on investment framework
Other benefits identified in the Social Return on Investment framework were unable to be appropriately quantified or monetised with the available data or evidence. This included improved sense of cultural wellbeing, life satisfaction, financial wellbeing, employment outcomes, family wellbeing, empowerment, self-determination in service delivery, and community research and reduced number of children and adults in the justice system.
To provide a more robust and accurate estimate of the value of Family Wellbeing: • More longitudinal data on Family Wellbeing participants is needed, at Yarrabah, but also at other sites. • More research into the value of cultural wellbeing and wellness is needed. Based on analysis of Mayi Kuwayu survey results, Williamson et al. (2023) found that almost three quarters of Family Wellbeing participants (70.9%) reported moderate or high cultural wellbeing outcomes, compared to 38.9% of non-participants (n = 9,843). Although this has been accounted for in this pilot evaluation by the proxy of healing assistance payments, more evidence is needed not only to prove value for money in Family Wellbeing but in other programs addressing the trauma of colonisation.
Sensitivity analysis
Sensitivity of social return on investment results to attribution rate assumption (present value, A$2021).
Discussion
This paper sought to explore the challenges and opportunities associated with economically evaluating the Family Wellbeing empowerment program, with a focus on the importance of leveraging existing data. The study’s significance should be assessed against its efficacy in aiding local decision-makers to address two pivotal questions crucial to Yarrabah’s comprehensive development plan.
The first question revolves around the appropriateness of Family Wellbeing as a foundational capacity-building tool. The study successfully fulfils its purpose by delving into the challenges and opportunities linked to the economic evaluation of the Family Wellbeing empowerment program. It underscores the utilisation of existing data, amalgamating quantitative data with qualitative insights to formulate a pilot Social Return on Investment analysis. The resulting framework, or theory of change, aligns seamlessly with the firsthand experiences of local leaders who initially commissioned the economic analysis (as presented at the following link: https://www.lowitja.org.au/news/kt-coffee-webinar-community-led-knowledge-translation/). Consequently, the local community-controlled employment agency, Wugu Nyambil, has adopted Family Wellbeing as a core part of its pre-employment training, benefiting over 350 job seekers since 2021. Importantly, the pilot Family Wellbeing theory of change now informs discrete but interrelated evaluation data collection methods designed to complement and reinforce each other. These methods include Family Wellbeing workshop feedback illustrating planned participant changes, follow-up feedback indicating challenges and opportunities in implementing the changes, pre/post Mayi Kuwayu survey results, and routine employment data from the local employment service. The study’s timeliness and relevance are evident in its prompt contribution to decision-making, resonating with local leaders and strengthening the impact evaluation of Family Wellbeing.
Another method to evaluate the study involves considering indicative returns on investments and utilising data collection for economic analysis. The findings suggest that each Australian dollar invested in implementing the FWB program in Yarrabah could potentially yield A$4.60 in benefits for participants and the community, establishing a quantitative foundation for decision-making. This numerical value, derived from the pilot Social Return on Investment study, not only aligns with the intuitive understanding of Family Wellbeing benefits but also empowers decision-makers to advocate for more sustainable resourcing for a program that has always been known to work. Additionally, the successful extrapolation of national Mayi Kuwayu data for the pilot study confirmed the Yarrabah Leadership Forum’s decision to use the Mayi Kuwayu survey for a whole-of-community household survey, effectively monitoring the 6-pillar development strategy in the right direction (Mayi Kuwayu National Study of Aboriginal and Torres Strait Islander Wellbeing, 2024). By reinforcing a commitment to community data collection and ownership, the study not only affirms the appropriateness of Family Wellbeing in the development plan but also promotes a data-driven approach to decision-making.
The innovative use of existing extrapolated data to inform timely decision-making aligns with the Lowitja Institute Research for Impact Tool (Tsey et al., 2016) and the associated Introduction to Indigenous Knowledge Translation online course (Lowitja Institute, 2023a). Developed in response to perennial concerns about the over-researching of Aboriginal and Torres Strait Islander issues without demonstrable benefits, these resources are grounded in Indigenous research principles and norms designed to ensure that research is culturally congruent and that benefits flow in ways valued by Indigenous people (Australian Institute of Aboriginal and Torres Strait Islander Studies and The Lowitja Institute, 2017; Bainbridge et al., 2015). Emphasising the critical role of Indigenous leadership in determining research priorities (i.e. the what, why, how, and the so-what of research), the tool and associated online course encourage research partners to transparently weigh whether existing knowledge is adequate to inform users’ decisions driving the research or if additional evidence is required Lowitja Institute, 2023a; Tsey et al., 2016). If adopted on a broader scale, this approach has the potential to enhance the societal value of research, minimising wasteful endeavours.
In summary, the study’s value is evident in its responsiveness to local needs, alignment with community experiences, and tangible contributions to decision-making processes, particularly regarding the appropriateness of Family Wellbeing as a foundational capacity-building tool and indicative returns on investments. The transparent acknowledgment of data limitations is crucial. Waiting for perfect data, which may or may never materialise, contrasts with the proactive use of available evidence to make informed decisions. The proposed approach, exemplified in the Yarrabah 6-pillar development strategy, signifies a shift towards more pragmatic and impactful research practices. Embracing this paradigm could significantly contribute to minimising research wastage, aligning with broader societal needs, and fostering a more efficient and effective research landscape.
Limitations
Despite these strengths, challenges arose when conducting a pilot Social Return on Investment analysis using existing Family Wellbeing evidence, primarily due to limitations in quantitative data. The Mayi Kuwayu study, while comprehensive in documenting participant outcomes, lacked pre-post intervention data, establishing correlation rather than causation. Uncertainty surrounding the timing and program stage completion further complicated data interpretation. The decentralised and customised nature of Family Wellbeing implementation across Australia hindered precise impact evaluation as the program’s flexible adaptation to local needs posed difficulties in generalising findings.
Beyond data challenges, concerns arose regarding the suitability of the Social Return on Investment framework for capturing the complexity of Family Wellbeing. The framework’s inclination towards monetised impacts risked oversimplifying and undervaluing program impact, especially when numerous benefits couldn’t be quantified or monetised. This emphasises the need for a nuanced understanding of Family Wellbeing’s multifaceted contributions beyond monetary metrics.
Future directions
This pilot Social Return on Investment provides important directions for strengthening the economic analysis of Family Wellbeing over time. Some important impacts of the program could not be quantified or monetised in the analysis with available data. For example, given the demonstrated and quantified impacts of participation in Family Wellbeing on participants’ cultural wellbeing, future evaluations can consider approaches to monetising the value of improved cultural wellbeing. This should be approached through conversations with Aboriginal and Torres Strait Islander stakeholders to create a meaningful valuing of cultural wellbeing in monetary terms and communicate the impacts that matter to Aboriginal and Torres Strait Islander communities. Longitudinal participant data could be used to better understand the attribution of Family Wellbeing to the expected impacts, with a more precise attribution factor able to be applied to each benefit. The identification and routine gathering of additional community level data would strengthen the evidence for community level impacts.
There is an opportunity in long-term studies such as this to push complex evaluation techniques even further, as is the plan here over the next five years. With supporting data collection, we’ll be able to blend the rich qualitative insights with the quantitative (Mayi Kuwayu data) to explore in detail the way in which programs like Family Wellbeing impact communities and individuals, and how this varies over contexts and time. This supports the findings of Corvo et al. (2022) that the future of effective Social Return on Investment analysis lies in the ability to effectively integrate monetisation methods alongside participatory and qualitative methods which allow for nuanced and complex understandings of impact.
Conclusion
Whilst there are limitations in this economic analysis, and there is scope for future research to strengthen the findings, this pilot Social Return on Investment can be seen as sufficiently robust to guide decision-making and resource allocation, particularly when compared to using no economic evidence at all.
Demand is growing in the public, private, and philanthropic sectors for a stronger evidence base articulating the value of differing social interventions, to help guide decisions about where limited time, money, and other resources can best deliver outcomes. However, the mechanisms with which to measure, translate, and communicate value, particularly for complex social interventions, are still maturing into a cohesive (and comparable) body of evaluative research. To produce the economic modelling sought by the decision-makers of today, it is necessary to capitalise on the existing evidence and data, and to optimise the use of qualitative research conducted in these fields through the use of methods such as Social Return on Investment which allow for the translation of qualitative evidence into economic arguments. This is especially critical where benefits are likely to be inter- or multi-generational, and therefore take years to be realised, or where outcomes are more often analysed qualitatively, which may be more culturally appropriate.
This study presents an approach to combining existing qualitative and quantitative data for the purposes of indicating a value range for the Family Wellbeing Program in Yarrabah. The analysis finds that by designing and applying a benefits framework to program outcomes identified by research, Australian dollar values can act as proxies for the value of some, but not all, outcomes. This translates the long-term nuanced outcomes of the program into an economic, in addition to a social justice, argument. As with any language, some meaning may be lost in the process of translation and careful presentation of the final Social Return on Investment ratio alongside a discussion of what benefits could and could not be captured is required.
In the case of the Family Wellbeing Program in Yarrabah, the ratio of 1:4.6 suggests that the investment is likely to be ‘worth’ the costs of further program delivery and extending quantitative research. Further research into the program at Yarrabah will not only strengthen the findings of this study but validate and test the suitability of this approach for other long running programs with sufficient qualitative and quantitative evidence.
Footnotes
Acknowledgements
All community members who have participated in Family Wellbeing and/or supported the process of research.
Author contribution
The contributions made to this work by the Mayi Kuwayu National Study of Aboriginal and Torres Strait Islander team, particularly Professor Ray Lovett and Ms Leonie Williamson.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Lowitja Institute for Aboriginal and Torres Strait Islander Health Research; and Deloitte Access Economics.
