Evaluating Ngaramura: An Aboriginal-Led Collaborative Approach to Understanding a Culturally Based Educational Program for Young Aboriginal Students Experiencing Difficulties Within Mainstream Education

Abstract

Ngaramura is an innovative, Aboriginal-designed, strengths-based education program for Aboriginal young people. This unique program addresses the complex and seemingly entrenched issues of disengagement and disproportionate suspension of Aboriginal students from high schools, tackling challenges faced by students in mainstream school environments that fail to meet their needs. Derived from the Dharawal word meaning ‘see the way’, Ngaramura provides a supportive pathway, assisting young people to re-engage with education through a cultural learning framework. Ngaramura is delivered by the Coomaditchie United Aboriginal Corporation an Aboriginal Community Controlled Organisation, in the Illawarra region of New South Wales. Coomaditchie invited our Aboriginal-led research team, Ngarruwan Ngadju (University of Wollongong) to conduct this culturally safe collaborative evaluation. This article locates Ngaramura in a place-based context, in which Aboriginal connection to place is an essential component of successful educational initiatives for Aboriginal youth. The 3-year evaluation (2018–2020) formed part of an Australian Research Council and NSW Health COVID research program. Drawing on an Indigenist research and evaluation methodological approach that centres Indigenous ways of knowing, being and doing, the process and outcomes evaluation incorporated cultural protocols, a collaboratively designed program logic model and mixed methods ethnographic data approach. The article provides important the lessons for undertaking culturally safe collaborative evaluation.

What we Already Know

There is an urgent need to increase the number, and improve the quality, of evaluations of Indigenous programs that address priority policy areas in Australia.

Few peer reviewed publications provide detailed accounts of Aboriginal-led collaborative approaches to evaluation.

Reporting on the Closing the Gap strategy over more than a decade demonstrates the failure of mainstream education institutions and practices to address the needs of Indigenous young people.

The Original Contribution the Article Makes to Theory and/or Practice

The evaluation of Ngaramura demonstrates the positive impacts of a place and culturally based, Aboriginal community-led program for Aboriginal young people experiencing difficulties within mainstream schools.

The evaluation provides a detailed example of an Aboriginal-led collaborative evaluation of an Aboriginal designed community-based education program.

One of the critical aspects in this Indigenous evaluation practice was relationality. Building on a longstanding relationship with a local Aboriginal Community Controlled Organisations, ensured a culturally safe approach which created opportunities for collaboration within the evaluation process and provided meaningful outcomes for all stakeholders.

Introduction

The evaluation of Indigenous programs in Australia has received increasing attention in recent years (Australian National Audit Office, 2019; Kelaher et al., 2018; Luke et al., 2020; Maddox et al., 2021; Productivity Commission, 2020). Growing awareness highlights the urgent need to increase the number and improve the quality of evaluations of Indigenous programs in Australia that address priority policy areas (Campbell et al., 2007; Coalition of Peaks, 2020; Department of the Prime Minister and Cabinet, 2018; Dudgeon et al., 2010; Productivity Commission, 2020, 2024). Indigenous evaluation has also been acknowledged for its critical role in strengthening the evidence base for effective community-based approaches and for guiding culturally informed approaches, methodology and commissioning practices that contribute to programs delivering better outcomes for Indigenous people (Cargo et al., 2019; Finlay et al., 2021, 2023; Vine et al., 2023). In education, there is a critical need for Indigenous evaluations focused on improving the outcomes for Aboriginal and Torres Strait Islander young people. This article contributes to these important areas by presenting the results of an Aboriginal-led collaborative evaluation of the Ngaramura See the Way Program (Ngaramura) undertaken over a 3-year period (2018–20), that encompasses the COVID-19 pandemic years (Clapham et al., 2022a, 2022b).

Ngaramura is an innovative, Aboriginal designed, community-based education program developed in 2018 by the Coomaditchie United Aboriginal Corporation (Coomaditchie), a local Aboriginal Community Controlled Organisation (ACCO) in the Illawarra region of New South Wales (Coomaditchie United Aboriginal Corporation, 2024). Ngaramura addresses the significant disparity between educational and employment outcomes for Indigenous young people. It addresses the complex and seemingly entrenched issues of disengagement and the disproportionate suspension of Aboriginal students from high schools, tackling challenges faced by students in mainstream school environments that fail to meet their needs (Anderson et al., 2025; Shay & Poed, 2024). Ngaramura addresses the unique challenges Aboriginal students encounter within mainstream school environments by providing a supportive pathway to assist young people disengaged or at risk of becoming disengaged with education (including suspension or risk of suspension) to re-engage with education through a cultural learning framework.

Based on its longstanding relationship with Coomaditchie, an Aboriginal-led research team from Ngarruwan Ngadju First Peoples Health and Wellbeing Research Centre at the University of Wollongong, was invited to provide a culturally safe collaborative evaluation that would provide meaningful outcomes for Aboriginal students, their parents and carers, local schools and the Coomaditchie organisation. The Ngaramura evaluation formed part of a larger program of study funded by the Australian Research Council (ARC, DPI IN190100026) and NSW Ministry of Health COVID-19 grant and was led by a research team comprising seven Aboriginal researchers (KC, FS, MW, BF, DB, ML, BH) and three non-Aboriginal (VH, KS, PK) researchers. This larger program of study investigated Indigenous community led solutions to complex health and social issues (Clapham et al., 2024a, 2024b), exploring the deep interconnectedness between place and health, and the important role of Aboriginal Community Controlled Organisations, as place-based entities that deliver a range of health and social services to local Aboriginal communities across Australia.

The purpose of the Aboriginal-led collaborative evaluation was:

• Provide evidence for the effectiveness of the program in terms of the contractual targets, particularly around number of young people attending and the number of Indigenous people employed.

• Describe how the program was implemented including detailed narrative accounts of how the program objectives were met through the conduct of the program; and

• Provide feedback to the organisation for program improvement and ongoing strategic planning.

Ngaramura was funded through the Australian Government’s Indigenous Advancement Strategy (IAS) (Department of the Prime Minister and Cabinet, 2018), then managed by the Department of Prime Minister and Cabinet, under the priority area focused on the positive impact that education has on the future success of individuals, families, and communities. The original funding was for a 2-year period (April 2018–2020) with the option of a 1-year extension (April 2020–21). Funding continued throughout 2024, with a further extension, but at the time of writing, ongoing funding is not secure. Despite having successfully established a culturally based model that supports students to re-engage in education and provide a service that improves the school environment, Coomaditchie has had to continually seek ongoing funding to sustain the Ngaramura program and share its successes. Given the lack of funds for the program and its evaluation, this article demonstrates the importance of collaborative research in the Ngaramura evaluation.

The Policy Context of Educational Injustice for Aboriginal Young People

We write this article in the aftermath of the devastating defeat of the Australian Referendum for recognition of Australia’s First Peoples in the Australian Constitution (Nakata, 2024) foreshadowing a period of change and uncertainty about what lies ahead in the struggle to address the educational injustices faced by Aboriginal and Torres Strait Islander people. The policy context for this evaluation is centred on addressing the enduring inequities in school education faced by Aboriginal children and young people, their families, and communities (Education Council, 2019).

Since 2008, Closing the Gap has been the overarching national framework to Indigenous affairs in Australia when all Australian governments, through the Council of Australian Governments agreed to work together towards a better future for Aboriginal and Torres Strait Islander peoples (Council of Australian Governments, 2008, 2009). Three of the original Closing the Gap targets are of relevance to Ngaramura: Year 12 Attendance; Reading and Numeracy; and School Attendance. In 2020, towards the end of the Ngaramura evaluation period, the Australian Government reported that it was not on track to meet these targets, reporting that majority of Indigenous children and young people attended school for an average of just over 4 days a week in 2019 and lived largely in major cities and regional areas. School attendance rates for Indigenous children and young people had not improved in the previous 5 years. Attendance rates remained lower than for non-Indigenous students (around 82% compared to 92% in 2019). Furthermore, that the attendance gap began in first year of schooling and widened during secondary school. In 2019, the attendance rate for Indigenous primary school students was 85%—a gap of around 9% points. By Year 10, Indigenous students attend school 72% of the time on average—a gap of around 17% points (Australian Government, 2020). Education continues to be an issue needing to be addressed; the most recent annual Closing the Gap Report recorded some improvements, but the Government is not on track to meet the 2031 target for Outcome 5 of the National Agreement - Aboriginal and Torres Strait Islander students achieving their full learning potential (National Indigenous Australians Agency, 2025).

The increasing dissatisfaction with the Closing the Gap arrangements, the narrowness of its targets, and the accusation that the strategy ignored the real issues underlying Indigenous disadvantage, led to a strategy rethink. In December 2018, the Council of Australian Governments committed to establishing a formal partnership with a group of Aboriginal Torres Strait Islander community-controlled organisations, known as the Coalition of Peaks. This led to the signing of the Partnership Agreement on Closing the Gap 2019–2029 (Council of Australian Governments & Coalition of Aboriginal and Torres Strait Islander Peak Organisations, 2019). In July 2020, a new National Agreement on Closing the Gap was signed by the National Federation Reform Council (NFRC) and the Coalition of Peaks in 2020 (Coalition of Peaks, 2020). The new National Agreement was based on formal partnerships and ‘shared decision making’ in government and Aboriginal and Torres Strait Islander representatives working together. In August 2021 the Commonwealth of Australia (2021) released the first the Closing the Gap: Commonwealth Implementation Plan outlining the key approaches for education in Australia for Aboriginal and Torres Strait Islander children and young people. It emphasises the vital importance of valuing culture, respecting culture and of having culture visibly present (Commonwealth of Australia, 2021). The Closing the Gap refresh in 2020 established four new targets and 19 socio-economic indicators, including Youth Engagement, a measure of the number of people aged 15–24 fully engaged in employment, education or training (Productivity Commission, 2025)

The Ngaramura ‘See the Way’ Program

Ngaramura in the Dharawal language means “see the way”. Ngaramura is delivered from the Coomaditchie Hall, in Kemblawarra, NSW, located next to the Coomaditchie Lagoon and close to Lake Illawarra; places which have cultural and historical significance to the local Aboriginal people. Images 1 and 2 show Coomaditchie Hall and the surrounding area. The Dharawal language area can be seen on the map of Indigenous Australia found at https://aiatsis.gov.au/explore/map-indigenous-australia (Horton, 1996).

Image 1.

Coomaditchie Hall With Lake Illawarra in the Background

Image 2.

Coomaditchie and Surrounds

The program is a collaboration between Coomaditchie, and five local high schools located in suburbs surrounding Lake Illawarra. Participating schools are part of the mainstream public schooling system in NSW. ‘Mainstream’ school environments, whether public (government funded) or independent (private fee paying) are part of the colonial enterprise of the invasion and colonisation of Australia. We describe this context of schooling since this is in line with our commitment to a Truth Telling process (Morris et al., 2023). With this understanding we can appreciate specialised Aboriginal responses are needed to address the negative consequences of mainstream education on Aboriginal children and young people in Australia (Anderson et al., 2025). Importantly, in this context Ngaramura recognises that disengagement from school is risk, that calls for a different approach.

Ngaramura provides a strengths-based educational and cultural learning environment that responds to the needs of Aboriginal young people who experience challenges in their school environments. The term ‘strengths-based’ is a broad term that refers to approaches that seek to move away from problem-based or deficit narratives of Indigenous peoples and societies (Fogarty et al., 2018). Emerging research evidence shows that Aboriginal and Torres Strait Islander people value programs that are: culturally safe, holistic, and take a strengths-based approach (Murrup-Stewart et al., 2025). Arts-based programs, connection to culture activities, yarning and storytelling, mentoring from Elders or other community members and skills building, for example, have been found to contribute to positive social and emotional wellbeing (SEWB) for Indigenous people (Kennedy et al., 2022; Murrup-Stewart et al., 2019).

In a culturally safe and structured environment Ngaramura provides Aboriginal young people with a range of learning activities with cultural teachings and cultural engagement a mainstay, alongside academic schoolwork from, living and social skills, cooking, gardening and sport. Young people are engaged in a mix of small groups and/or one-on-one activity. For example: they may be working on individual school activities with Ngaramura staff assisting them; engaged as a group with the Elders and Ngaramura staff in a cultural activity such as painting, storytelling, book publications; working as a team to make lunch; or in outdoor activities on Country. A broad range of partner organisations are engaged to assist in delivery of learning activities.

Art and painting at Coomaditchie, and in Ngaramura is inextricably connected with culture At Ngaramura the young people are regularly in an environment where the accomplished artists of Coomaditchie can be seen painting and where the artwork, which is of cultural subjects, is displayed on the interior walls in the main hall of the building. The Coomaditchie artists hold exhibitions in the Coomaditchie Hall and in local indoor and outdoor venues. In 2024 Coomaditchie hosted a major exhibition Coomaditchie the Art of Place at the Museum of Sydney (Museums of History New South Wales, 2025).

The young people’s art is connected into authentic learning activity at Coomaditchie, with works included in a range of outputs from paintings and published books to structures in the environment such as murals. Describing Ngaramura, the young people brought up the theme of culture as a key activity. Jack described how he was “learning about my culture, painting, learning about our Aboriginal ancestors. And all stuff like that”. These experiences occur in the context of positive relationships with the Ngaramura staff. For instance, Max explained,

Workers here are good… Well, I mean we can really be ourselves and they won’t really judge us on anything like that. Not – they're not like teachers or anything. More laid back really than teachers are...

Tom, when asked how the Ngaramura staff had helped him stated, “Them being cultural”. Mia talked about the connection between learning about culture at Ngaramura and maintaining connection to school, “painting, learning about culture, they helped me get through Year 10”. Max described how 1 day at Ngaramura he had “done some painting. Some [schoolwork] to catch up on. Started a garden up there. Painting a mural for the shed that’s up there as well”. The young people’s artistic cultural practices are also recognised more widely. An example of tangible success in art is when one of the young people won an art competition with a $1000 prize.

Methodology

An Indigenist Paradigm

The growing trend in Indigenous-led evaluation establishes clear benefits and greater self-determination for Indigenous communities (Finighan, 2019; Hurworth & Harvey, 2012; Williams, 2018). This trend has been influenced by decades of development of Indigenous scholarship and decolonising research methodologies (Battiste, 2008; Dudgeon et al., 2010; Durie, 2005; Fredericks et al., 2011; Martin & Mirraboopa, 2003; Rigney, 1999; Smith, 2021). Indigenous communities have a long history of being seen as the ‘other’, of being ‘researched on’ and having their experiences and lifestyles interpreted by outsiders (Smith, 2021). Indigenous scholarship demands that we pay attention to who carries out, and who benefits from, research in Indigenous communities. Specific Indigenous ethical codes designed for research (AIATSIS, 2020; NHMRC, 2018) also articulate the principles and frameworks for Indigenous evaluation that better reflect Indigenous ways of knowing, being and doing (Cargo et al., 2019; Marri et al., 2014; Williams, 2018).

The Ngaramura evaluation was guided by Indigenous research methodologies and ethics and utilised an Indigenist paradigm that combines Indigenous and western knowledge systems. The scholarship around Indigenism in Australian research spans several decades, including pioneers such as Aileen Moreton-Robinson, Martin Nakata and Lester Irabinna Rigney (Moreton-Robinson, 2000; Nakata, 2007a, 2007; Rigney, 1999) who defined indigenist as ‘the body of knowledge by Indigenous scholars in relation to research methodological approaches’ (Rigney, 1999, p. 1). The subsequent growth in Indigenist scholarship internationally means that they do not form a homogeneous group. Rix et al. (2019) define the term broadly; ‘Indigenist research respects and honours Indigenous ways of knowing, being, and doing through using methods that are informed by, resonate with, and are driven and supported by Indigenous peoples’ and is characterized by approaches grounded in relationality and the inclusion of Indigenous ways of communicating such as storytelling or “yarning” (Bessarab & Ng’andu, 2010; Rix et al., 2019, pp. 254–255).

The concept of relationality has been widely discussed in the Indigenous research literature and seen as a central component to an Indigenous decolonising and self-determining research paradigm (Kovach, 2009; Martin, 2008; Wilson, 2008). Relationality is about the connections that inform all parts of the research process. It is also about acknowledgment and responsibility. The research team drew from the longstanding existing relationships between the lead and other team members and the Coomaditchie Elders and staff, acknowledging their wisdom, leadership and cultural knowledge. Relationality is also central to a holistic view of Indigenous knowledge systems, which affirms the interrelatedness between life, land, culture, and spirituality (Dudgeon, 2020, pp. 105–106). This resonated with the way Ngaramura holistically approaches young people; by linking the processes of learning, wellbeing, connection to Country and Aboriginal identity. Responsibility comes with relationality, and this demands integrity in the research and evaluation process. Researchers are required to demonstrate merit in terms of appropriate methods, the evaluation is based in current literature and is conducted by appropriately qualified and experienced teams (NHMRC, 2018, p. 12).

The underlying principles of reciprocity, respect and responsibility are strongly aligned with the worldviews, practices, and values of Indigenous people, and with ensuring culturally safety. Culturally safety in evaluation has been defined as ‘an experience determined by First Nations peoples when they are in situations where their presence is welcomed and respected, their experiences are believed and validated, their cultures are centred and valued, their knowledges and skills are recognised and supported, their advice is listened to and acted upon, and they do not experience racism in any form’ (Gollan & Stacey, 2021). This demands critical self-reflection from evaluators by acknowledging colonisation and system racism, broad social and economic impacts on Indigenous health, and acknowledging individual bias (Australian Health Practitioners Regulation Agency (AHPRA), 2024).

Collaborative Evaluation

The Ngaramura evaluation also drew from participatory and collaborative evaluation principles developed by Rodriguez-Campos and colleagues (Rodríguez-Campos, 2012; Rodriguez-Campos & Rincones-Gomez, 2013). Recognising that evaluation is dynamic, and needs to adapt to changing environmental demands, these principles are intended as guidelines, with the following component parts: identify the situation; clarify the expectations; establish a collective commitment; ensure open communication; encourage effective practices; and follow specific guidelines. In an Indigenous evaluation context, there are numerous advantages to using this methodology including, access to information, quality of information gathered, opportunities for creative problem-solving, and receptivity to findings (Rodríguez-Campos, 2012, p. 253).

Indigenous program evaluation creates an opportunity for evaluators to provide benefits to the community by contributing to program quality improvement. This enables the development of culturally informed evaluation outcomes of Indigenous programs defined in collaboration with Indigenous program partners, which may be place-based, that is specific to the community and its context and linked to the core goals of what the program is trying to achieve. In this evaluation, collaboration with Ngaramura occurred at all stages, a key success criteria noted by Finlay et al. that is “from identifying the need for an evaluation to the final reporting” (Finlay et al., 2023, p. 225).

The collaborative evaluation approach may also present challenges, such as ensuring there is sufficient time for the evaluation and program teams to establish a working relationship and form a shared understanding of the task at hand. Therefore, commitment to ensuring trust in the relationship, adherence to Indigenous research ethical protocols (AIATSIS, 2020) often in resource poor environments, is of fundamental importance.

The research team also drew on their previous experience of conducting Aboriginal led collaborative evaluation, inclusive of the logic model, data collection, analysis, and dissemination (Clapham et al., 2014; Clapham et al., 2017; Clapham et al., 2018; Clapham et al., 2021; Clapham et al., 2022a; Kelly et al., 2022; Longbottom et al., 2018; Senior et al., 2009; The AIME Evaluation Team et al., 2013) . It also supported capacity building, for example in program and qualitative data collection, and the employment of an Aboriginal community researcher to enhance her skills in research and evaluation through each stage of the evaluation and discussions around emerging results. Coomaditchie elders and program staff were supported by the research team to co-present findings at international and national conferences and local events (Bessarab et al., 2023; Clapham et al., 2022b, 2022c; Sheppeard et al., 2022).

Relationships

Aboriginal leadership was present in all aspects of the evaluation and played a critical decision-making role in choosing the evaluation design. The authorship team included six Aboriginal researchers, four of whom are senior Aboriginal academics. Two of the Aboriginal research team were also postgraduate students (PhD and Master of Indigenous Health) who received support for capacity building. Three non-Aboriginal research team members all worked within the Aboriginal-led leadership structure. Aboriginal Elders and program staff from Coomaditchie were involved in all stages, particularly in the work around the design of the logic model. The steering committee included Aboriginal representatives from the Aboriginal Community Controlled Organisations participating in the larger ARC and NSW Health COVID studies included the Ngaramura evaluation.

Our experience demonstrates that Indigenous collaborative evaluation takes time to build trusting relationships and commitment on the part of both evaluators and the collaborating partners. This can be facilitated by geographic proximity; the lead Aboriginal researchers and assistants were Aboriginal community members in the region. The non-Aboriginal researchers have relationships with the Aboriginal local communities in the Illawarra and have collaborated on previous research and evaluation projects (Clapham et al., 2024a, 2024b; Harwood et al., 2015, 2024; Kelly et al., 2022; Sheppeard et al., 2025). Importantly collaborative evaluation demands reciprocity the ongoing involvement throughout the evaluation; adherence to collaborative and ethical principles ensures that this occurs throughout the evaluation process.

Prior to commencing data collection, the research team committed time and resources to understanding the program and establishing a shared understanding of the needs of the community organisation; the aims and objectives of the program; and the contractual obligations. During the evaluation the team established a working relationship with the program coordinators for the collection of program data; made frequent visits to the site to establish rapport; and negotiated appropriate access to young people, parents/carers, program staff/schools for their consent to collect their data. A culturally safe working relationship was fostered through the joint leadership of the project; ongoing critical reflection of the evaluation practice was enabled through the members of the research team informally checking in with Ngaramura program staff, and through steering committee meetings.

Members of the research team worked closely with Ngaramura to ensure that important moments in the flexibly delivered program were recorded, orally, in writing or visually. The two teams also worked to ensure that the de-identification of student data did not create an administrative overburden for the stretched program staff. The research team’s regular meetings with Coomaditchie also ensured that the evaluation was part of the organisation’s core business and controlled by the organisation not something being conducted by ‘outsiders’. This is a problem which Price et al. (2012) identify as a major issue with numerous evaluation practice carried out in Aboriginal contexts (Price et al., 2012).

The evaluation team prioritised a collaborative evaluation design that which ensured ongoing engagement with the program and the organisation throughout the evaluation period. This approach made sure that the evaluation questions were not just those devised by the evaluation team but included an iterative process of understanding the changing circumstances in which the program was being rolled out. The research team designed the evaluation as needing to work from the perspective of the ‘culture’ of the organisation, and to gain the best possible understanding of the context in which the program was being conducted. This enabled the research team to gain a better insight of how and why the program was being implemented in particular ways, the decisions that were being made around resources, and an understanding of emerging issues that may have impacted the program and were critically important to its success and sustainability.

The Covid-19 Pandemic had a significant impact on the delivery of Ngaramura and the evaluation. From July 2018 to July 2020 Ngaramura was delivered at the Coomaditchie Hall. In March 2020, when the COVID-19 pandemic caused major disruption to the program, the Ngaramura team responded quickly, modifying the program from face to face to online delivery, supporting Aboriginal young people who had been previously suspended and disengaged from school, to start to navigate the online environment. Like other research teams over this period (e.g., Best & Fredericks, 2021; Fredericks et al., 2021), we had to develop novel ways to collect data, such as regular online sessions with Ngaramura staff, and access to online program recordings.

Data Collection Design

A mixed methods process and outcomes evaluation of Ngaramura was undertaken over the 3-year period (2018–2020). Data collection design and design elements were selected through a collaborative process to best support the evaluation as described below.

Stage 1 Development of the Logic Model

The main objective was to develop a program logic model as a visual map which represents the shared understanding of the relationships between investments or inputs, activities, and the intended short, medium, and long-term outcomes. The logic model also helped to articulate the program theory of how the intended outcomes would be achieved through the planned activities. Further detail of the Logic Model is provided in the following section.

Stage 2 Mixed Methods Ethnographic Approach

Data was collected from a variety of sources, which we have grouped into three data types: (1) Ngaramura program data including teaching plans and reports; (2) qualitative participant data using interviews, yarning methods, ethnographic methods and participant observation; and (3) quantitative descriptive participant data based on student records.

Data Type 1: Program Data

Documentary analysis of plans, notes, and reports, regular performance reports and good news stories from reports to the funding body. All program data was de-identified by Ngaramura staff. A range of administrative records documented the support provided to the young people including student action plans developed with the referring school; case notes; monitoring reports, feedback from schools on student progress following participation in Ngaramura; and student reports. This program data enabled the evaluation team to gain an understanding of the referral processes, how the young people were engaged in Ngaramura, how schools engaged with Ngaramura and the young people, and follow up with the young people who returned to school. Documentary analysis of teaching plans, and good news stories (taken from reports to the funding body) were also collected over a 3-year period.

Data Type 2: Qualitative Participant Data, Yarning and Ethnographic Methods

Qualitative participant data was collected from 22 participants using yarning methods, and ethnographic methods including participant observation. Yarning and interview participants included: adult participants in yarns at Ngaramura (staff/volunteers n = 8); yarns with young people at Ngaramura (n = 5); interviews with teaching staff from the schools participating in Ngaramura (n = 8); and parent and carers (n = 1). Participant numbers are reflective of the small program size, and with the evaluation focussed on the Ngaramura program itself, in-depth data collection on the program delivery was prioritised. Data collection with the parent/carer group was restricted to one parent/carer. This low number can be interpreted as primarily due to personal and family related constraints, given the complexity of issues the young people and their families were experiencing. As outlined previously, the program maintained a respectful approach to participation so the decisions of individual parents and carers of young people not to participate or not participate in the evaluation was respected. A second factor was the impact of COVID-19 restrictions in the later part of the evaluation, that inhibited widening recruitment efforts to further engage parents/carers in the evaluation activities. Based on discussions with the Ngaramura staff, and given the evaluation focus on the program itself, this part of the data collection was not pursued further.

Fieldwork was conducted on site at Ngaramura during program days, in observations at meetings, and participant observation of Ngaramura sessions with the young people. While semi-structured interviews were used with non-Indigenous school teaching staff, Yarning methods were used at Ngaramura with all Aboriginal participants. Yarning methods have become an established data collection method within an Indigenous methodological approach. As Bessarab and Ng’andu (2010) explain in their seminal article, “(a)cross Australia, Aboriginal people constantly refer to and use yarning in the telling and sharing of stories and information” (p. 38). Applied in Indigenous research, yarning is,

…an informal and relaxed discussion through which both the researcher and participant journey together visiting places and topics of interest relevant to the research study. Yarning is a process that requires the researcher to develop and build a relationship that is accountable to Indigenous people participating in the research. (Bessarab & Ng’andu, 2010, p. 38)

Bessarab and Ng’andu (2010) outline four types of yarning methods: Social Yarning, Collaborative Yarning, Research Topic Yarning and Therapeutic Yarning. The yarning methods used in our project were Social Yarning and Research Topic Yarning. Social yarning,

…takes place before the research or topic yarn is informal and often unstructured, follows a meandering course that is guided by the topic that both people choose to introduce into the discussion... It is usually during the social yarn that trust is developed, and the relationship is built. The researcher is accountable to the research participant. (Bessarab & Ng’andu, 2010, p. 40)

Research Topic yarns were introduced after the researchers had engaged in Social Yarning. With the young people, in some instances these occurred on the same day, for others it was appropriate for these to wait for another occasion, after more Social Yarning. This is because it was important for relationships to be made with the young people before the yarning about the research occurred. Research Topic Yarning is a,

(y)arn that takes place in a un or semi structured research interview. The sole purpose is to gather information through participants’ stories that are related to the research topic. While the yarn is relaxed and interactive it is also purposeful with a defined beginning and end. Research topic yarning is a conversation with a purpose. The purpose is to obtain information relating to the research question. (Bessarab & Ng’andu, 2010, p. 40)

Regular ongoing in-depth yarns (Social and Research Topic) were held with the two Ngaramura coordinators during fieldwork visits, on the phone and, during COVID-19 restrictions, via internet video calls.

Researchers used ethnographic methods such as the recording of notes and observations at meetings and committees, and participant observation of Ngaramura sessions with young people. Ethnographic methods enabled the researchers to better understand program activities and observe the impacts on participants. Key findings included documenting Ngaramura pedagogical approaches that engaged the young people in education within a cultural framework.

Data Type 3: Quantitative Participant Data

Quantitative participant data for each young person who referred to Ngaramura was collected over the 3-year period 2018-2020 (n = 87). The funding agreement required Ngaramura to engage an average of 25 Aboriginal young people per year. Engagement numbers were impacted during the 2020 COVID-19 shutdown of Ngaramura. Taking this impact on engagement numbers in 2020 into consideration, Ngaramura clearly exceeded this requirement, with a total attendance of 87 young people over the 3 years.

Significantly, the descriptive data provides insight on the young people’s patterns of engagement with Ngaramura. For instance, while the number of young people referred to and attending the program over the 3-year period totalled 87, program data revealed attendance patterns were more complex. In 2018 for example, the program commenced offering places in Term 2 of the school year (NSW schools have four terms per year of approximately 10 weeks each). In its first term offering programs, Ngaramura successfully supported engaging 18 young people to attend. However, this figure alone, does not reveal the full ‘picture’ of activity and positive educational connections that occur at Ngaramura. We found it was important to understand the patterns of attendance such as the number of days attended by each young person. Attendance ranged from 3 to 46 days; while the average number of days attended were 12.5 days, some young people required considerably more days, with the greatest number being 46 days. This is important as it shows that the needs for each young person can vary, and that programs such as Ngaramura need to be responsive to such variation.

The program description above demonstrates the importance of working closely across a range of data types to produce a comprehensive account of Aboriginal programs such as Ngaramura. Limitations include the small number of some types of participants: only one parent consented to participate in a research yarn and only one Aboriginal Education Officer participated.

Research Ethics

Ethical processes were incorporated across all stages of the evaluation. Research was guided by the AIATSIS Code of Ethics for Aboriginal and Torres Strait Islander Research (AIATSIS, 2020). Ethical approval was obtained from the University (name removed) Human Research Ethics Committee (HE/2018/328). The Ngaramura Steering Committee, consisting of Indigenous and non-Indigenous members of the research team, Coomaditchie and other regional Aboriginal Community Controlled Organisation representatives. The Steering Committee provided oversight of the evaluation throughout the evaluation period and assisted the evaluation team.

Ngaramura Logic Model and Theory of Change

Using an Indigenist approach the Ngaramura evaluation drew both western and Indigenous science. Program logic has been utilised by Indigenous evaluators, particularly in policy where there is a need to build the evidence base around Indigenous program areas (Arabena et al., 2019; Clarke et al., 2021; Nadeau et al., 2023). In this evaluation we adopted the western program logic as a tool for collaboration. The development of the Ngaramura logic model was guided by a model of collaborative evaluation (Gugiu & Rodriguez-Campos, 2007; Rodríguez-Campos, 2012; Rodriguez-Campos & Rincones-Gomez, 2013). The research team spent considerable time in the initial planning stages identifying the situation and clarifying expectations. Through individual and group face to face meetings with program staff and their advisors, at the Coomaditchie Hall, we identified key stakeholders, gained a full understanding of the scope and processes and clarified the respective roles of the evaluators and program staff. This paved the way to establishing a shared vision for the evaluation that was operationalised through the development of the Ngaramura logic model.

To develop the program logic model, we conducted a series of semi-structured interviews with program staff, with questions derived from the peer reviewed literature (Gugiu & Rodriguez-Campos, 2007). We used the answers to these questions to construct the initial model that was presented in a group discussion. The model did not a assume or unchanging program. Like most community programs Ngaramura is dynamic and continually evolving; the logic model provided a ‘snapshot’ in time and was revised over the course of a program’s life. This also generated further learnings and program adaptations from the evaluation findings. This process was done collaboratively with involvement of key stakeholders.

Once we reached agreement about the model, we designed the evaluation framework that set out the key questions for the evaluation and identified sources of data, the tools and processes for data collection and modes of analysis. Collaboration around the development of the logic model and evaluation framework involved the team workshopping the desired program outcomes and how to evaluate them in face-to-face group sessions. These were also aligned to the agreed key deliverables in the program funding agreement. This process involved detailed and nuanced discussions around the alignment of ‘the Coomie way’ and the expectation of funders, so that the cultural underpinnings of the program were in no way compromised. Data collection methods were chosen to ensure that the methods used were not only valid and reliable, but also feasible in terms of the burden on the program staff. Importantly, the data collection methods needed to be respectful and culturally safe with due consideration of the sensitivities of working with the Aboriginal young people and their families who have been impacted by the history of colonisation, social exclusion and ongoing systemic racism.

The completed Ngaramura logic model provides a visual representation and a shared understanding of the relationships between investments or inputs, activities, and intended outcomes for the program. The primary purpose of a logic model is to articulate the underlying assumptions about how the expected outcomes of a program will be reached, through the planned activities, in the short, medium, and long term (Gugiu and Rodriguez-Campos 2007). The underlying program theory explains how the program works and how the components of the program work together; it also helps to inform and guide the development of the evaluation framework (Weiss, 1972). The six components make up the Ngaramura logic model: Inputs; Activities; Participation; Outcomes; Assumptions; External Environment can be seen in Figure 1 (below). The logic model also provided a framework for the analysis of the evaluation data, enabling the evaluation team to report on each of the short and medium outcomes for each of the for main participant groups – students, families, Coomaditchie and participating schools. It also addressed the longer-term impacts expected to occur. For the analysis of outcomes and impact we drew on the sources of data identified above.

Figure 1.

Ngaramura Program Logic Model

Dissemination

A plan for the dissemination of the evaluation findings, developed by the research team, in collaboration with Ngaramura and the Steering Committee. resulted in a multi-level strategy that sought to report on evaluation outcomes to government agencies and to disseminate findings to the research, indigenous and health audiences. Dissemination activities included a Final Evaluation Report (Clapham et al., 2022a); Community Report (Clapham et al., 2022b), a regional launch, presentations at Indigenous and health conferences; and academic publications. Evaluation reports are freely accessible online via university repository, and the Community Report is also available on the Ngaramura website. Workshops were delivered by the Ngaramura staff and/or key Elders along with academic evaluators at national and international conferences (Bessarab et al., 2023; Clapham et al., 2022c; Harwood et al., 2024; Sheppeard et al., 2022).

Discussion

It is significant that Ngaramura, operating from a small Aboriginal Community Controlled Organisation in the southern part of the Illawarra region of NSW, has been able to sustain a flexibly delivered program that addresses a wide range of educational needs. The evaluation findings demonstrate that Ngaramura contributes to the much-needed response to the overrepresentation of Aboriginal young people in school suspension statistics. Currently, in NSW schools, Aboriginal young people make up 8% of all student enrolments but represent 25% of all short and long student suspensions (Sullivan et al., 2020). Ngaramura was able to successfully re-engage young people in education (see Logic Model MT1). This was evidenced in the young people’s attendance at Ngaramura, and for many, later engagement in school or training. The educational relationship with Ngaramura is of key significance, creating a connection to education that is supportive of opportunities to engage in further learning.

Ngaramura ensures that the young people were able to engage in an educational service that suits their needs. They young people described a sense of being able to be themselves at Ngaramura and of not being judged in the ways they experience at school. These experiences were closely linked to how the young people engaged with and related to the Ngaramura staff. These positive experiences with Ngaramura staff supported the young people to engage with learning and the range of educational activities Ngaramura. This engagement in learning at Ngaramura contrasted sharply with the young people’s experiences at school

The Aboriginal-led evaluation helped with recognising the impacts that experiences of racism have had on the Aboriginal young people, and the work that Ngaramura does to support the young people. We note that the National Agreement on Closing the Gap (Coalition of Peaks, 2020) states that data is needed for development on “student experiences of racism”. This is significant data to collect to improve understanding of the experiences of Aboriginal young students in schools. Our approach to evaluation supported our access to this important data, and this points to how this type of approach to evaluation may be of benefit to developing more data on Aboriginal students’ experiences of racism in schools.

The Ngaramura evaluation demonstrates that Coomaditchie excelled in their program delivery and achieved key outcomes. Ngaramura successfully established a culturally based model that supported the students and provided a service that can help the school environment to improve. We argue that attention needs to be placed on what we are terming the ‘social and cultural determinants of education’; both in terms of understanding the impacts of these determinants, and for informing responses and strengths-based approaches. There is a growing recognition of the failures of a ‘deficit’ thinking and practice in education, as evidenced in the 2020 Closing the Gap Report (Australian Government, 2020). Coomaditchie as an organisation has been very successful in opening a door for schools to have a genuine Aboriginal community connection. The evaluation provided exemplars and language to push back against the deficit discourse. Interviews with teachers show that teachers who engage with Ngaramura strongly support the Program; they report attitudinal change and begin to understand why Aboriginal young people are not comfortable going to school, and how structures and systems within schools contribute to the large number of Aboriginal suspensions. The evaluation found that Ngaramura is a highly successful example of an Aboriginal-led cross-cultural collaboration with mainstream schooling that demonstrates the benefit of Aboriginal-led specialist sites of learning/practice which can be connected with mainstream schooling to improve the outcomes of Aboriginal young people. Ngaramura should be recognized as a preventative measure, which addresses the needs of vulnerable young people at a critical time in their lives.

Importantly, the Aboriginal-led collaborative evaluation design supported our fieldwork to learn of the cultural work of the program, and the role of the Elders in this work. Elders as educators of cultural knowledge and understanding for Aboriginal young people is a respected and long held educational practice. For instance, Scrine and colleagues (2020) explain the role of Elders in the education of children,

Elders occupy an important position in Aboriginal communities. They are recognised and respected as authority figures who hold Wisdom, cultural knowledge and stories that link the present to the past. …Whether as direct family members or members of the wider community, an accepted part of Aboriginal childrearing was the role of Elders in guiding Aboriginal children. (Scrine et al., 2020)

Significantly, these authors emphasise that “A critical element of children’s cultural knowledge was hearing stories from Elders about their lives and their experiences” (p. 42) and that the participants in their study “talked about the importance of the education children receive from Elders and other family members” (p. 44).

One of the features of the evaluation was access afforded to the evaluators to meetings and discussions with the government representatives responsible for funding management. We observed how Coomaditchie built a strong relationship with the government funding agency. The relational place-based design to commissioning, and the recognition from the local staff of the NIAA that Aboriginal programs require time to become established and build capabilities, were positive factors enabling Ngaramura to develop and deliver a successful and innovative program. Considerable organisational learning has occurred. The program has benefited from being able to incorporate feedback from the evaluation. Relationships with local schools have been strengthened and staff capabilities greatly increased. However, the lack of secure, ongoing funding remains a barrier to the ongoing sustainability of Ngaramura. For example, even existing staff positions cannot be guaranteed into the future, making it difficult for the program to employ and train more staff and for the program to flourish and expand, thus benefitting more disadvantaged young people and schools.

According to Canadian Aboriginal academic, Marie Battiste, “Indigenous pedagogy values a person’s ability to learn independently by observing, listening, participating with a minimum of intervention and instruction” (Battiste, 2002, n. p.). The Ngaramura evaluation showed the positive impacts that a small, place-based, and Aboriginal community-led program can offer to Aboriginal young people who are having trouble with mainstream schools. Culturally based education programs offer nuanced perspectives of educational inequality and educational disadvantage as it is experienced by Aboriginal people. Our evaluation of Ngaramura shows the importance of such culturally based programs for Aboriginal young people, underscoring the importance of programs such as Ngaramura.

Conclusion

The evaluation of Ngaramura highlights the importance of a culturally safe, Indigenous led, collaborative evaluation design. This has significant implications for evaluation practice with Indigenous communities and organisations but there are lessons to be learnt for evaluation practice more broadly. The purpose of the evaluation was to provide Ngaramura with knowledge which was locally relevant and useful for the organisation and could support the sustainability and improvement of the program over time. Our Indigenous evaluation was able to pick up some of the nuances of program implementation and achievement; nuances often missed in traditional evaluations. These important underpinnings include the importance context, place and culture have in understanding how the program works, and why its achievements are so significant for the participants, organisation, and the community. The evaluation team needed to be flexible and responsive to the needs of the program, the participants, and their families. Ethical and responsible research demands that participant voices be included so that they can be heard. This Aboriginal-led collaborative evaluation supported inclusion of all voices, including the lived experience of one family member because it gave an insight into the issues, that were reflected in the experience of other families whose children were in the program, and because the participant wanted their voice to be heard.

In 2025 Ngaramura continues to experience funding uncertainty. Despite its successes and the program now running since 2018, funding is not guaranteed from year to year. This stands in stark contrast to mainstream education services, which do not operate with anywhere near the high level of uncertainty experienced by Ngaramura (and arguably many other Indigenous programs). We argue that small, place-based Indigenous education programs such as Ngaramura should be recognised as deserving of funding security in the same way that the mainstream schooling sector expects support from year to year.

The Ngaramura evaluation addresses the recognized lack of evaluation of Indigenous community programs, and shines a light on the value of small, underfunded programs targeting complex issues; Aboriginal programs that are working effectively and demonstrating impact on very small budgets. Strengths-based community-led solutions to the issues facing Indigenous families and communities such as Ngaramura urgently need recognition and investment by government at all levels.