A scoping review of commissioning practices used in the evaluation of Indigenous health and wellbeing programs: Protocol article

Abstract

Despite the billions of dollars invested in improving Indigenous health and wellbeing outcomes in Australia, there is little evidence of program effectiveness to inform policy and practice. The deficiency of evaluations is problematic. Critical to this process is the effective engagement of commissioners with Indigenous peoples, which is not well documented. Currently, there is scant evidence on modes of commissioning practices used. This scoping review will aim to identify the spectrum of commissioning practices used when evaluating Indigenous health and wellbeing programs in Australia, codifying them into a model set. Documents (between 2008 and 2020) will be retrieved from Scopus, Proquest, Informit, Google Scholar and via a web-based search that refers to the commissioning of Indigenous health and wellbeing program evaluations in Australia, New Zealand, Canada or the United States. Importantly, the research team is Indigenous-led and the project’s governance, quality and translation framework will be informed by a project advisory group, including Indigenous associates. This will be the first scoping review globally to identify practices used to commission Indigenous health and wellbeing program evaluations. Results will be utilised to strengthen the commissioning practices of Indigenous health and wellbeing programs in Australia and overseas.

Keywords

Aboriginal commissioning evaluation health Indigenous wellbeing

What we already know:

• Calls for evidence-based Indigenous health and wellbeing policy development have increased over the last few decades.

• One way of generating evidence to inform this policy and practice is through funded program evaluation.

The original contribution the article makes to theory and/or practice:

• This review considers what little is known about the commissioning processes for Indigenous health and wellbeing evaluations.

• To date there are no developed models for commissioning Indigenous health and wellbeing programs. This will be the first step in their development to assist commissioners, evaluators or services providers when undertaking commissioning.

Introduction

The colonised experience of Aboriginal and Torres Strait Islander peoples, the Indigenous people of the country now called Australia, is often problematised via the presence of inequitable health outcomes, including complexities associated with morbidity and premature mortality (National Indigenous Australians Agency, 2023). The Aboriginal and Torres Strait Islander peoples’ experience is typically accompanied by a narrative which argues that substantial funds have been ‘invested in’ improving health with mixed outcomes or benefits to the wider society (Steering Committee for the Review of Government Service Provision, 2017).

Over several decades, calls for evidence-based health and wellbeing policy development have increased, particularly for policies that impact Indigenous peoples (Coalition of Peaks, 2023; Dwyer, 2004; Fredericks, Adams, & Edwards, 2011; Head, 2010; Larkin, 2006; Lowitja Institute, 2022; Productivity Commission, 2021). These have principally been to determine whether these policies attain their stated goals and, if there have been unintentional consequences, to determine to what extent these by-products have been positive or negative (Oliver et al., 2019). One way to generate evidence to inform policy and practice is for program commissioners and service providers to evaluate funded programs (Bowman et al., 2015; Broc et al., 2019; James, 2013; Maddox et al., 2021; Weaver & Cousins, 2007). In the case of complex evaluations in political contexts, it is suggested this should be with iterative, flexible approaches that consult with, and respond to, key stakeholders throughout the evaluation process so as to benefit the quality of findings (Stack et al., 2018). This includes evaluations of Indigenous health and wellbeing programs.

The Need to Consider the Complexity Surrounding Indigenous Health and Wellbeing Program Evaluations From Their Inauguration

Indigenous health and wellbeing program evaluations are inherently ‘situated within complex systems and influenced by their social, political, historical, and cultural contexts’ (Maddox et al., 2021, p. 333), which means their effectiveness rests on their sensitivity to cultural circumstances (Chouinard & Cram, 2019; Gollan & Stacey, 2021). Considering associated Indigenous contexts from the point of commissioning to evaluation conclusion facilitates the incorporation of local circumstances, standpoints and knowledges within the evaluation from the initiation of evaluation, making it more likely that evaluation outcomes consider the ideals of local Indigenous peoples and their communities (LaFrance & Nichols, 2008; Maddox et al., 2021). This is especially important as evaluations commissioned without considering community contexts may be culturally unsafe or inappropriate, as they can enact dominant Euro-Celtic standpoints and practices, further subjugating Indigenous practices and knowledges (Gollan & Stacey, 2021; Maddox et al., 2021; Rigney, 1999). Finally, this lack of consideration may ultimately result in reduced community participation, a lack of methodology and reduced data reliability and validity (Maddox et al., 2021). These deficiencies in turn, have been suggested to inevitably lead to a failure to truly evaluate program and policy achievements (Williams, 2018).

Commissioning in Practice

In Australia, Canada, New Zealand and the USA, substantial investments from national and state/provincial/regional governments have been made in Indigenous health and wellbeing programs, which are ostensibly aimed at achieving health parity between Indigenous peoples and other people in their respective jurisdictions (Allard-Côté et al., 2016; Council of Australian Governments (COAG), 2020; Government of Canada, 2021; Productivity Commission, 2020b; Steering Committee for the Review of Government Service Provision, 2017). Yet only a small number of Aboriginal and Torres Strait Islander programs are evaluated, and fewer still from Indigenous peoples’ perspectives (Fredericks et al., 2017; Hudson, 2017; Williams, 2018). Within this context, Indigenous leaders have been calling for opportunities to influence decision-making processes within the health system and across sectors and for the commissioning of health and wellbeing programs to reflect their needs, priorities and views on program design, delivery and evaluation (Aboriginal & Torres Strait Islander Social Justice Commissioner, 2009; Gibbs, 2020; Lowitja Institute, 2020; Malezer, 2013; National Aboriginal Community Controlled Health Organisation (NACCHO), 2022; Reconciliation Australia, 2016). This call necessitates better designed and evaluated programs to benefit Indigenous populations and achieve the Closing the Gap (CTG) targets. This is supported by recommendations from the Productivity Commission for government and non-government agencies to partner with Indigenous peoples to ensure that programs and their evaluation capture Indigenous interests (Productivity Commission, 2012). It also aligns with Articles 18–21 and 23 in the United Nations Declaration on the Rights of Indigenous Peoples (United Nations, 2009), which outline the right of Indigenous peoples to act as full and effective participants in the decision-making processes that affect their health, economic and social wellbeing. However, recent scholarship points to the dichotomous relationship between what funders want and what Indigenous communities need when evaluations are being commissioned (Finlay et al., 2021). Ultimately if robust evidence is generated from this project, that meets the needs of stakeholders, it may then be applied to further improve policy and practice.

The commissioning of program evaluations tends to be conceptualised as a series of activities or steps (Finlay et al., 2021). Typically, the process begins when a funding body initiates an evaluation and ends with the final reporting of the evaluation findings. From the moment the evaluation is initiated until its completion, the commissioner of evaluation is central to the way program evaluations are carried out and the quality of evidence that is generated (Finlay et al., 2021). The commissioning process often sets the budget, selects the evaluators, and places parameters around the evaluation aims, objectives and data collection methods. These pre-determined variables, set the stage for the cultural safety and power dynamics of the evaluation (Finlay et al., 2021), or lack thereof for participating individuals and communities. This process can be particularly problematic with Indigenous evaluations, which are typically considered a threat to Indigenous peoples. This is firstly due to the historical role of research in the colonisation of Aboriginal nations (Andersen & O'Brien, 2017; Fredericks, 2008; Katz et al., 2016; Moreton-Robinson, 2021; Rigney, 1999; Walter & Andersen, 2016) and secondly, because evaluative research methodologies can be grounded in ‘world views’ which are hostile towards non-Eurocentric belief systems or do not have methods designed around other ways of knowing (Martin & Mirraboopa, 2003; Moreton-Robinson, 2021; Rigney, 1999; Smith, 2021). However, this is changing as evaluation gradually comes to be viewed as a resource that can be utilised for self-determination, advocacy and cultural-linguistic maintenance (Blair, 2015; Katz et al., 2016; Kovach, 2021; Wilson, 2008) and may further change through decolonisation.

The Decolonisation of Evaluation Commissioning

Decolonising denotes the dismantling of the legacy of colonisation and coloniality (Auriacombe & Cloete, 2019; Laenui, 2000; Sherwood, 2009), where coloniality is defined as the established (matrixes or) ‘patterns of power’ (Maldonado-Torres, 2007, p. 243), which are the result of colonialism (Mignolo & Walsh, 2018). These have come to define ‘culture, labour, intersubjective relations, and knowledge production well beyond the strict limits of colonial administrations’ (Maldonado-Torres, 2007, p. 243), so that they colour all aspects on individuals’ lives (Mignolo & Walsh, 2018). Decolonisation does not infer the ‘absence of coloniality’ (Mignolo & Walsh, 2018, p. 81), rather it entails active, continuous, but sometimes indirect, movement towards the ‘possibility of other modes of being, thinking, knowing, sensing and living; that is, an otherwise in plural’ (Mignolo & Walsh, 2018, p. 81; Muller, 2023). It is additionally ‘a process that requires the positioning of oneself in history and the recognition of ideas and assumptions that have informed one’s worldview’ (Sherwood et al., 2011, p. 194). In essence, it doesn’t just change the content of the discussion, as de-westernisation does, it changes its terms or provisions (Mignolo & Walsh, 2018) and our selves.

In the context of evaluations or research praxis, this has been suggested to be actions guided by transparent consultation, principles (i.e. researcher responsibility), confidentiality, respect for and recognition of rights, consent, safety, collaboration, reflection and mutual benefits (Brant et al., 2023; Fredericks et al., 2011; Herman, 2018; Muller, 2007; Sherwood, 2010; Sherwood & Edwards, 2006; Smith, 2008; Tunón et al., 2016). Evaluative research with the consent of the community and individuals who participate aligns with Indigenous epistemologies and the political realities of Indigenous lives. This approach also respects social and communal knowledge as well as individual knowledge, making it consistent (Brant et al., 2023; Cajete, 2017; Chilisa, 2012; Rigney, 1999). Put simply, it is the evaluations that honour the principles of conducting research with and for Indigenous peoples and which move away from the harmful and unethical practices of the past, where the word ‘research’ became ‘one of the dirtiest words in the Indigenous world’s vocabulary’ (Smith, 2021, p. 1). It is changing the way of thinking to ‘give First Nation people a voice’ (Radcliffe in Gollan & Stacey, 2021, p. 8), through meaningful engagement, reciprocity, the recognition of diversity and the incorporation and utilisation of Indigenous knowledge systems (Andersen & O’Brien, 2017; Bower et al., 2015; Moreton-Robinson, 2021).

It is well documented that the success of evaluations is determined by Indigenous peoples’ engagement throughout all stages of the evaluation, from identifying the need for an evaluation to the final reporting (Auditor-General, 2019; Better, 2022; Bowman et al., 2015; Productivity Commission, 2020). This is also evidenced in the numerous human research ethics processes and other documents that describe the best practice principles for evaluating Indigenous health and wellbeing programs (Centre for Social Research and Evaluation: Te Pokapü Rangahau Arotake Hapori, 2004; Williams, 2018). For example, in New Zealand, evaluations involving Māori should be conducted using the principles outlined in ‘Nga Ara Tohutohu Rangahau Māori Guidelines for Research and Evaluation with Māori’ (Centre for Social Research and Evaluation: Te Pokapü Rangahau Arotake Hapori, 2004). In Australia, evaluations are guided by two sets of research guidelines, inclusive of evaluation: the National Health and Medical Research Council’s (NHMRC) (2018) Ethical conduct in research with Aboriginal and Torres Strait Islander Peoples and communities: Guidelines for researchers and stakeholders 2018 and the Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS) (2020) AIATSIS Code of Ethics for Aboriginal and Torres Strait Islander Research. More recently, we have seen the development of Indigenous and cultural evaluation frameworks by other organisations in Australia – such as the National Indigenous Australians Agency (Department of the Prime Minister and Cabinet, 2018), Productivity Commission (Productivity Commission, 2020b), Lowitja Institute (Kelaher et al., 2018) and the Australian Evaluation Society (Gollan & Stacey, 2021) – all of which offer additional principles and further guidance about pursuing Indigenous evaluations.

What is unclear is how and to what extent best practice principles, including effective Indigenous engagement, are being considered or supported in the commissioning of program evaluations, if at all, as there is currently no research on this topic. This gap in knowledge makes it difficult for commissioners to apply best practices in procuring evaluation tenders for Indigenous health and wellbeing programs. It also impedes the development of an evidence base to inform policy decisions regarding strategic financial investments to improve Indigenous health outcomes. With better-designed programs and evaluations, and improved commissioning practices, there is an increased likelihood of achieving improved Indigenous health outcomes.

The first step in understanding the commissioning practices of Indigenous health and wellbeing evaluations is to undertake a scoping review of the literature from Australia, New Zealand, USA and Canada. This protocol outlines the scoping review methodology that will be used to fill in this important knowledge gap.

Objectives

The objectives of the scoping review are:

1. To systematically search the literature to identify government and non-government agencies commissioning practices of Indigenous health and wellbeing program evaluations, and characterise commissioning practices according to Indigenous best practice principles,

2. To review commissioning practices and construct models that represent the range of commissioning practices using Indigenous best practice principles and

3. To identify knowledge gaps in the commissioning of Indigenous health program evaluations.

Context

This scoping review was initiated to identify existing commissioning models of evaluation commissioning as part of a broader project considering the implications of modes of evaluation commissioning of Aboriginal and Torres Strait Islander Health and Wellbeing Programs. A team of eight researchers led by Yorta Yorta woman Dr Summer May Finlay will conduct this scoping review with guidance from a stakeholder project advisory group (which has strong Indigenous representation from across government and the Aboriginal community-controlled sectors), in collaboration with the research team of which over 50% identify as Indigenous. Foley’s (2003) Indigenous standpoint theory, along with other seminal Indigenous standpoint theory work by Moreton-Robinson (2021) and Nakata (2007), will be used as the lens to assist with interpreting the results. This is because central to this review and all aspects of the broader research project is the adoption of an Indigenous standpoint.

Indigenous standpoint theory recognises that historical knowledge remains prejudiced by Eurocentric or colonial perspectives, which restrict Indigenous voices, therefore, it opposingly privileges Indigenous voices and subjectivities (Henry & Foley, 2018; Hokowhitu et al., 2020; Moreton-Robinson, 2021; Nakata, 2007; Rowe et al., 2015; Walker et al., 2013). It ‘is research by, with and for Indigenous peoples, not just “about” Indigenous peoples’ (Henry & Foley, 2018, pp. 221–222), which means it is built on reciprocal respect and understanding of culture and identity (Bessarab & Ng’Andu, 2010; Bishop, 2008; Bower et al., 2015; Rogers et al., 2017). It utilises Indigenous ontology, or beliefs about existence and reality, an epistemology founded on an Indigenous perspective of what counts as knowledge and axiology grounded in those cultural values and a methodology influenced by these three factors (Denzin et al., 2008; Durie, 2004; Foley, 2018; Henry & Foley, 2018; Moreton-Robinson, 2021; Nakata, 2007; Rigney, 1999). With this in mind, throughout the production of this scoping review, Indigenous researchers and the project advisory group will play key roles in ensuring reciprocal appreciation for, and understanding of, Indigenous culture and identity is paramount in the way this review is presented, in the research questions asked and in interpretations of results (Chilisa, 2012; Fredericks, 2010; Hokowhitu et al., 2020; Walter & Andersen, 2016).

This article reports on the methods used to conduct this review.

Methods

Funding Agency

This study was funded by the National Health and Medical Research Council of Australia (GNT1165913), with the aim of strengthening the commissioning practices of Indigenous health and wellbeing programs in Australia.

Study Design

A scoping review will be conducted following the methodological framework of Arksey and O’Malley (2005) and Levac et al.’s (2010) methodological enhancements. Distinct from systematic literature reviews which assess the quality of studies (Arksey & O'Malley, 2005; Levac et al., 2010), scoping reviews clarify key concepts in the literature and the extent to which they are reported, identify key characteristics or factors related to a concept and uncover knowledge gaps to inform future research (Munn et al., 2018; Peters et al., 2020). As to date, there has been no formal synthesis of commissioning practices or models applied to Indigenous health program evaluation; researchers feel this study best aligns with the aim of a scoping review to clarify concepts (such as commissioning models) in the literature.

Four academic databases will be searched, however, given the focus of this study, grey literature will also be included (i.e. print and electronic documents produced by government, academic and industry sources that are not controlled by commercial publishers). This may include evaluation studies published by consultants as reports or policy papers on the websites of government departments (Lawrence et al., 2014). The websites of commissioning organisations will also be searched from a starter list of organisations identified by the research team, project advisory group and their respective professional networks. We expect to find tender documents, call for proposals, principles and frameworks and gain a sense of timelines and conditions in the funding agreements and where and how Indigenous peoples are engaged in the commissioning process. Research organisations and their electronic databases that publish journal articles, research reports, evaluation reports and policy directives relevant to the topic will also be reviewed (e.g. Australian Indigenous Health Info net).

The scoping review framework has six stages: (1) identifying the research question, (2) identifying relevant studies, (3) selecting studies, (4) charting the data, (5) collating, summarising and reporting the results and (6) consultation. The Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) extension for scoping reviews and the PRISMA flow diagram will be used to guide the conduct and reporting of the scoping review (Tricco et al., 2018). This review is the first of its kind and will identify knowledge gaps in commissioning practices of Indigenous health and wellbeing program evaluations, with the aim of informing future health policy and practice.

Stage 1: Identifying the Research Question

This scoping review will be the first step to characterising the commissioning practices of Indigenous health and wellbeing program evaluations. It seeks to answer the following question: What practices are used by government and non-government agencies to commission Indigenous health and wellbeing program evaluations and how are they aligned with Indigenous best practice principles?

Stage 2: Identifying Relevant Studies

The following databases will be systematically searched: Scopus, Proquest, Informit and Google Scholar. Grey literature will be retrieved by searching the world-wide web via the Google search engine. Following Pham (2014), we aim to screen the first 100 hits from the Google search as further screening is deemed unlikely to yield many more relevant documents (Stevinson & Lawlor, 2004). We anticipate the search will lead to reports, policy submissions and potentially websites with links to commissioning documents. Supplementary documents, including websites to relevant commissioning documents, will be obtained by discussing the scoping review search strategies with the project advisory group at scheduled advisory meetings and then following up with individual advisory group members. Research team members will utilise their professional networks to obtain reports, policy submissions and websites that may not surface in the database searches. Finally, we will circulate a list of documents to advisory group members and colleagues to see whether any grey literature documents are missing.

Specific search strategies will be designed for each database. Separate search strings will be designed to capture terms related to: ‘Indigenous’, ‘evaluation’, ‘program’, ‘commissioning’ and ‘health’ related documents. Following a period of trialling the search, with advice from a research librarian, an optimal set of search strings will be generated. We anticipate that the search strategy will be iteratively refined. Documents will be downloaded, imported and stored in Covidence, the web-based software platform.

Stage 3: Study Selection

In the first step of stage 3, using Covidence, duplicate documents will be removed and full-text documents retained for the second step of screening. Subsequently, all titles and abstracts will be screened independently by two reviewers, based on the eligibility criteria below. Where no abstracts exist, such as is the case for some grey literature, full-text scanning will occur. Any discrepancies between the reviewers will be resolved through discussion and, if needed, moderated by a third reviewer. In the second step, documents meeting the eligibility criteria will be included in the full-text review. Both reviewers will independently assess the full-text documents, with disagreements resolved through discussion and moderation by a third reviewer, if needed. As per Levac et al.'s (2010) enhancement, the two reviewers (initials blinded) will meet weekly at the beginning, midpoint and final stages of the study selection process to discuss any challenges that arise in the study selection process. We are open to the study selection process occurring iteratively with the literature search and initial review of documents used to refine the search strategy.

Eligibility Criteria

Documents meeting the following criteria will be included in the review: English-based studies from 2008 to 2020 that refer to the commissioning practices of Indigenous evaluations of health or wellbeing programs for Indigenous populations in Australia, New Zealand, Canada or the USA. In this manner, the search will identify practices that can inform Australia’s ‘Closing the Gap’ health policy, initiated in 2008 (Australian Government, 2008), a focus of the broader funded research grant. Limiting the search to August 2020 coincides with the release of Australia’s Indigenous Evaluation Strategy (Productivity Commission, 2020a), which provides a whole-of-government framework for selecting, planning, conducting and using evaluations of policies and programs affecting Indigenous peoples in Australia. The search was limited to 2020, with the knowledge that very recent advancements associated with the development of Australia’s Indigenous Evaluation Strategy and National Indigenous Australians Agency (Department of the Prime Minister and Cabinet, 2018) Indigenous Evaluation Framework may artificially skew the review findings. Another review will be conducted at a later date to capture these developments. The search will be restricted to Indigenous populations in developed countries to capture commissioning practices within countries with similar histories of colonisation and subsequent policies impacting Indigenous peoples in Australia.

Documents will need to refer to one or more aspects of the commissioning of an evaluation, that is, refer to the terms of reference for an evaluation, process/criteria for selecting an evaluator, contractual obligations, how variations to an evaluation protocol are managed, how the commissioned evaluation was conducted in relation to commissioner expectations/contractual arrangements, or express a viewpoint about the commissioning practices of Indigenous health and wellbeing evaluations. Health and wellbeing programs in this scoping review will be restricted to programs with a focus on health promotion/social determinants of health and a non-clinical focus.

Stage 4: Charting the Data

Best practice principles relevant to the research and evaluation with Indigenous populations will be extracted from included documents (see Table 1). An initial list of 16 coding categories derived from documents which included best practice principles such as Chandna et al. (2019), Cargo et al. (2019) and Kelaher et al. (2018) was adapted, trialled on four documents and reduced to the 13 coding categories shown in Table 1. This was a complex process involving repeated review of the abovementioned documents and iterative discussion and debate between lead and chief investigators and the project advisory group to inform the refinement of the coding categories.

Table 1.

Data extraction tool that includes definitions of best practice principles and extraction guidance.

Principle	Definition	What to look for
1. Indigenous engagement	Commissioned evaluation is designed and delivered so Indigenous communities are engaged prior to the start of the evaluation, during the evaluation, or after data collection (i.e. during data analysis/interpretation, or during knowledge transfer/dissemination).	References to how Indigenous peoples, communities or organisations were or can be engaged in the commissioning of evaluation or in different phases of the evaluation.
2. Partnership	The commissioning of the evaluation and delivery is done in a way that builds and maintains partnerships with Indigenous communities and organisations during the commissioning and evaluation process.	References to efforts made by commissioners and evaluators to connect with the community to work together so the evaluation can move forward in a good way.
3. Self-determination	The commissioned evaluation is guided by Indigenous self-determination, that is, led and defined by Indigenous peoples. Respect Indigenous peoples’ right to self-determination.	Reference to how Indigenous peoples were/can be self-determined in the commissioning and delivery of evaluations. Refers to governance structures that support Indigenous decision-making and ownership of the evaluation.
4. Accountability	The commissioning of an evaluation and its delivery demonstrates responsibility to the community that the evaluation meets their needs, benefits community and in a culturally safe way.	Reference to how a commissioned evaluation demonstrates that it meets community needs and benefits community in a culturally safe way.
5. Cultural capability	All stakeholders in a commissioned evaluation have the cultural knowledge, skills, understanding and sensitivity to undertake the work.	References to whether the commissioners took into consideration whether the evaluators have the capacity to undertake the evaluation, and this capacity is formally assessed in the commissioning process. References to commissioners and evaluators having the cultural capability to commission and conduct the evaluation.
6. Culturally responsive	All stakeholders in a commissioned evaluation understand and respond to Indigenous cultural protocols.	References to whether commissioners and evaluators understand and respond to Indigenous cultural protocols.
7. Respect	All stakeholders in a commissioned evaluation demonstrate respect for participants. Respect for diversity amongst Indigenous peoples and their respective cultural protocols.	References to how the commissioning of evaluations were/can be designed to demonstrate respect for participants.
8. Community context	All stakeholders in a commissioned evaluation are designed and delivered with the Indigenous community, are specific to the community’s context and are relevant and responsive to community objectives and needs.	References about how community objectives and needs were/can be built into the commissioning of an evaluation. References to how the commissioning of evaluations were/can be adapted to the community context.
9. Trust/relationship building	All stakeholders in a commissioned evaluation take the time to develop a relationship with the community and delivering what you have agreed to deliver in a culturally safe way. Advocating when required on behalf of the community.	References to trust and relationships and how they are developed with Indigenous peoples by the commissioners and evaluators.
10. Community benefit	The benefits of commissioned evaluations are greater than its risks, that is, the evaluations are beneficial for the participating community. Evaluation projects must be relevant and findings used in a way that meet Indigenous community needs and priorities.	Reference to how the benefits of the commissioning process and implementation of evaluation outweigh the risks.
11. Hospitality/generosity	Evaluation projects must factor in time and resource to foster mutually beneficial relationships, where the views of all stakeholders are respected. A community’s commitment to support evaluation (through time/resources/knowledge) should be acknowledged appropriately by commissioners/evaluators.	References the basis of all well-intentioned human interaction, to nurturing and fostering relationships and treating the other with care and respect. Also references a responsibility to look after people.
12. Capacity building	Commissioned evaluations help to facilitate local capacity building.	Reference about how local capacity were/can be built into the commissioning of evaluations.
13. Two-way learning	All stakeholders in a commissioned evaluation learn from each other’s skills and knowledges.	References to how commissioners/evaluators and Indigenous peoples learn from each other’s skills and knowledges.

Throughout the review, each primary document will be coded for the presence, or absence, of these principles using NVivo qualitative software. MS Excel will be used to capture the following descriptive characteristics from primary documents: country, document type, purpose of document and whether it is empirical or not. The extraction of data from primary documents will be Indigenous-led with support from a second research team member. Primary document authors will not be contacted to provide additional information. At this stage, we do not anticipate undertaking a critical appraisal of the primary documents as we anticipate the documents will be policy and practice-based.

Stage 5: Collating, Summarising and Reporting the Results

Document characteristics will be extracted and analysed using frequencies, with each primary document coded according to the 13 principles. Given the interdependence of the best practice principles (e.g. self-determination and Indigenous participation), we anticipate using simultaneous or co-occurrence coding (Saldana, 2012). This qualitative coding procedure is appropriate when a segment of text suggests multiple meanings that justify the use of more than one code. Simultaneous coding will also allow for the more complex social interactions that would be expected between commissioners, evaluators and Indigenous community members or service providers in the evaluation of Indigenous health and wellbeing programs. The simultaneous coding of documents using principles, with consideration to their presence or absence, may facilitate the identification of commissioning models – the anticipated output of the synthesis process. Notes for each primary document will be summarised in memos and contribute to the two data analysts building consensus on the development of the commissioning models, through a synthesis of the included documents. We anticipate that commissioning models identification will be an iterative process between the two researchers with ongoing feedback from project stakeholders used to refine the commissioning models. The commissioning models will be visually represented according to the best practice principles.

Stage 6: Consultation Exercise

As noted earlier, the scoping review will be guided by the project advisory group and research team (‘project stakeholders’) and underpinned by Indigenous research methodologies. The project advisory group is comprised of Indigenous and non-Indigenous national and state representatives from Australian government and non-government agencies/associations. The members have broad based experience in Indigenous health policy/health policy, the commissioning/procurement/contracting of Indigenous program evaluations and Aboriginal Community Controlled Health Services. The Indigenous-led research team is comprised of Indigenous and non-Indigenous experts in health promotion/population health, education, health program planning and evaluation, Indigenous health policy and Aboriginal community-controlled health services. Together, the project advisory group and research team have extensive experience in the commissioning of Indigenous program evaluations by government and non-government agencies. The commissioning practices will be coded using the best practice principles into draft models. The models, as they are developed, will be regularly presented to both groups to check whether any models are missing, whether dimensions from models are absent and to check whether the models are consistent with members’ experiences with the commissioning of Indigenous health program evaluations.

Patients and Public Involvement

Patients and the public were not involved in the development of this scoping review protocol.

Discussion

Implications

The findings of this scoping review will be used to strengthen the commissioning practices of Indigenous health and wellbeing programs by government and non-government agencies in Australia and may also have broader applicability to other countries with Indigenous populations The researchers believe that the identification of the scope and nature of evaluation commissioning models and the ways they are understood to operate in practice will aid in the further development of commissioning Indigenous health and wellbeing evaluations.

The scoping review will also contribute to stage two of an Australian-based NHMRC project, where they will be used as a coding framework to further consider the ways in which the commissioning of Indigenous health and wellbeing program evaluations occurs in practice in Australia. More specifically, the commissioning models identified from this scoping review will be elaborated on through interviews with both commissioners of evaluation, evaluators and Indigenous health program providers in Australia, and in-depth case studies of completed commissioned evaluations of Indigenous health and wellbeing programs.

Ethics and Dissemination

Since this scoping review will only review publicly available materials, ethics approval from a Human Research Ethics Committee (HREC) is not required. The scoping review will however adhere to the National Health and Medical Research Council National Statement (National Health & Medical Research Council, 2007 (Updated 2018)) and the Ethical conduct in research with Aboriginal and Torres Strait Islander peoples and communities: Guidelines for researchers and stakeholders 2018 (National Health and Medical Research Council, 2018). A policy brief will be disseminated to project stakeholders, government and non-government commissioning agencies (e.g. the National Indigenous Agency of Australia) and relevant professional associations (e.g. the Australian Evaluation Society Cultural and Diversity Committee). The research team will work closely with project advisory group members to find optimal media to translate the review findings to influence and alter commissioning practices for the benefit of Indigenous communities. This effort may include face-to-face or online workshops with commissioners and evaluators. Scoping review results additionally will be disseminated through a peer reviewed publication and conferences/seminars.

Strengths and Limitations

A major strength of this scoping review is (and will continue to be) that from its induction Aboriginal researchers (and members of the project advisory group) have led and contribute(d) to its design and production. Resultingly this scoping review will reflect a broad range of Indigenous and non-Indigenous perspectives and evaluation and commissioning experience. The latter aside, importantly this research uses ‘evaluation principles derived from [a variety] Indigenous contexts’ (Maddox et al., 2021, p. 345).

Another strength will be its selection of databases. Given the risk associated with methodologies that limit themselves to academic literature and thereby potentially exclude literature regarding evaluation commissioning that may not have been published due to publication barriers (McGrail et al., 2006), this scoping review will include a web-based search as one method of identifying grey literature. Nonetheless, in limiting our research to include evaluations within the health and wellbeing context, we risk missing relevant literature as there may also be examples of good evaluation commissioning practice outside this field. A further limitation will be that only publicly available documents are identified. There may be government or non-government reports that include commissioning practices which are not accessible to the research team.

Conclusion

In conclusion, following calls from Indigenous leaders and academics for more recognition in the direction and processes of evaluations from their inauguration, this scoping review will inform evaluation stakeholders and policy makers in two ways. First, to characterise the scope and nature of commissioning models of evaluation commissioning that are applied in practice, and second, determine how these align with Indigenous best practice principles. This will be done from an Indigenous standpoint, allowing for the inclusion and direction of other ways of knowing and being.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by National Health and Medical Research Council under grant GNT1165913.

ORCID iDs

Jenni Judd

Janaya Pender

References

Aboriginal, & Torres Strait Islander Social Justice Commissioner . (2009). Social justice report 2008 (1/2009). Aboriginal, & Torres Strait Islander Social Justice Commissioner. https://humanrights.gov.au/our-work/aboriginal-and-torres-strait-islander-social-justice/publications/social-justice-report-2

Allard-Côté

E. J.

Bradford

Durocher

(2016). Indigenous health policy in Australia, New Zealand, and Canada: A comparative overview. University of Ottawa. http://web5.uottawa.ca/www5/pendakur/aboriginal_issues_course/Indigenous_Health_Comparative_Overview_NZ_AUS_CAN.pdf

Andersen

O’Brien

J. M.

(2017). Sources and methods in Indigenous studies. Routledge. https://doi.org/10.4324/9781315528854

Arksey

O’Malley

(2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616

Auditor-General

(2019). Evaluating Aboriginal and Torres Strait Islander programs. Department of the Prime Minister and Cabinet. https://www.anao.gov.au/sites/default/files/Auditor-General_Report_2018-2019_47.PDF

Australian Government . (2008). Statement of intent. Paper presented at the Indigenous health equality summit: Close the gap. Australian Government.

Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS) . (2020). AIATSIS Code of ethics for Aboriginal and Torres Strait Islander research. Australian Institute of Aboriginal and Torres Strait Islander Studies.

Bessarab

Ng’Andu

(2010). Yarning about yarning as a legitimate method in Indigenous research. International Journal of Critical Indigenous Studies, 3(1), 37–50. https://doi.org/10.5204/ijcis.v3i1.57

Better

Evaluation.

(2022). Evaluation practice in Aboriginal and Torres Strait Islander settings. Better Evaluation. https://www.betterevaluation.org/en/themes/evaluation-practice-aboriginal-and-torres-strait-islander-settings

10.

Bishop

(2008). Freeing ourselves from neo- colonial domination in research: A Kaupapa Māori approach to creating knowledge. In Denzin

Lincoln

(Eds.), The landscape of qualitative research (3rd ed., pp. 109–135). Sage.

11.

Blair

(2015). Privileging Australian Indigenous knowledge: Sweet potatoes, spiders, waterlilys and brick walls. Common Ground Publishing.

12.

Bower

Malla

Manhire

Rogers

(2015). A cultural protocol for evaluation: A guide for the Indigenous Australia program team and external consultants to support and encourage good practice. Better Evaluation. https://www.betterevaluation.org/sites/default/files/Cultural_Framework_Eval_FINAL_WEB_21-1-16.pdf

13.

Bowman

N. R.

Francis

C. D.

Tyndall

(2015). Culturally responsive Indigenous evaluation: A practical approach for evaluating Indigenous projects in tribal reservation contexts. In Hood

Hopson

Frierson

(Eds.), Continuing the journey to reposition culture and cultural context in evaluation theory and practice (pp. 335–359). Information Age Publishing.

14.

Brant

Stagg Peterson

Friedrich

(2023). Partnership research with Indigenous communities: Fostering community engagement and relational accountability. Brock Education Journal, 32(1), 99–118. https://doi.org/10.26522/BROCKED.V32I1.962

15.

Broc

J.-S.

Oikonomou

Dragovic

(2019). Guidelines on how to integrate evaluation into the policy cycle, Evaluation into Practice to Achieve Targets for Energy Efficiency. European Commission. https://ec.europa.eu/research/participants/documents/downloadPublic?documentIds=080166e5c205eacf&appId=PPGMS

16.

Cajete

G. A.

(2017). Children, myth and storytelling: An Indigenous perspective. Global Studies of Childhood, 7(2), 113–130. https://doi.org/10.1177/2043610617703832

17.

Cargo

Potaka-Osborne

Cvitanovic

Warner

Clarke

Judd

Chakraborty

Boulton

(2019). Strategies to support culturally safe health and wellbeing evaluations in Indigenous settings in Australia and New Zealand: A concept mapping study. International Journal for Equity in Health, 18(1), 194. https://doi.org/10.1186/s12939-019-1094-z

18.

Centre for Social Research and Evaluation: Te Pokapü Rangahau Arotake Hapori . (2004). Nga Ara Tohutohu Rangahau Māori: Guidelines for research and evaluation with Māori. Centre for Social Research and Evaluation.

19.

Chandna

Vine

M. M.

Snelling

Harris

Smylie

Manson

(2019). Principles, approaches, and methods for evaluation in Indigenous contexts: A grey literature scoping review. Canadian Journal of Program Evaluation, 34(1), 21–47. https://doi.org/10.3138/cjpe.43050

20.

Chilisa

(2012). Indigenous research methodologies. Sage Publications.

21.

Chouinard

J. A.

Cram

(2019). Culturally responsive approaches to evaluation: Empirical implications for theory and practice. Sage Publications.

22.

Cloete

Auriacombe

(2019). Revisiting decoloniality for more effective research and evaluation. African Evaluation Journal, 7(1), 1–10. https://doi.org/10.4102/aej.v7i1.363

23.

Coalition of Peaks . (2023). Closing the Gap implementation plan 2023. Coalition of Peaks. https://www.coalitionofpeaks.org.au/implementation

24.

Council of Australian Governments (COAG) . (2020). National agreement on closing the gap. Council of Australian Governments (COAG). https://www.closingthegap.gov.au/national-agreement/national-agreement-closing-the-gap

25.

Denzin

N. K.

Lincoln

Y. S.

Smith

L. T.

(2008). Handbook of critical and Indigenous methodologies. Sage.

26.

Department of the Prime Minister and Cabinet . (2018). Indigenous advancement strategy evaluation framework. Department of the Prime Minister and Cabinet. https://www.niaa.gov.au/resource-centre/indigenous-affairs/indigenous-advancement-strategy-evaluation-framework

27.

Durie

(2004). Understanding health and illness: Research at the interface between science and Indigenous knowledge. International Journal of Epidemiology, 33(5), 1138–1143. https://doi.org/10.1093/ije/dyh250

28.

Dwyer

(2004). A critical review of evidence-based policy making. Australian Housing and Urban Research Institute. https://www.ahuri.edu.au/sites/default/files/migration/documents/AHURI_Final_Report_No58_A_critical_review_of_evidence_based_policy_making.pdf

29.

Finlay

S. M.

Cargo

Smith

J. A.

Judd

Boulton

Foley

Roe

Fredericks

(2021). The dichotomy of commissioning Indigenous health and wellbeing program evaluations: What the funder wants vs what the community needs. Health Promotion Journal of Australia, 32(2), 149–151. https://doi.org/10.1002/hpja.486

30.

Foley

(2003). Indigenous epistemology and Indigenous standpoint theory. Social Alternatives, 22(1), 44–52.

31.

Foley

(2018). Indigenous methodology: Is it invented or is it legitimate? Journal of Australian Indigenous Issues, 21(3), 20–38. https://search.informit.org/doi/10.3316/informit.142821743320555

32.

Fredericks

(2008). Researching with Aboriginal women as an Aboriginal woman researcher. Australian Feminist Studies, 23(55), 113–129. https://doi.org/10.1080/08164640701816272

33.

Fredericks

(2010). Reempowering ourselves: Australian Aboriginal women. Signs: Journal of Women in Culture and Society, 35(3), 546–550. https://doi.org/10.1086/648511

34.

Fredericks

Adams

Edwards

(2011). Aboriginal community control and decolonizing health policy: A yarn from Australia. In Lofgren

de Leeuw

Leahy

(Eds.), Democratizing health: Consumer groups in policy process (pp. 81–97). Edward Elgar Publishing.

35.

Fredericks

Adams

Finlay

Fletcher

Andy

Briggs

Hall

(2011). Engaging the practice of Indigenous yarning in action research. ALAR: Action Learning and Action Research Journal, 17(2), 12–24. https://doi.org/10.3316/informit.463112109856313

36.

Fredericks

Longbottom

McPhail-Bell

Worner

Board

(2017). “Dead or deadly makes me feel healthy and fit”: Findings from an Aboriginal women’s health and wellbeing program within the Shoalhaven region of New South Wales, Australia. Journal of Australian Indigenous Issues, 20(3), 44–62. https://doi.org/10.3316/informit.149529612973507.

37.

Gibbs

(2020). To close the gap, we need a strengths based discourse and respect for Aboriginal and Torres Strait Islander culture. NHLF. https://www.nhlf.org.au/wp-content/uploads/2021/12/NHLF_MR_Close-the-Gap-Campaign-Report_19-March-2020.pdf.

38.

Gollan

Stacey

(2021). Australian evaluation society first nations cultural safety framework. Australian Evaluation Society.

39.

Government of Canada . (2021). Indigenous health care in Canada. Government of Canada. https://www.sac-isc.gc.ca/eng/1626810177053/1626810219482

40.

Head

B. W.

(2010). Reconsidering evidence-based policy: Key issues and challenges. Policy and Society, 29(2), 77–94. https://doi.org/10.1016/j.polsoc.2010.03.001

41.

Henry

Foley

(2018). Indigenous research: Ontologies, axiologies, epistemologies and methodologies. In Booysen

L. A. E.

Bendl

Pringle

J. K.

(Eds.), Handbook of research methods in diversity management, equality and inclusion at work (pp. 212–227). Edward Elgar Publishing. https://doi.org/10.4337/9781783476084.00019

42.

Herman

R. D. K.

(2018). Approaching research in Indigenous settings: Nine guidelines. Toolbox on the research principles in and Aboriginal context: Ethics, respect, equity, reciprocity, collaboration and culture. In Gentelet

Basile

Gros-Louis McHugh

(Eds.), Toolbox of research principles in an Aboriginal context. First Nations of Quebec and Labrador Health and Social Services Commission, Université du Québec en Abitibi-Témiscamingue, Université du Québec en Outaouais. https://reseaudialog.ca/wp-content/uploads/2021/01/Toolbox_Research_Principles_Aboriginal_Context_eng.pdf

43.

Hokowhitu

Moreton-Robinson

Tuhiwai-Smith

Andersen

Larkin

(2020). Routledge handbook of critical Indigenous studies. Routledge.

44.

Hudson

(2017). Evaluation Indigenous programs: A toolkit for change (RR28). National Library of Australia. https://www.cis.org.au/wp-content/uploads/2017/06/rr28.pdf

45.

James

(2013). Designing evaluation strategies. In Productivity Commission (Ed.), Better Indigenous policies: The role of evaluation, roundtable proceedings (pp. 107–118). Productivity Commission.

46.

Katz

Newton

Bates

Raven

(2016). Evaluation theories and approaches, relevance for Aboriginal contexts. UNSW. https://www.aboriginalaffairs.nsw.gov.au/conversations/ochre/reports/Evaluation-theories-and-approaches-relevance-for-Aboriginal-contexts.pdf

47.

Kelaher

Luke

Ferdinand

Chamravi

Ewen

Paradies

(2018). An evaluation framework to improve Aboriginal and Torres Strait Islander health. The Lowitja Institute. https://www.lowitja.org.au/page/services/resources/health-services-and-workforce/service-solutions/Evaluation-Framework

48.

Kovach

(2021). Indigenous methodologies: Characteristics, conversations, and contexts. University of Toronto Press.

49.

Laenui

(2000). Reclaiming Indigenous voice and vision. In Battiste

M. A.

(Ed.), Processes of decolonization (pp. 150–160). University of British Columbia Press. https://www.deslibris.ca/ID/404356

50.

LaFrance

Nichols

(2008). Reframing evaluation: Defining an Indigenous evaluation framework. Canadian Journal of Program Evaluation, 23(2), 13–31. https://doi.org/10.3138/cjpe.23.003

51.

Larkin

(2006). Evidence-based policy making in Aboriginal and Torres Strait Islander health. Australian Aboriginal Studies (Canberra), 0(2), 17–26. https://doi.org/10.3316/informit.429508027685933.

52.

Lawrence

Houghton

Thomas

Weldon

(2014). Where is the evidence? Realising the value of grey literature for public policy and practice. Swinburne Institute for Social Research. [Discussion paper]. https://doi.org/10.4225/50/5580b1e02daf9

53.

Levac

Colquhoun

O’Brien

K. K.

(2010). Scoping studies: Advancing the methodology. Implementation Science, 5(1), 69–69. https://doi.org/10.1186/1748-5908-5-69

54.

Lowitja Institute . (2020). Close the gap (2020). Lowitja Institute. https://humanrights.gov.au/our-work/aboriginal-and-torres-strait-islander-social-justice/publications/close-gap-2020

55.

Lowitja Institute . (2022). Transforming power: Voices for generational change close the gap campaign report 2022. Lowitja Institute.

56.

Maddox

Blais

Mashford-Pringle

Monchalin

Firestone

Ziegler

Ninomiya

M. M.

Smylie

(2021). Reviewing health service and program evaluations in Indigenous contexts: A systematic review. American Journal of Evaluation, 42(3), 332–353. https://doi.org/10.1177/1098214020940409

57.

Maldonado-Torres

(2007). On the coloniality of being: Contributions to the development of a concept. Cultural Studies, 21(2–3), 240–270. https://doi.org/10.1080/09502380601162548

58.

Malezer

(2013). Challenges in evaluating Indigenous policy. Better Indigenous policies: The role of evaluation. In Productivity Commission (Ed.), Better Indigenous policies: The role of evaluation (pp. 69–79). Productivity Commission. https://www.pc.gov.au/research/supporting/better-indigenous-policies/better-indigenous-policies.pdf#page=75

59.

Martin

Mirraboopa

(2003). Ways of knowing, being and doing: A theoretical framework and methods for Indigenous and Indigenist re‐search. Journal of Australian Studies, 27(76), 203–214. https://doi.org/10.1080/14443050309387838

60.

McGrail

M. R.

Rickard

C. M.

Jones

(2006). Publish or perish: A systematic review of interventions to increase academic publication rates. Higher Education Research and Development, 25(1), 19–35. https://doi.org/10.1080/07294360500453053

61.

Mignolo

W. D.

Walsh

C. E.

(2018). On decoloniality: Concepts, analytics, praxis. Duke University Press.

62.

Moreton-Robinson

(2021). Talkin' up to the white woman: Indigenous women and feminism. University of Queensland Press.

63.

Muller

(2007). De-colonisation: Reflections and implications for social work practice. Communities, Children and Families Australia, 3(1), 80–87.

64.

Muller

(2023). Decolonisation: More than a trendy word. Australian Social Work, 76(3), 295–299. https://doi.org/10.1080/0312407X.2023.2193168

65.

Munn

Peters

M. D.

Stern

Tufanaru

McArthur

Aromataris

(2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18(1), 143–147. https://doi.org/10.1186/s12874-018-0611-x

66.

Nakata

M. N.

(2007). Disciplining the savages, savaging the disciplines. Aboriginal Studies Press. https://doi.org/10.3316/informit.9780855755485

67.

National Aboriginal Community Controlled Health Organisation (NACCHO) . (2022). Aboriginal-led initiatives and locally-led solutions is the answer! NACCHO on the close the gap campaign report. National Aboriginal Community Controlled Health Organisation (NACCHO). https://www.naccho.org.au/aboriginal-led-initiatives-and-locally-led-solutions-is-the-answer-naccho-on-the-close-the-gap-campaign-report/

68.

National Health and Medical Research Council . (2007, updated 2018). National statement on ethical conduct in human research. National Health and Medical Research Council. https://www.nhmrc.gov.au/about-us/publications/national-statement-ethical-conduct-human-research-2007-updated-2018

69.

National Health and Medical Research Council . (2018). Ethical conduct in research with Aboriginal and Torres Strait Islander Peoples and communities: Guidelines for researchers and stakeholders (IND2). National Health and Medical Research Council.

70.

National Indigenous Australians Agency . (2023). 2023 commonwealth closing the gap implementation plan. Commonwealth of Australia. https://www.niaa.gov.au/sites/default/files/publications/closing-the-gap-implementation-plan-2023.pdf

71.

Oliver

Lorenc

Tinkler

Bonell

(2019). Understanding the unintended consequences of public health policies: The views of policymakers and evaluators. BMC Public Health, 19(1), 1057–1059. https://doi.org/10.1186/s12889-019-7389-6

72.

Peters

Marnie

Tricco

Pollock

Munn

Alexander

McInerney

Godfrey

Khalil

(2020). Updated methodological guidance for the conduct of scoping reviews. JBI Evidence Synthesis, 18(10), 2119–2126. https://doi.org/10.11124/JBIES-20-00167

73.

Pham

M. T.

Rajić

Greig

J. D.

Sargeant

J. M.

Papadopoulos

McEwen

S. A.

(2014). A scoping review of scoping reviews: Advancing the approach and enhancing the consistency. Research Synthesis Methods, 5(4), 371–385. https://doi.org/10.1002/jrsm.1123

74.

Productivity Commission . (2020a). A guide to evaluation under the Indigenous Evaluation strategy. Commonwealth of Australia.

75.

Productivity Commission . (2012). Better Indigenous policies: The role of evaluation – [roundtable proceedings]. Productivity Commission. https://www.pc.gov.au/research/supporting/better-indigenous-policies

76.

Productivity Commission . (2020b). Indigenous evaluation strategy. Commonwealth of Australia.

77.

Productivity Commission . (2021). Closing the gap annual data compilation report july 2021. Commonwealth of Australia.

78.

Reconciliation Australia . (2016). The Redfern statement. Reconciliation Australia. https://www.reconciliation.org.au/election-2016-the-redfern-statement/

79.

Rigney

L.-I.

(1999). Internationalization of an Indigenous anticolonial cultural critique of research methodologies: A guide to indigenist research methodology and its principles. Wicazo Sa Review, 14(2), 109–121. https://doi.org/10.2307/1409555

80.

Rogers

Bower

Malla

Manhire

Rhodes

(2017). Developing a cultural protocol for evaluation. Evaluation Journal of Australasia, 17(2), 11–19. https://doi.org/10.1177/1035719X1701700203

81.

Rowe

Baldry

Earles

(2015). Decolonising social work research: Learning from critical Indigenous approaches. Australian Social Work, 68(3), 296–308. https://doi.org/10.1080/0312407X.2015.1024264

82.

Saldana

(2012). The coding manual for qualitative researchers (2nd ed.). Sage.

83.

Sherwood

(2009). Who is not coping with colonization? Laying out the map for decolonization. Australasian Psychiatry: Bulletin of Royal Australian and New Zealand College of Psychiatrists, 17(Suppl 1), S24–S27. https://doi.org/10.1080/10398560902948662

84.

Sherwood

Edwards

(2006). Decolonisation: A critical step for improving Aboriginal health. Contemporary Nurse, 22(2), 178–190. https://doi.org/10.5172/conu.2006.22.2.178

85.

Sherwood

Keenan

Kelly

Keech

(2011). Indigenous studies: Teaching and learning together. In Purdie

Milgate

Bell

H. R.

(Eds.), Two way teaching and learning: Towards culturally reflective and relevant education (pp. 189–204). ACER Press.

86.

Sherwood

J. M.

(2010). Do no harm. Decolonising Aboriginal Health Research University of NSW. https://doi.org/10.26190/unsworks/23129

87.

Smith

(2008). On tricky ground: Researching the native in the age of uncertainty. In Denzin

Lincoln

(Eds.), The Sage handbook of qualitative research (3rd ed., pp. 113–143). Sage.

88.

Smith

L. T.

(2021). Decolonizing methodologies: Research and Indigenous peoples. Bloomsbury Academic and Professional.

89.

Stack

Leal

Stamp

Reveruzzi

Middlin

Lennon

(2018). Complex evaluations in the political context: Engaging stakeholders in evaluation design. Evaluation Journal of Australasia, 18(2), 122–131. https://doi.org/10.1177/1035719X18778699

90.

Steering Committee for the Review of Government Service Provision . (2017). 2017 Indigenous expenditure report. Steering Committee for the Review of Government Service Provision. https://www.pc.gov.au/research/ongoing/indigenous-expenditure-report/2017/ier-2017-indigenous-expenditure-report.pdf

91.

Stevinson

Lawlor

D. A.

(2004). Searching multiple databases for systematic reviews: Added value or diminishing returns? Complementary Therapies in Medicine, 12(4), 228–232. https://doi.org/10.1016/j.ctim.2004.09.003

92.

Tricco

A. C.

Lillie

Zarin

O’Brien

K. K.

Colquhoun

Levac

Moher

Peters

M. D. J.

Horsley

Weeks

Hempel

Akl

E. A.

Chang

McGowan

Stewart

Hartling

Aldcroft

Wilson

M. G.

Garritty

Straus

S. E.

(2018). PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/m18-0850

93.

Tunón

Kvarnström

Lerner

(2016). Ethical codes of conduct for research related to Indigenous peoples and local communities – core principles, challenges and opportunities. In Drugge

A.-L.

(Ed.), Ethics in Indigenous research past experiences - future challenges (pp. 57–80). Vaartoe - Centre for Sami Research Umeå Universitet. http://umu.diva-portal.org/smash/get/diva2:943266/FULLTEXT03.pdf.

94.

United Nations . (2009). State of the world’s Indigenous peoples. The Department of Economic and Social Affairs of the United Nation Secretariat.

95.

Walker

Fredericks

Mills

Anderson

(2013). Perspectives on a decolonizing approach to research about Indigenous women’s health: The Indigenous women’s wellness study. Alternative: An International Journal of Indigenous Peoples, 9(3), 204–216. https://doi.org/10.1177/117718011300900302

96.

Walter

Andersen

(2016). Indigenous statistics: A quantitative research methodology. Routledge. https://doi.org/10.4324/9781315426570

97.

Weaver

Cousins

J. B.

(2007). Unpacking the participatory process. Journal of Multidisciplinary Evaluation, 1(1), 19–40. https://doi.org/10.56645/jmde.v1i1.144

98.

Williams

(2018). Ngaa-bi-nya Aboriginal and Torres Strait Islander program evaluation framework. Evaluation Journal of Australasia, 18(1), 6–20. https://doi.org/10.1177/1035719X18760141

99.

Wilson

(2008). Research is ceremony. Indigenous research methods. Fernwood Publishing.