Working towards accountability in embedding Indigenous studies: Evidence from an Indigenous Graduate Attribute evaluation instrument

Abstract

Whilst Indigenous Graduate Attributes – or the embedding Indigenous cultural competencies within broad graduate attributes – are becoming increasingly popular within some universities, it is essential that universities be held accountable for the realisation of such policies. Considering that Australian Indigenous Studies – an essential component of Indigenous Graduate Attributes – is a highly contested space where colonial and Indigenous knowledges collide, this article presents evidence from analyses, engaging with Indigenous Standpoints, aimed at evaluating critically the degree to which university subjects may contribute to the realisation of Indigenous Graduate Attributes. Results identify not only an array of psychometrically sound factors which measure Student Knowledges and Attitudes to Indigenous Issues as well as Applied Indigenous Learning, but also indicate that the embedding of Australian Indigenous Studies content may vary across disciplines. In addition, results show that the impact of such embedding on student attitudes also varies greatly according to discipline, with positive, negative, and contradicting results across disciplines. These findings strongly suggest that any commitment to embedding Indigenous Graduate Attributes must be monitored very carefully.

Keywords

Australian Indigenous Studies Indigenous Graduate Attributes higher education Indigenous standpoints confirmatory factor analysis colonisation

Universities have long committed to a diversity of generalised student graduate attributes not only to guide their teaching and learning environments, but also to highlight that graduating students have developed a valuable set of professional capabilities that would make them more employable (Bradley, Noonan, Nugent, & Scales, 2008). A growing number of government reports and scholarly articles have called for the inclusion of graduate attributes related to “Indigenous-specific competencies” that may facilitate the graduate workforce in producing better outcomes for Aboriginal and Torres Strait Islander peoples and communities (Anning, 2010; Behrendt, Larkin, Griew, & Kelly, 2012; Bradley et al., 2008; Frawley, 2017; Universities Australia, 2011, 2017).

Whilst the ideal of Indigenous Graduate Attributes has considerable potential, care must be taken to recognise that policy, practice, and “positive outcomes” do not automatically align. This may be especially the case when considering the wide diversity of disciplines within universities and the highly contested space of cultural competencies and cultural responsiveness within Indigenous Studies (Norman, 2014; Page, Trudgett, & Bodkin-Andrews, 2016; Rigney, 2017; Sherwood & Russell-Mundine, 2017).

This possible discord between university policy and teaching and learning practices is the focal point of this article, as it will outline the development of a critical Indigenous framework that can contribute toward a greater level of accountability in teaching and learning commitment to Indigenous Graduate Attributes across the disciplines. Therefore, it is the overarching aim of this article to report on the pilot testing of a student evaluation measure that, in part, critically evaluates the nature of university classroom practices that may or may not align with a university’s development and engagement with Indigenous Graduate Attributes and associated cultural competencies.

The Indigenisation of graduate attributes?

At the turn of the decade, Universities Australia (2011) called for the development of specific Indigenous cultural competencies within graduate attribute frameworks. More recently, Universities Australia (2017) gave a concrete target for all universities to commit to Indigenous Graduate Attributes:

By 2020, universities commit to have plans for, or have already in place, processes that ensure all students will encounter and engage with Aboriginal and Torres Strait Islander cultural content as integral parts of their course of study. (p. 30)

Whilst an increasing number of Australian universities have begun to implement Indigenous Graduate Attributes, concern should be raised as to the rate of development if the Universities Australia target is to be met. Indeed, in an earlier review of university graduate attributes, Bidjara scholar Anning (2010) reported that only 9 out of 36 audited universities had referenced Indigenous peoples within their graduate attribute statements. Roughly seven years later, Frawley (2017) reviewed 43 Australian universities and found that only 12 included direct reference to Indigenous Australians or contained graduate attributes that implied a possible Indigenous dimension (e.g., cultural competence).

Hence, the rate of university engagement with Indigenous Graduate Attributes is of concern, as is the very nature of how Indigenous Graduate Attributes may be related to content within the diverse teaching and learning environments of universities (Flavell, Thackrah, & Hoffman, 2013; Virdun et al., 2013). As noted by Anning (2010), teaching and learning is not simply teaching “about” Indigenous Australians and promoting values associated with diversity, reconciliation, and respect, but also the inclusion of learning of the foundations of Indigenous studies content, engagement with critical theory, critical engagement with professional contexts and impacts, access and knowledge of services and events related to Indigenous peoples, fostering positive and safe learning environments, engagement with specific case studies to emphasise the diversities of Indigenous contexts and knowledges, use of varying teaching methodologies to engage with varying student learning styles, promoting critical self-reflection, supporting Indigenous and non-Indigenous teaching staff, and promoting collaboration with a diversity of academics across disciplines.

The emphasis on varying approaches to critical thought within Anning’s (2010) strategic framework is essential, as many Indigenous scholars have noted the epistemological tensions within the very knowledge production, and learning content, that falls under the broad label of Indigenous studies (Judd, 2014; Moodie, 2019; Rigney, 2017; Smith, 2012). For example, Biripi and Worimi scholar Cross-Townsend (2011) powerfully argues that:

Indigenous studies, in many instances, continues to be informed by a preponderance of research where Indigenous peoples are the sole focus of predominantly ethnographic analysis that fails to critically examine or problematize the impact of dominant cultural ideologies, theories or practices on Indigenous peoples’ knowledges and socio-economic realities. (p. 72)

Universities, as institutionalised systems of knowledge transmission and production, have historically – and in many cases continually – failed to accurately represent, and be governed by, diverse Indigenous Knowledges, pedagogies, and practices (Andersen & Kukutai, 2016; Bodkin-Andrews & Carlson, 2013; Moodie, 2019; Rigney, 2017). Whilst there has been considerable recognition of the changing landscape of Indigenous Studies within the higher education sector (Nakata, 2013; Shay, 2016; Tallbear, 2016), Narungga scholar Rigney (2017) suggests that this progress has been inconsistent, fragmented, and poorly resourced within and across Australian universities.

The fragmentation to which Rigney (2017) refers is also evident within the lack of recognition of Indigenous Australians across the majority of university graduate attributes nation-wide and in some cases the very teaching and learning content (and institutional support) that ultimately should see the realisation of Indigenous Graduate Attributes (Aberdeen, Carter, Grogan, & Hollinsworth, 2013; Hogarth, 2017). It is critical, then, that universities, which have committed to Indigenous Graduate Attributes, reflect on teaching and learning practices that engage with Indigenous studies across disciples, and their subsequent impact on both Indigenous and non-Indigenous students.

Indigenous data sovereignty and Indigenous Graduate Attributes

The Indigenous Graduate Attribute project at the University of Technology Sydney (UTS) was in direct response to the Behrendt Review’s (Behrendt et al., 2012) call for whole of university approaches to enabling all university graduates to obtain a minimum-level capacity to both respect and work with Aboriginal and Torres Strait Islander peoples and communities (Sherwood, McDaniel, & McKenzie, 2013). To aid in the realisation that all graduates develop a required level of Indigenous professional competency, regardless of discipline, an Indigenous employment strategy (UTS, 2017) was implemented to ensure at least one Indigenous academic leader (at Associate Professor and Professor levels) was to be situated within every faculty to oversee teaching, learning, and research practices. As of August 2018, UTS held the highest number of Indigenous Professorial appointments in the country, with 13 appointments in total. Embedded within this strategic initiative was the creation of a specialist Indigenous research centre with three senior appointments – two Professorial, one Associate Professor – to oversee the development, implementation, and evaluation of the university-wide Indigenous Graduate Attribute (Page et al., 2016). It may be argued that these appointments and strategies moved beyond tokenistic attempts by some institutions to “Indigenise the curriculum” by appointing a small number of staff members at lower levels who may or may not be Indigenous. It should also be recognised that within the evaluation context, the university has placed the responsibilities of identifying, defining, creating, and implementing the evaluation tools and data collection processes under the ownership of an all Indigenous staffed centre (Page et al., 2016).

With the Indigenous Graduate Attribute development and evaluation being placed under Indigenous leadership, the university has aligned its practices within the principles of varying Indigenous Data Sovereignty movements that have emerged in Canada, New Zealand, and Australia (Kukutai & Taylor, 2016; Walter, 2018). At its simplest, the Indigenous Data Sovereignty movement seeks to actively redress the paternalistic legacy of Indigenous peoples being alienated and missing from conversations and practices pertaining to the “collection, ownership and application of data about their people, lifeways and territories” (Kukutai & Taylor, 2016, p. 2). Numerous scholars have argued that the foundations of Indigenous Data Sovereignty align with the United Nations Declaration on the Rights of Indigenous Peoples (United Nations General Assembly, 2007). with a particular emphasis placed on articles surrounding the right of self-determination, autonomy, and self-governance of Indigenous peoples (Davis, 2016; Kukutai & Taylor, 2016; Rainie, Schultz, Briggs, Riggs, & Palmanteer-Holder, 2017; Walter, 2018). From this, the collection of data aimed at “representing” Indigenous peoples moves beyond simplistic and problematic data about Indigenous disadvantage (Hogarth, 2017; Walter, 2018) to focus on data which work for Indigenous peoples and their sovereign needs. This is eloquently argued by palawa scholar Walter (2018), who stated that:

On the ‘data we need’ side, Aboriginal and Torres Strait Islander peoples require data that: are meaningful and useful, informing a comprehensive, nuanced narrative of who we are as peoples, of our culture, our communities, our resilience, goals and successes; recognise cultural and geographical diversity and can provide evidence for community-level planning and service delivery; can be contextualised to include the wider social structural complexities in which Indigenous disadvantage occurs; measure priorities and agendas, not just problems; and to be accessible and useable. (pp. 258–259)

Within this article, it is important to note that the data collected are not focussed on the responses of Indigenous Australian students who are reporting or voicing their lived experiences within the higher education sector (e.g., Moodie, Ewen, McLeod, & Platania-Phung, 2018), but rather it is an attempt to critically understand the nature of student perceptions of Indigenous learning content within their discipline (e.g., Aberdeen et al., 2013). Although this is effectively “non-Indigenous data”, the very content in which non-Indigenous students are reporting on has direct implications for Indigenous Australian peoples and communities. To place this article utilising “non-Indigenous data” in an Indigenous Data Sovereignty context, we take a similar vein of reasoning to that of Snipp (2016), who notes that while UNDRIP is ultimately framed in a context that defines Indigeneity with regard to Indigenous peoples relationship to colonial contact, this in no ways suggests that Indigenous peoples should not be given the right to self-determine their own identities. This article then is focused on the teaching and learning content that seeks to represent not only Indigenous peoples and communities, but also how non-Indigenous peoples may positively learn to work with Indigenous peoples.

An Indigenous quantitative research methodology

With Indigenous Data Sovereignty frameworks ultimately pertaining to the governance, access, collection, and use of multiple types of data by Indigenous nations and communities (Kukutai & Taylor, 2016; Rainie et al., 2017), it is also important to note that quantitative data (e.g., survey and population data) are not excluded from such frameworks. Whilst quantitative data are frequently cited as misrepresenting Indigenous peoples and communities (Nakata, 2007), an increasing number of Indigenous scholars are committed to deeply engaging with Indigenous epistemologies as a primary driver for realising Indigenous quantitative methodologies and methods (Andersen & Kukutai, 2016; Bodkin-Andrews et al., 2017; Rainie et al., 2017; Walter, 2016). Walter and Andersen’s (2013) seminal text centred quantitative research on the complexities of Indigenous Standpoints (Indigenous social positioning, epistemology, axiology, and ontology). This resulted in statistical methods which do not prioritise non-Indigenous standpoints that minimise and erase Indigeneity in all its diversity, but instead seek to breakdown systemic power differentials within research itself, and be continually mindful of the risk of perpetuating the norms of colonisation that statistics have too often reinforced (Walter & Andersen, 2013).

Within this article, the methodological engagement is led by three Indigenous Australian scholars who have dedicated much of their academic research to centring Indigenous standpoints and knowledges within teaching learning and research (cf. Bodkin-Andrews, Page, & Trudgett, 2018; Page, Trudgett, & Bodkin-Andrews, 2019). This prioritisation of Indigenous lenses has led the authors to deeply engage with numerous critical Indigenous Standpoint theories and perspectives (Behrendt, 2016; Foley, 2003; Moreton-Robinson, 2015; Nakata, 2007; Smallwood, 2015; Walter & Andersen, 2013), which serve as the primary driver for the development of the measures within this article. Inspired by these works, and under the guidance of a majority Indigenous Management Committee, consisting of six Indigenous scholars and three non-Indigenous scholars, an overarching evaluation framework was developed to monitor the sources and quality of teaching, learning, and research practices dedicated to embedding Indigenous Knowledges and studies at UTS (see Figure 1).

Figure 1.

The [anonymous] Indigenous Graduate Attribute evaluation framework.

The development of this evaluation model is more fully articulated in Bodkin-Andrews et al. (2019). Still, the model can be summarised as being drawn from an intersection of existing Indigenous Graduate Attribute literature, university-specific Indigenous Graduate Attribute policy documents, key Critical Indigenous Studies scholarly works, and extensive consultations with the project steering committee, Indigenous Graduate Attribute (IGA) “champions” (those who had already contributed to varying IGA-related frameworks across the disciplines), and a broad range of executive teaching and learning committees. Overall, five overarching themes were articulated and may contribute to identifying increased levels of accountability for embedding Indigenous content within both mainstream and Indigenous-specific teaching and learning practice.

Firstly, one must recognise that a strong and re-occurring theme within the critical Indigenous scholarly literature is the recognition that Eurocentric research has too often minimised and misrepresented the lived realities of Indigenous Australian peoples and communities (Colonial Knowledges). The very foundations of such Eurocentrism is still evident within research today requiring a development of critical awareness of self-and-othering narratives “about” Indigenous peoples and the very “knowledges” to which both Indigenous and non-Indigenous students are exposed (Reflectivity). A reflexive engagement with such a limited knowledge base should assist in developing a greater understanding of how all people may be better informed by the voices and lived experiences of Indigenous Australian peoples, scholars, and communities (Indigenous Voice). By listening deeply to these voices, the awareness of, and engagement with Indigenous ways of knowing, being, and doing becomes imperative, as students must be prepared to move beyond Western theory and practice to engage with Indigenous Knowledges and communities and their protocols (Indigenous Ways). In doing so, students will begin to have the potential to commit to deeper and more meaningful levels of engagement with Indigenous Australian peoples and communities (Indigenous Engagement).

Based on the foundations of this IGA framework, a self-report student evaluation instrument was developed to measure the beliefs, attitudes, and learning experiences of students who have undertaken course work that may have Indigenous learning content embedded within it (see materials section for more detail). The instrument consists of two measures, namely, the Student Knowledges and Attitudes to Indigenous Issues and the Applied Indigenous Learning measures. The Student Knowledges and Attitudes to Indigenous Issues measure captures seven factors represented by between three and five items for each factor, while the Applied Indigenous Learning measure captures six factors represented by between four and five items for each factor. These 13 interrelated attitudinal and Indigenous learning factors (see also the oval shaped boxes in Figure 2) fell within the five IGA Evaluations themes. Simple definitions of the five IGA evaluation themes together with the student attitude and Indigenous learning factors that are considered to be linked to each theme for the IGA Evaluation Instrument are provided below (see Appendix 1 for the 52 items written to capture these 13 factors):

Figure 2.

Nested structural model for Student Demographics and Learning predicting Student Beliefs.

Colonial Knowledge: An understanding of the historical and contemporary narratives directed at Indigenous peoples and communities within the media, politics, and research. This theme is captured in the factors Deficit discourses and simplistic representations of Indigenous disadvantage (e.g., key indicators for close-the-gap), but also overlaps with Critical Cultural Representations and Two-ways.

Reflectivity: An open and critical dissemination of the antecedents to both self-and-other knowledges “about” Indigenous peoples and communities, and how they interact with the systemic silences and racisms that can plague Indigenous studies (e.g., epistemological silences, racism, bias). Factors related to the theme of Reflectivity include Critical Cultural Representations, Future Indigenous Engagement, Indigenous Assessment, and Indigenous Learning Value.

Indigenous Voice: An in-depth engagement with the complexities of Indigenous led research and community practices, and how this may sit against ongoing colonial narratives (e.g., Indigenous standpoint theories and Indigenous research methodologies). Factors tapping into the theme of Indigenous Voice include Two-ways, Indigenous Diversities, Indigenous Standpoints, and Indigenous Representatives.

Indigenous Ways: The development of a stronger appreciation and engagement with Indigenous protocols and ethics (e.g., Australian Institute of Aboriginal and Torres Strait Islander Studies, 2012), respecting and working with Indigenous communities and Indigenous Knowledges. The factors of Respectful Learning, Indigenous Diversities, Two-ways, Indigenous Learning Value, and Indigenous Standpoints all draw from this theme.

Indigenous Engagement: The applied knowledge, ability, and confidence to work with Indigenous Australian peoples and communities (e.g., willingness to engage across disciplines and Country). The factors linking into this theme include Future Indigenous Engagement, Indigenous representatives, Indigenous Assessment, and Indigenous Learning Value.

Research questions

Based on the overarching aim of this article to report on the development of an IGA Instrument, results of a series of psychometric and inferential tests are reported in the following sections to address the five main research questions of this article:

Overall, can psychometric testing reveal acceptable model fits for the Student Knowledges and Attitudes to Indigenous Issues and Applied Indigenous Learning measures?

Based on prior best practice in psychometric research (Byrne, 2012; Parker, Martin, Martinez, Marsh, & Jackson, 2010), through the use of invariance testing, to what extent are the two measures psychometrically equivalent in responses across male and female students?

How did the students respond (agree/disagree) to each of the two measures?

After controlling for student background variables, to what extent are the Student Knowledges and Attitudes to Indigenous Issues and Applied Indigenous Learning measures statistically associated with each other?

How do the two measures operate differently across disciplines?

Methods

Participants and procedure

A total of 276 higher education students from the University of Technology Sydney were recruited across five broad university disciplines (three first-year subjects, one second-year subject, and one post-graduate). Due to ethical requirements, specific subjects were not named to protect the identities of unit coordinators. The total sample consisted of 112 male students (mean age of 20.40 years) and 164 female students (mean age of 20.67 years), with an overall age range from 17 to 56 years (mean age of 20.56 years) who volunteered to participate. Table 1 presents the basic demographic variables for each discipline, split by gender.

Table 1.

Student background variables.

	Mean age	Gender		Number of Indigenous students	Mode university year	Number of international students	Mean financial status^a
	Mean age	Male	Female	Number of Indigenous students	Mode university year	Number of international students	Male	Female
Education (n = 35)	19.23	<5^b	31	0	1	<5^b	4.25	4.90
Law (n = 43)	19.37	21	22	0	1	<5^b	4.90	4.37
Design (n = 82)	20.78	18	64	<5^b	2	<5^b	4.56	4.05
Engineering (n = 88)	19.78	65	23	0	1	18	4.48	4.32
Health (n = 28)	26.04	<5^b	24	<5^b	5	0	4.25	3.92

Financial status scores ranged from 1 (“I run out of money”) to 6 (“I can save a lot”).

Due to protection of anonymity concerns, cell sizes of 5 or under are not reported.

After clearance from a project steering committee was granted (five of eight members being Indigenous Australian, all employed by the university at an Associate Professor level or above) and the UTS Human Resources ethics committee (HREC approval number ETH16-0585¹), invitation emails were sent to faculty and school leaders to nominate subject unit-coordinators. This invitation explained that the project centred on pilot testing for the development of an evaluation survey under the wider implementation of a university-wide Indigenous Graduate Attribute project. Unit-coordinators who agreed to participate then allowed the researchers to enter specific tutorial classes in the last three weeks of the semester to administer the pilot survey. Surveys were handed to all attending students, who were instructed that their participation was completely voluntary (students were allowed to withdraw without consequence), that all surveys were de-identified (consent forms removed), and individual responses were kept confidential (results were to be only reported in a generalised manner). The full survey took 20 minutes to complete, and after the collection of all surveys by the researchers, a $20 gift voucher was given to one of the randomly selected students from each tutorial class (regardless of participation or not) as a small thank you for their time (this was not mentioned prior to survey administration).

Materials

For the purpose of this article, questions related to student background variables, student knowledge and attitudes about Aboriginal and Torres Strait Islander peoples and issues, and experiences of Indigenous learning content are summarised below.

Student background variables

As listed in Table 1, a series of student background variables were utilised for analyses within this investigation. Age was an open-ended question and treated as a continuous variable. Gender was coded as a dichotomous variable –1 (male) and 1 (female), University year was treated as an ordinal variable – ranging from first year (1) through to fourth year (4), then post-graduate (5). A dichotomous variable was used to indicate place of birth status with –1 indicating Australian and 1 indicating international (born overseas), and the Financial status variable ranged from 1 (“I run out of money”) to 6 (“I can save a lot”).

Student Knowledge and Attitudes to Indigenous Issues

For the purposes of this analysis, the seven latent factors of Student Knowledge and Attitudes to Indigenous Issues were conceptualised as outcome variables. All questions pertaining to each factor were scored on a six-point Likert Scale ranging from 1 (Completely False) to 6 (Completely True). See Appendix 1 for the full item listing.

Indigenous Disadvantage: Drawing from the Colonial Knowledges theme, this five-item factor was designed to capture student knowledge of some of the systemic disadvantages endured by Indigenous Australians. Each question beginning with the item-stem “On average, when compared to non-Indigenous Australians, Indigenous Australians.” An example item is “have lower educational outcomes.”

Indigenous Deficit Thinking: Drawing from the Colonial Knowledges theme, this is a four-item factor designed to capture the extent to which students feel Indigenous Australians are responsible for the ongoing inequalities today. Each question began with the item-stem “Most of the inequalities suffered by Indigenous Australians.” An example item includes “exist because of their fixation on outdated cultural practices.”

Critical Cultural Representations: Drawing from the Colonial Knowledges and Reflectivity themes, this is a six-item factor written to assess the extent to which students feel that Indigenous Australians are misrepresented by Australian knowledge production bodies (media, politics, and research). Each question began with the item-stem “Too many,” and an example item is “politicians fail to truly understand the needs of Indigenous Australians.”

Future Indigenous Engagement: Drawing from the Reflectivity and Indigenous Engagement themes, this is a six-item factor attempting to address the degree to which students’ may be willing to work with Indigenous peoples and communities. Each question began with the item-stem “In the future, I would,” and an example item is “be confident to work with Indigenous Australians.”

Two-ways: Drawing from the Colonial Knowledges, Indigenous Voice, and Indigenous Ways themes, this is a four-item factor produced to assess the degree to which students feel that Indigenous and non-Indigenous peoples and knowledges can combine to produce positive outcomes for Indigenous communities. Each question began with the item-stem “Both Indigenous and non-Indigenous,” and an example item is “can combine to produce productive outcomes for Indigenous communities.”

Indigenous Diversities: Drawing from the Indigenous Voice and Indigenous Ways themes, this is a four-item factor designed to examine the extent to which students’ recognise the diversities across Indigenous Australian nations. Each question began with the item-stem “Indigenous Australians,” and an example item is “have nations and clans that have many different customs.”

Applied Indigenous Learning

As mentioned earlier, six latent factors were conceptualised as strategic teaching and learning practices that are Indigenous centred, and may be experienced by students within the classroom. All questions pertaining to each factor were scored on a six-point Likert Scale ranging from 1 (Completely False) to 6 (Completely True). See Appendix 1 for the full item listing.

Respectful Learning: Drawing from the Indigenous Ways theme, this is a four-item factor assessing the degree to which the students felt the classroom was a safe place to discuss Indigenous issues. Each question began with the item-stem “In the classroom,” and an example item includes “Indigenous Australian issues were discussed in an atmosphere of mutual respect.”

Indigenous Standpoints: Drawing from the Indigenous Voice and Indigenous Ways themes, this is a five-item factor designed to capture the extent to which students feel they were required to engage with Indigenous standpoints within their subject. The item stem for this measure was “Within this subject,” and an example item is “we learned of many unique Indigenous Australian Knowledges that have survived colonisation.”

Indigenous Representatives: Drawing from the Indigenous Voice and Indigenous Engagement themes, this is a four-item factor assessing the degree to which the students experienced learning directly from Indigenous Australian representatives within the subject. Each question began with the item-stem “In this subject, we,” and an example item is “we were given the opportunity to listen to a number of Indigenous Australian representatives.”

Indigenous Assessment: Drawing from the Reflectivity and Indigenous Engagement themes, this is a four-item factor assessing the degree to which students agreed that at least one of their subject assessments was focussed on Indigenous Australian issues. The item stem was “For at least one of our assessments,” and an example item is “we had to critically evaluate multiple perspectives on an issue affecting Indigenous Australians.”

Indigenous Learning Value: Drawing from the Reflectivity and Indigenous Engagement themes, this is a four-item factor assessing the degree to which students agreed that the Indigenous learning content within their subject was important. The item stem was “Overall,” and an example item included “I think this subject showed how important Indigenous Australian perspectives are as part of my degree.”

Statistical Analysis

Utilising MPLUS 7.31, a mixture of Confirmatory Factor Analysis (CFA) and Structural Equation Modelling (SEM) techniques (Byrne, 2012) was employed. Firstly, with regard to examining the psychometric properties, the initial analysis consisted of two total sample CFAs to assess the psychometric validity of the Student Knowledge and Attitudes to Indigenous Learning and Applied Indigenous Learning measures. Based on the advice of Marsh, Tracey, and Craven (2006), the two models’ a-priori factor structure was assessed using a variety of goodness of fit indices. Overall, goodness of fit indices allow for an assessment of the degree to which the theoretical model may be consistent (or fits) with the empirical data itself (Byrne, 2012). As there is a general consensus that no single fit index is superior, and that varying indices operate differently under varying conditions (e.g., sample size, normality, model complexity), the following fit indices were utilised, the Root Mean Square Error of Approximation (RMSEA), Comparative Fit Index (CFI), and Tucker-Lewis Index (TLI). For the RMSEA, values less than .05 indicate close fit (.08 acceptable), and values above .95 for the CFI and TLI represent excellent fits (.90 acceptable). Considerable attention was also placed on factor-to-item loadings (values below .50 were considered unacceptable), factor correlations (correlations above .90 were considered unacceptable), and correlated residuals. If these conditions were violated, offending items were deleted (or merged in the case of factor correlations) and the CFA re-run.

Once satisfactory fit-indices were established for the two CFAs, factorial invariance tests were conducted where equivalence restraints were placed on the factor loadings (metric) and item intercepts (scalar) to establish if the factor structure was similarly understood by varying student groups (Parker et al., 2010). Variation of less than or equal to .01 of the CFI, overlap in the 90% confidence interval of the RMSEA, and overall model fits were used indicate the degree to which the assumptions of equivalence/invariance were met. The factor invariance testing was conducted across students split by gender. Although invariance testing across each discipline would have been ideal, larger sample sizes for each discipline would have been required for this to occur.

A total-group step-wise partial SEM was conducted to identify if the standardised paths emanating from the background variables (e.g., gender, finance) predicted the Student Knowledges and Attitudes to Indigenous Issues as well as the Applied Indigenous Learning measures (see Figure 2). As the next step, the standardised predictive paths between the Student Knowledges and Attitudes to Indigenous Issues and Applied Indigenous Learning measures were examined – effectively acting as a partial regression with the predictive variance of the background variables being taken into account. This was followed by a spitting of the total sample into disciplines where Pearson’s r correlations were used to examine variation in associations across disciplines.

Results

The results section is split into the five primary research questions.

Research Question 1: Psychometric properties of the measures

Student Knowledges and Attitudes to Indigenous Issues

A preliminary CFA revealed only marginally acceptable goodness of fit criteria for the 6-factor, 30-item model (χ² = 762.02, df = 390, TLI = .91, CFI = .92, RMSEA = .06). An examination of the correlated residual estimates revealed 45 paired items with significant residual estimates above 3.84. To further strengthen the fit indices of the CFA model, the pattern of correlated residuals was examined carefully and excessively offending items were deleted from the model (residual estimates > 10.00). The model was re-run after every item deletion until the overall goodness of fit estimates were deemed strong. Whilst detailed reporting of this process is beyond the scope of this article, it is important to note that the Future Indigenous Engagement factor held a pattern of correlated residuals that suggested two separate factors may exist within the original one-factor conceptualisation. That is, items 1 to 3 held strong correlated residuals estimates (from 7.15 to 15.83) as did items 4 to 6 (from 6.83 to 27.51). After an examination of the individual items (see Appendix 1), it was decided that this factor would be split into two spate factors named Future Confidence and Future Adaptability.

After the deletion of three items and the splitting of the Future Indigenous Engagement factor, a final model was formed with strong fit indices. Table 2 offers the revised seven-factor model’s goodness of fit criteria, factor loadings, and correlations. As can be noted from this model, all goodness if fit indices are strong, the factor-to-item loadings are highly significant and substantial in magnitude (from .53 to .94), and the factor correlations are within an acceptable range (–.66 to .76).

Table 2.

Student Knowledges and Attitudes to Indigenous Issues CFA goodness of fit indices.

	Goodness of fit indices
	χ²	df	TLI	CFI	RMSEA
	451.22	303	.96	.96	.04
	Disad	Deficit	Crit Rep	Confi	Adapt	Two-ways	Diversity
Factor-to-item loadings
Item 1	.81	.67	.75	.73	.59	.75	.94
Item 2	.66	.89	.53	.89	.76	.83	.94
Item 3	.88	.84	.66	.88	.74	.90	.91
Item 4	.72	.88	.82	–	–	.80	–
Item 5	.75	–	.59	–	–	–	–
Factor correlations
Disad	1.00
Deficit	–.13	1.00
Crit Rep	.36***	–.40***	1.00
Confi	.12	–.39***	.13	1.00
Adapt	.11	–.66***	.33***	.76***	1.00
Two-ways	.08	–.42***	.31***	.43***	.67***	1.00
Diversity	.31***	–.40***	.33***	.23***	.38***	.34***	1.00

χ²: Chi square; df: degrees of freedom; TLI: Tucker Lewis Index; CFI: Confirmatory Fit Index; RMSEA: Root Mean Square Error of Approximation; Disad: Disadvantage; Crit Rep: Critical Representations; Confi: Confidence; Adapt: Adaptability. * = p < .05, ** = p < .01,*** = p < .001.

Applied Indigenous Learning measure

Table 3 offers the goodness of fit criteria for the 6-factor, 25-item CFA model. Although there were 14 correlated residual estimates above 3.84, due to the strong TLI, CFI fit indices, and acceptable RMSEA fit, no further adjustments were made to this model. All standardised factor loadings are significant and moderate to strong in magnitude (ranging from .58 to .95). Latent factor correlations ranged from .30 (Respectful Classroom with Indigenous Representatives) to .85 (Value of Indigenous Learning with Indigenous Standpoints).

Research Question 1 summary – Overall, can psychometric testing reveal acceptable model fits for the Student Knowledges and Attitudes to Indigenous Issues and the Applied Indigenous Learning measures?

Two separate models pertaining to Student Knowledges and Attitudes to Indigenous Issues and Applied Indigenous Learning perceptions were found to have strong fit indices, suggesting that for the factors identified (e.g., Confidence, Indigenous Standpoints), they were strong and independent representations of their designated questions. This included the splitting of the Future Indigenous Engagement factor into Future Confidence and Future Adaptability for working with Indigenous Australians. Overall, after some model modifications, Research Question 1 was supported as acceptable (strong) model fits were achieved.

Table 3.

Applied Indigenous Learning CFA goodness of fit indices.

	Goodness of fit indices
	χ²	df	TLI	CFI	RMSEA
	566.14	260	.96	.95	.06
	Respect	Author	Represent	Standpoint	Assessment	Value
Factor-to-item loadings
Item 1	.60	.90	.85	.87	.95	.95
Item 2	.87	.92	.92	.93	.95	.95
Item 3	.79	.90	.95	.87	.97	.94
Item 4	.92	.89	.91	.90	.95	.58
Item 5	–	–	–	.93	–	–
Factor correlations
Respect	1.00
Author	.43***	1.00
Represent	.30***	.64***	1.00
Standpoint	.45***	.72***	.71***	1.00
Assessment	.36***	.67***	.39***	.68***	1.00
Value	.39***	.64***	.56***	.85***	.68***	1.00

χ²: Chi square; df: degrees of freedom; TLI: Tucker Lewis Index; CFI: Confirmatory Fit Index, RMSEA: Root Mean Square Error of Approximation. * = p < .05, ** = p < .01, *** = p < .001.

Research Question 2: Invariance testing across gender

The second phase of psychometric testing for the Student Knowledges and Attitudes to Indigenous Issues and Applied Indigenous Learning measures consisted of Factorial Invariance testing to ascertain if the response patterns to the measure were consistent across gender (allowing greater confidence that the measures have similar meaning across groups). Table 4 offers a summary of the equivalence testing across the completely free, metric, and scale models (Parker et al., 2010) and reveals little change in not only the overall fit indices, but also the CFI, and overall in the 90% confidence interval for the RMSEA for the Student Knowledges and Attitudes to Indigenous Issues measure.

Table 4.

Student Knowledges and Attitudes to Indigenous Issues as well as Applied Indigenous Learning Invariance across Gender (Male, n = 162; Female, n = 112).

	χ²	df	CFI	TLI	RMSEA	90% RMSEA
Student Knowledges and Attitudes to Indigenous Issues
Model 1: Completely free	906.41	606	.931	.920	.060	.052−.068
Model 2: Metric (factor loadings)	940.81	633	.930	.922	.060	.051−.067
Model 3: Metric + scalar (item intercepts)	967.53	660	.930	.925	.059	.050−.066
Applied Indigenous Learning
Model 1: Completely free	886.91	520	.948	.941	.074	.065−.082
Model 2: Metric (factor loadings)	958.39	545	.942	.936	.077	.068−.084
Model 3: Metric + scalar (item intercepts)	1058.79	570	.931	.928	.081	.074−.089

χ²: Chi square; df: degrees of freedom; TLI: Tucker Lewis Index; CFI: Confirmatory Fit Index, RMSEA: Root Mean Square Error of Approximation.

As per Table 4, the invariance testing for the Applied Indigenous Learning measures across gender also revealed little change in the goodness of fit indices for metric invariance (acceptable overall fit indices, less than ± .01 change in CFI, and overlap in 90% confidence interval of RMSEA – Parker et al., 2010), yet the ± .01 change in the CFI rule was violated for scalar invariance (a change of .017). This suggests that the minimal requirement for invariance was met for the Applied Indigenous Learning measure, but some care should be taken in direct comparisons for this measure across gender.

Research Question 2 summary – Through the use of invariance testing, to what extent are the measures equally understood by male and female students?

Both Student Knowledges and Attitudes to Indigenous Issues and Applied Indigenous Learning measures achieved the minimal requirement for invariance (metric), suggesting that the factors were equally good representatives of the questions for both male and female students. A more desirable invariance assumption (scalar) was met fully for the Student Knowledges and Attitudes to Indigenous Issues measure but only partially for the Applied Indigenous Learning measure). Overall, as per Research Question 2, it can be assumed that the measures were understood equally well by male and female students.

Research Question 3: Student responses to the measures

With regard to the degree to which males and females may agree with the questions (as opposed to patterns of understanding), Table 5 revealed that for the Student Knowledges and Attitudes to Indigenous Issues measure, on average, both males and females agreed that Indigenous Australians suffer higher levels of disadvantage, that Indigenous Australians are more likely to be misrepresented by knowledge production agencies (females were significantly more likely to agree), that they would be confident to work with Indigenous Australians in the future, that they would be willing to adapt their future workplace practices to work with Indigenous Australians (females were significantly more likely to agree), that Indigenous and non-Indigenous peoples can work together (females were significantly more likely to agree), and that Indigenous Australians are a diverse group of peoples (males were significantly more likely to agree). In addition, both males and females were more likely to have disagreed with questions relating to Indigenous Australians being the cause of systemic disadvantage (females were significantly less likely to agree).

Table 5.

Total sample mean, standard deviations, and significance differences for the latent measures.

	Mean		SD		t-test
	Male	Female	Male	Female	Sig
Student Knowledges and Attitudes to Indigenous Issues
Disad	4.36	4.28	.91	.92	ns
Deficit	2.18	1.84	1.04	1.02	.007
Crit Rep	3.63	4.05	.79	.83	.000
Confi	5.35	5.46	.74	.75	ns
Adapt	4.94	5.39	.85	.68	.000
Two-ways	5.21	5.37	.72	.61	.045
Diversity	4.53	4.11	.73	.63	.007
Applied Indigenous Learning
Respect	4.67	4.87	1.07	1.02	ns
Author	2.59	2.77	1.52	1.37	ns
Represent	2.22	2.48	1.31	1.30	ns
Standpoint	2.78	3.26	1.53	1.51	.010
Assessment	2.64	2.51	1.93	1.58	ns
Value	2.86	3.45	1.45	1.44	.001

SD: Standard deviation; Disad: Disadvantage; Crit Rep: Critical Representations; Confi: Confidence; Adapt: Adaptability.

For the Applied Indigenous Learning measures, Table 8 revealed that overall, both males and females agreed to the university classroom being a respectful environment for addressing Indigenous issues, yet both males and females disagreed to experiencing engagement with Indigenous authors within their subject, having Indigenous representatives who taught within the subject, and having an assessment related to Indigenous issues. Overall, females agreed to engaging with Indigenous standpoints in their studies, and valuing the Indigenous learning content (where as males tended to significantly disagree with these questions).

Research Question 3 summary – How did the students respond (agree/disagree) to each of the measures?

For the Student Knowledges and Attitudes to Indigenous Issues measure, on average both males and females agreed to all questions (with the exception of Deficit Thinking). This suggested that student knowledges and attitudes towards Indigenous Australians are largely accurate or positively orientated. For the Applied Indigenous Learning factors, whilst students on average agreed to the classroom being a respectful space for discussing Indigenous issues, students mostly did not perceive much engagement with Indigenous studies within their discipline (although the standard deviations suggested that this may vary across disciplines). It should be noted though that female students (but not males), did, on average, reported to agreeing to engaging with Indigenous Standpoints, and Value Indigenous Learning within their discipline. Overall, for Research Question 3, for the most part male and female students held positive knowledges and attitudes towards Indigenous Australians, but there was more variation with regard to their experiences of Indigenous learning content within their disciplines.

Table 6.

Nested SEM for predictors of student Knowledges and Attitudes to Indigenous Issues factors.

	Disad	Deficit	Crit Rep	Confi	Adapt	Two-ways	Diversity
Model 1
Age	–.10	–.03	.06	–.01	–.00	.05	–.09
Gender	.0.2	.00	–.01	.11	.10	.02	.01
Born Overseas	–.17*	.16*	.03	–.08	–.01	–.02	–.35***
Years Study	.17*	–.26*	.39***	.03	.18*	.09	.23***
Finance	–.04	–.04	–.10	–.02	–.08	–.15*	.14
TOTAL R	7.4%	11.7%	18.3%	2.2%	4.8%	3.8%	21.9%
	χ² = 3014.54		df = 1448			Mean R² = 10.01%
Model 2
Age	–.01	–.02	.06	–.01	–.01	.05	–.09
Gender	–.09	.05	.07	.06	.04	.04	.00
Born Overseas	–.16*	.14*	.07	–.06	–.03	.01	–.33***
Years Study	.17^	–.20*	.28**	.04	.16	–.03	.10
Finance	–.03	–.04	–.09	–.03	–.10	–.16*	.13*
Respect	.05	–.08	–.03	.25***	.32***	.27***	.11^
Author	–.09	.13	–.04	.01	.05	–.04	–.09
Represent	–.17^	.10	–.06	–.15	–.07	–.19*	–.16
Standpoint	.11	–.22^	.11	.18	–.06	.14	–.01
Assessment	.12	–.04	–.24*	–.01	.01	–.21*	–.09
Value	.03	–.02	.09	–.13	.06	.09	.29**
TOTAL R	14.3%	21.5%	25.9%	15.6%	16.1%	19.8%	34.1%
	χ² = 2937.32		df = 1406	χ² dif = 77.22***		Mean R² = 21.04%

Note. Disad: Disadvantage; Crit Rep: Critical Representations; Confi: Confidence; Adapt: Adaptability. Born Overseas is coded as –1 = Australian and 1 = Overseas. * = p < .05, ** = p < .01, *** = p < .001, ^ = p < .10.

Table 7.

Mean, standard deviations, and difference for the IGA evaluation measures.

	Education (i)	Law (ii)	Design (iii)	Engineering (iv)	Health (v)	ANOVA SIG
Disad	4.33	4.68 (iii,iv)	4.21 (ii,v)	4.08 (ii,v)	4.82 (iii,iv)	.001
Deficit	1.99 (v)	1.62 (iv)	2.04 (v)	2.33 (ii,v)	1.19 (i,iii,v)	.001
Crit Rep	3.97 (v)	3.69 (v)	3.99 (iv,v)	3.61 (iii,v)	4.57 (i,ii,iii,iv)	.001
Confi	5.42	5.62	5.48	5.31	5.21	ns
Adapt	5.30	5.41 (iv)	5.23	4.97 (ii,v)	5.45 (iv)	.01
Two-ways	5.38	5.39	5.29	5.18	5.48	ns
Diversity	4.15 (iv)	4.31 (iii,iv)	3.98 (ii,v)	3.73 (i,ii,v)	4.43 (iii,iv)	.001
Respect	5.21 (iv)	5.35 (iii,iv)	4.76 (ii,iv)	4.34 (i,ii,iii)	4.88	.001
Author	3.40 (iv,v)	3.65 (iii,iv,v)	2.69 (ii,iv)	2.12 (i,ii,iii)	2.19 (i,ii)	.001
Represent	2.74 (iv)	2.37	2.57 (iv)	1.98 (i,iii)	2.58	.01
Standpoint	3.76 (iii,iv)	4.24 (iii,iv)	2.99 (i,ii,iv,v)	2.04 (i,ii,iii,v)	3.79 (iii,iv)	.001
Assess	2.87 (ii,iii,iv,v)	5.43 (i,iii,iv,v)	2.02 (i,ii)	1.83 (i,ii)	1.64 (i,ii)	.001
Value	3.67 (ii,ii,iv,v)	4.59 (i,iii,iv)	2.94 (i,ii,iv,v)	2.18 (i,ii,iii,v)	4.55 (i,iii,iv)	.001

(i) to (iv) indicate significant differences across disciplines within post-hoc difference tests. SD: Standard deviation; Disad: Disadvantage; Crit Rep: Critical Representations; Confi: Confidence; Adapt: Adaptability.

Table 8.

Figures of correlational analyses across disciplines.

DAB: Design, Architecture, and Building.

Research Question 4: Associations between Applied Indigenous Learning and Student Knowledges and Attitudes to Indigenous Issues

A partial two-stage SEM was conducted to assess the relations between the Applied Indigenous Learning and Student Knowledges and Attitudes to Indigenous Issues measures. Firstly (model 1), student background variables were used to independently predict the Student Knowledges and Attitudes to Indigenous Issues measures. The second stage of the SEM model then included the uncorrelated (to avoid multicollinearity) Applied Indigenous Learning measures to determine the extent to which each of these measures predicted Student Knowledges and Attitudes to Indigenous Issues over-and-above the effects of the student demographic variables.

Table 6 revealed that for model 1, a range of student demographic variables predicted the factors underlying Student Knowledges and Attitudes to Indigenous Issues (nine significant associations in total). One of the most consistent positive predictors (five of the nine significant predictors) was number of years studying at university, which was associated with higher levels of knowledge of Disadvantage (β = .17, p < .05), Critical Representation (β = .39, p < .001), Adaptability (β = .18, p < .05), and Diversity (β = .23, p < .001) and lower levels of Deficit thinking (β = –.26, p < .05). Being born overseas was positively associated with Deficit thinking (β = .16, p < .05) and negatively associated with knowledge of Disadvantage (β = –.17, p < .05) and knowledge of Diversity (β = –.35, p < .001). The only other significant predictor was a stronger rating of finance was associated with lower levels of believing Indigenous and non-Indigenous knowledges can work together (β = –.15, p < .05).

In model 2, the inclusion of Applied Indigenous Learning measures predicting the Student Knowledges and Attitudes to Indigenous Issues saw an increase of around 11% in the average variance explained estimates and significantly improved the model (χ² dif = 77.22, p < .001) . Over-and-above the effects of the student background variables, six significant relations were identified for the Applied Indigenous Learning measures. Students who perceived higher levels of Respectful Learning in the classroom were significantly more likely to report higher levels Confidence in working with Indigenous Australians (β = .25, p < .001), willingness to Adapt their future work place practices when working with Indigenous Australians (β = .32, p < .001), and belief that Indigenous and non-Indigenous Australian can work together (Two-ways – β = .27, p < .001).

Two of the most powerful associations stemmed from the Value of Indigenous Learning variable when predicting Indigenous Diversity (β = .29, p < .001) and knowledge of systemic Disadvantages faced by Indigenous Australians (β = .27, p < .01). Whilst most of the associations were associated with more positive student knowledges and attitudes, four negative associations were also identified (excluding Deficit Thinking). Having Indigenous Assessments was associated with a decreased level of cynicism directed at representations of Indigenous peoples and communities by media/politics/research (Crit Rep – β = –.24, p < .001), and a weaker belief that Indigenous and non-Indigenous peoples can work together (Two-ways – β = –.22, p < .001). Intriguingly, Two-ways was negatively associated with both having Indigenous Representatives in the classroom (β = –.19, p < .05), and having Indigenous Assessments within the discipline (β = –.21, p < .05).

Research Question 4 Summary – After controlling for student background variables, to what extent are the Applied Indigenous Learning and Student Knowledges and Attitudes to Indigenous Issues measures statistically associated with each other?

In examining the relations between student background variables, experiences of Applied Indigenous Learning, and Knowledges and Attitudes to Indigenous Australians, a wide variety of significant results were identified. With regard to student background variables, the most consistent significant predictor of these knowledges and attitudes was Years Study at university, suggesting that the longer students studied, the more accurate their knowledges and positive attitudes towards Indigenous Australians. Students Born Overseas were more likely to hold more negative attitudes though.

Of the Applied Indigenous Learning measures, perceiving Respectful Learning was associated with higher levels of confidence and adaptability for working with Indigenous Australians, and a stronger belief that Indigenous and non-Indigenous knowledges can work together (Two-ways). The more students were likely to perceive Indigenous Assessments in their discipline, the less likely they were to be critical of broader narratives about Indigenous Australians (Critical Representations), and the less likely they be in terms of believing that Indigenous and non-Indigenous Knowledges can work together.

Overall, these findings suggest that with regard to Research Question 4, student experiences of Indigenous learning were significantly associated with a number of student knowledge and attitudes outcome variables.

Research Question 5: Disciplinary variation across the measures

The final set of analyses was to investigate in more detail how the Applied Indigenous Learning and Student Knowledges and Attitudes to Indigenous Issues measures interact across the disciplines. Due to the smaller sample size across each discipline, latent variable analyses were not conducted (e.g., CFA, SEM), rather mean scores for each measure were analysed (through analysis of variance and linear regression techniques). Table 7 provides the mean scores for all measures across disciplines, and Table 8 contains the correlations between the Applied Indigenous Learning and Student Knowledge and Attitudes to Indigenous Learning measures for each discipline.

Table 7 reveals significant overall variation in the mean scores for nearly all of the IGA evaluation measures (with the exception of Confidence and Two-ways). Whilst the post-hoc analyses revealed a vast array of significant differences (p < .05) across the disciplines, it should be noted that the degree of difference did not transition across the mid-point of the Likert scale for any of the factors in the Student Knowledges and Attitudes to Indigenous Issues measure – indicating consistency in the use of the response scale across disciplines. The most substantial differences, though, could be noted for a number of the Applied Indigenous Learning factors, particularly for reports of Indigenous Authorship, Indigenous Standpoints, Indigenous Assessment, and Value of Indigenous Learning.

Table 8 contains pictorial summaries of the range or correlations between the Applied Indigenous Learning measure and Student Knowledges and Attitudes to Indigenous Issues measure as split by disciplines (see Appendix 2 for the Pearson’s correlations). Whilst discussion of the reporting of each individual correlation is beyond the scope of this article, the authors wish to emphasis the diverse nature of these correlations when split by disciplines (often noted by positive and negative correlations for the same associations). For example, correlations between Indigenous Representatives and Confidence to work with Indigenous Australians range from r = –.36 (p < .05) for Law and r = .53 (p < .05) for Design, Architecture, and Building.

Research Question 5 Summary – How do the measures operate differently across disciplines?:

There was substantial and significant variation in mean responses across disciplines, suggesting a wide diversity of Student Knowledges and Attitudes to Indigenous Issues as well as Applied Indigenous Learning experiences pending the discipline which they were studying. This diversity was also reflected in how Applied Indigenous Learning was associated with Student Knowledges and Attitudes to Indigenous Issues. In response to Research Question 5, it can be noted that the findings highlight not only a diversity of ways students may perceive and engage with Indigenous Studies being embedded within diverse disciplines, but also that the impact of embedding Indigenous studies may vary substantially across disciplines. In interpreting these findings, care needs to be taken as any university’s engagement with Indigenous Graduate Attribute can vary substantially within and across disciplines.

Discussion

This article presents a complex array of findings pertaining to the psychometric validation of an evaluation measure for the embedding of Indigenous Studies across disciplines as part of Indigenous Graduate Attribute initiatives. Thirteen factors pertaining to Student Knowledges and Attitudes to Indigenous Issues and Applied Indigenous Learning perceptions were found to hold strong psychometric properties, suggesting that the link between the measurement items and their designated factors were sound. Invariance tests found that the factor structure of the scale was consistent across a female and male student grouping. Across the male and female students, there was general consistency to students reporting positive responses to the Student Knowledges and Attitudes measure, but there was more variation to responses to the Applied Indigenous Learning measure (with students more likely to disagree with perceiving Indigenous Learning content). This variation became more apparent when examining the extent to which the Student Knowledges and Attitudes measure was associated with the Indigenous Learning measures (e.g., support for Two-ways learning being positively associated with a respectful classroom, yet negatively associated with students undertaking Indigenous assessments). This complexity was even further exacerbated when the statistical associations were split by disciplines, suggesting that the embedding of Indigenous studies potentially acts differently according to specific disciplines. Overall, these findings reveal the care that must be taken when embedding Indigenous studies across diverse disciplines, and that any attempt to generalise findings across disciplines – and levels of embeddedness – is problematic.

In committing to an Indigenous Quantitative Methodology (Walter & Andersen, 2013), Bodkin-Andrews et al. (2017) argued that in discussing statistical results centred on Indigenous issues, a more reflexive approach needs to be taken “as Indigenous-specific interpretive and epistemic relevancies are too often not adequately engaged with by subjugating positivistic, and Western frameworks” (p. 242). From this standpoint, it is essential that articles and their discussions engage with Indigenist research that prioritises Indigenous standpoints (Martin, 2008; Nakata, 2007; Rigney, 1999; Walter & Andersen, 2013).

As already discussed, the work of Bidjara scholar Anning (2010) highlights that not only do a minority of Australian universities commit to Indigenous Graduate Attributes (see also Frawley, 2017), but the realisation of all students graduating with some form of Indigenous professional and cultural competency moves well beyond simply teaching about Indigenous Australians, but ethically engaging with Indigenous Knowledges and critical Indigenous studies frameworks that promote awareness, sensitivity, respect, and competency for working with Aboriginal and Torres Strait Islander peoples and communities (Anning, 2010; Sherwood & Russell-Mundine, 2017; Virdun et al., 2013). Enthusiasm towards embracing a more culturally responsive and centred approach to Indigenous studies must be tempered with caution though, as many Indigenous (and non-Indigenous) scholars have noted that Indigenous studies as a discipline is continually in danger of repeating and perpetuating assimilative colonial narratives (Cross-Townsend, 2011; Judd, 2014).

Whilst there has been some positive findings emerging from studies directly assessing foundational subjects dedicated to Indigenous studies and their impact on predominately non-Indigenous student cohorts (Aberdeen et al., 2013; Jackson, Power, Sherwood, & Geia, 2013; Kickett, Hoffman & Flavell, 2014; McDermott & Sjoberg, 2012), it is critical to note that Indigenous studies is often embedded as partial subject matter within broader disciplinary focused subjects. For example, Gomeroi scholar Norman (2014) mapped Indigenous learning content across eight separate disciplines (within the same university as the authors of this article). The findings revealed a wide diversity in the rate of engagement with Indigenous learning content across undergraduate and post-graduate subjects (e.g., 0 in Faculty of Engineering and Information Technology to 73 in the Faculty of Arts and Social Sciences). In addition, there was a wide variety of teaching and learning methods through which Indigenous learning content was introduced, with the majority being set readings/resources and “relevant” lectures, but also including field trips, student placements, guest lectures, etc. These findings, when coupled with themes related to self-reported staff skillsets, linking studies to professional practices, and barriers to teaching Indigenous content, led to the conclusion that whilst there was considerable staff interest and passion for teaching Indigenous learning content, there was also “disinterest among the student body, anxiety among colleagues to confidently teach Indigenous material, limited scope to assess Indigenous related content, and shortage of quality resources and curriculum building guides” (Norman, 2014, p. 50). These findings, when coupled with the complex results emerging from this article, highlight the need for all universities to comprehensively and critically examine the nature and impact of Indigenous Learning content that may “adhere” to Indigenous Graduate Attributes and/or increased professional competencies (and related resources available) to work with Aboriginal and Torres Strait Islander peoples and communities.

As already stated, this need for a critical engagement in monitoring IGA-related progress (particularly concerning its overlap with the embedding of Indigenous Studies) is most evidenced by the correlation figures across disciplines situated in Table 8. Across nearly every Student Knowledges and Attitude to Indigenous Learning outcome, substantial variation (from significantly positive to significantly negative correlations) could be observed with regard to the potential impact of embedding of Applied Indigenous Learning experiences across disciplines. For example, substantive opposing correlations (±.60 correlational difference) could be observed for associations between Respectful Classroom learning with Critical Representation (education vs. law), Indigenous Authorship with Future Confidence (law vs. engineering), Indigenous Authorship with Critical Representations (design vs. engineering), Indigenous Representatives with Future Confidence (design vs. law), Indigenous Assessment with Deficit Thinking (design vs. education) and Critical Representations (education vs. law), and Two-ways (engineering and health), and Value of Indigenous Learning with Deficit Thinking (education vs. law). What this finding suggests is that a simple tick-a-box approach to embedding Indigenous Studies across disciplines is highly problematic, particularly when such approaches may be used to justify the success of any university’s IGA commitment.

Relating these findings back to the issue of IGAs, it should be noted that the primary selling point of any university’s graduate attribute is the employability of their graduating students (Page et al., 2018). It must be noted that Page and colleagues have previously warned that whilst many graduates may not work directly with Indigenous Australians and their communities, many may find themselves in positions that can impact upon the lives of Indigenous peoples and communities (e.g., policy, service delivery, commercial activities). As a result, any university seeking to commit to learning and teaching activities that may meaningfully realise Indigenous Graduate Attributes should commit to a strategic framework that spans across a whole degree. Page et al. (2018) suggested an IGA Degree Framework that builds on first-year students’ general foundations of knowledge of Indigenous content (thus normalising the learning of such content – e.g., learning of colonisation, racism, critical reflexivity, whiteness, decolonisation). Second-year subjects should then seek to embed and connect Indigenous content to specific disciplines, whilst third (and fourth)-year cap-stone subjects should seek to apply student knowledge of Indigenous content into practice (e.g., ranging from internships with Indigenous organisations to investigating online Reconciliation Action Plans of any organisation). In a more recent article, Page et al. (2019) further argue that any IGA-related framework should not solely rest on the shoulders of too often over-burdened teaching and learning academics, but must be supported by university-specific strategic plans (e.g., Indigenous employment and education strategies), high-level governance committees (that include senior Indigenous staff), as well as being present and made visible within key teaching and learning committees across faculties. Page et al. (2019, p. 11) conclude by stressing the importance of a strong Indigenous presence across all of their strategic initiatives as, “without Indigenous representation in these bastions of institutional authority, Indigenous outcomes are relegated to the shadows, where too often intended activities are under-resourced and consequently fail to thrive on a larger institutional scale.” This key critique by Page et al. highlights the need for universities to more strongly commit to Indigenous governance and Indigenous Data Sovereignty frameworks to hold both policy and teaching and learning practices accountable.

Based on the above reasoning though, this article only provides a small fragment of potential accountability that universities should commit to if they seek to pursue any IGA-based framework (namely, student perceptions as opposed to a systemic critique of ongoing Eurocentric university practices), and that is a key limitation to this article. Numerous other limitations are also evident, including that the measurement instruments are self-report in nature and focus only on student perceptions. It would be interesting to note the degree to which student perceptions align with reports from tutorial staff and unit-coordinators (and even independent unit-outline evaluations). Related to the survey was the fact that the administration was cross-sectional in nature (that is at the end of specified subjects), thus limiting confidence towards any causal inferences (e.g., perceptions of Indigenous Learning content predicting Student Attitudes over time). Additionally, the small sample size and inclusion of only one subject within each of the six disciplines is problematic (and the subject self-selection of unit-coordinators – a requirement of university ethics) as there was no guarantee that the students’ reporting of Indigenous Learning content was representative of the disciplinary engagement within Indigenous Studies. As a result, student perceptions should be triangulated with not only tutor and unit-coordinator perceptions, but actual course content (e.g., unit outlines, required readings), and as alluded to above, faculty and university-wide policy, practice, and support.

Despite these limitations, this article seeks to provide scholars, teachers, and universities with the opportunity to, in part, engage more meaningfully with discovering the implications of embedding Indigenous studies within and across disciplines. It must be stressed though that simply attempting to redress non-Indigenous student perceptions of Indigenous learning content does not guarantee the realisation of any Indigenous Graduate Attribute. As argued by Rigney (2017, p. 60), “there is no simple formula for successful university-wide change. Strong intervention strategies require a suite of multi-faceted responses to particular needs of different institution groups.” This includes university commitment to Indigenous staff capacity development, cultural training for all staff, meaningful resourcing and funding for staff to respectfully engage and collaborate with Aboriginal and Torres Strait Islander scholars, community representatives and organisations, providing a visible, supportive, and safe environment for future Indigenous scholars (and students), and commitment to (and accountability for) Indigenous leadership across all disciplines and levels of university employment and governance (Behrendt et al., 2012; Frawley, 2017; Lester, 2017; Page et al., 2016).

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Note

References

Aberdeen, L., Carter, J., Grogan, J., & Hollinsworth, D. (2013). Rocking the foundations: the struggle for effective Indigenous Studies in Australian higher education. Higher Education Review, 45(3), 36--55.

Andersen, C., & Kukutai, T. (2016). Reclaiming the Statistical Native: Quantitative historical research beyond the pale. In C. Andersen, & J. M. O'Brien (Eds.), Sources and Methods in Indigenous Studies (pp. 41--48). London, UK and New York, USA: Routledge.

Anning

(2010). Embedding an indigenous graduate attribute into University of Western Sydney’s courses. The Australian Journal of Indigenous Education, 39(S1), 40–52.

Australian Institute of Aboriginal and Torres Strait Islander Studies. (2012). Guidelines for ethical research in Australian indigenous studies. Acton, ACT: Author.

Behrendt, L. (2016). Finding Eliza: Power and colonial storytelling. St Lucia, Australia: University of Queensland Press.

Behrendt

L. Y.

Larkin

Griew

Kelly

(2012). Review of higher education access and outcomes for Aboriginal and Torres Strait Islander people. Canberra, Australia: Commonwealth of Australia.

Bodkin-Andrews

Whittaker

Harrison

Craven

Parker

Trudgett

Page

(2017). Exposing the patterns of statistical blindness: Centring indigenous standpoints on student identity, motivation, and future aspirations. Australian Journal of Education, 61(3), 225–249.

Bodkin-Andrews, G., & Carlson, B. (2013). Racism, Aboriginal and Torres Strait Islander Identities, and higher education: reviewing the burden of epistemological and other racisms. In Seeding success in Indigenous Australian Higher Education (pp. 29--54). Bingley, United Kingdom: Emerald Group Publishing Limited.

Bodkin-Andrews, G., Page, S., & Trudgett, M. (2018). Shaming the silences: Indigenous Graduate Attributes and the privileging of Aboriginal and Torres Strait Islander voices. Critical Studies in Education, 1--18. Published online 25th November, 2018.

10.

Bradley

Noonan

Nugent

Scales

(2008). Review of Australian higher education (final report). Canberra, Australia: Commonwealth of Australia.

11.

Byrne

B. M.

(2012). Structural equation modelling with Mplus: Basic concepts, applications, and programming. New York, NY: Routledge.

12.

Cross-Townsend

(2011). Indigenous education and Indigenous studies in the Australian Academy: Assimilationism, critical pedagogy, dominant culture learners, and indigenous knowledges. Counterpoints, 379, 68–79.

13.

Davis

(2016). Data and the United Nations declaration on the rights of indigenous peoples. In T. Kukutai & J. Taylor (Eds.), Indigenous data sovereignty: Toward an agenda (pp. 25–38). Acton, ACT: Anu Press.

14.

Flavell

Thackrah

Hoffman

(2013). Developing indigenous Australian cultural competence: A model for implementing Indigenous content into curricula. Journal of Teaching and Learning for Graduate Employability, 4(1), 39.

15.

Foley

(2003). Indigenous epistemology and Indigenous standpoint theory. Social Alternatives, 22(1), 44.

16.

Frawley, J. (2017). Indigenous knowledges, graduate attributes and recognition of prior learning for advanced standing: Tensions within the academy. In J. Frawley, S. Larkin, & J. Smith (Eds), Indigenous pathways, transitions and participation in higher education (pp. 65--80). Singapore: Springer.

17.

Hogarth

(2017). Speaking back to the deficit discourses: A theoretical and methodological approach. The Australian Educational Researcher, 44(1), 21–34.

18.

Jackson, D., Power, T., Sherwood, J., & Geia, L. (2013). Amazingly resilient Indigenous people! Using transformative learning to facilitate positive student engagement with sensitive material. Contemporary Nurse, 46(1), 105--112.

19.

Judd, B. (2014). From Paris to Papunya: Postcolonial theory, Australian Indigenous studies and ‘knowing the Aborigine’. In V. Castejon, A. Cole, O. Haag, & Hughes K (Eds), Ngapartji Ngapartji: In turn, in turn: Ego-histoire, Europe and Indigenous Australia (pp. 143--158). Canberra, Australia: ANU Press.

20.

Kickett

Hoffman

Flavell

(2014). A model for large-scale, interprofessional, compulsory cross-cultural education with an indigenous focus. Journal of Allied Health, 43(1), 38–44.

21.

Kukutai

Taylor

(Eds.). (2016). Indigenous data sovereignty: Toward an agenda. Acton: ACT: Anu Press.

22.

Marsh

H. W.

Tracey

D. K.

Craven

R. G.

(2006). Multidimensional self-concept structure for preadolescents with mild intellectual disabilities: A hybrid multigroup–MIMC approach to factorial invariance and latent mean differences. Educational and Psychological Measurement, 66(5), 795–818.

23.

Martin

K. L.

(2008). Please knock before you enter: Aboriginal regulation of outsiders and the implications for researchers. Brisbane, Australia: Post Pressed.

24.

McDermott

Sjoberg

(2012). Managing a diverse student discomfort with an Indigenous health curriculum. LIME Good Practice Case Studies, 1, 26.

25.

Moreton-Robinson

(2015). The white possessive: Property, power, and indigenous sovereignty. Minneapolis: University of Minnesota Press.

26.

Moodie

(2019). Learning about knowledge: Threshold concepts for Indigenous studies in education. The Australian Educational Researcher, 1–15.

27.

Moodie

Ewen

McLeod

Platania-Phung

(2018). Indigenous graduate research students in Australia: A critical review of the research. Higher Education Research & Development, 37, 805–2.

28.

Nakata

(2013). The rights and blights of the politics in Indigenous higher education. Anthropological Forum, 23(3), 289–303.

29.

Nakata

M. N.

(2007). Disciplining the savages, savaging the disciplines. Canberra, Australia: Aboriginal Studies Press.

30.

Norman

(2014). Mapping more than Aboriginal studies: Pedagogy, professional practice and knowledge. The Australian Journal of Indigenous Education, 43(1), 42–51.

31.

Page, S., Trudgett, M., & Bodkin-Andrews, G. (2019). Tactics or strategies? Exploring everyday conditions to facilitate implementation of an Indigenous graduate attributes project. Journal of Higher Education Policy and Management, 1--14. Published online 27th April, 2019.

32.

Page

Trudgett

Bodkin-Andrews

(2018). Creating a degree-focused pedagogical framework to guide Indigenous graduate attribute curriculum development. Higher Education, 1–15. Published online October, 2018.

33.

Page

Trudgett

Bodkin-Andrews

(2016, July). Exploring an Indigenous graduate attribute project through a critical race theory lens. In M. Davis & A. Goody (Eds.), Research and development in higher education: The shape of higher education (Vol. 39, pp. 258–267). Hammondville, Australia: Higher Education Research and Development Society of Australasia.

34.

Parker

P. D.

Martin

A. J.

Martinez

Marsh

H. W.

Jackson

S. A.

(2010). Stages of change in physical activity: A validation study in late adolescence. Health Education and Behavior, 37(3), 318–329.

35.

Rainie

S. C.

Schultz

J. L.

Briggs

Riggs

Palmanteer-Holder

N. L.

(2017). Data as a strategic resource: Self-determination, governance, and the data challenge for indigenous nations in the United States. The International Indigenous Policy Journal, 8(2), 1–29.

36.

Rigney, L. I. (1999). Internationalization of an Indigenous anticolonial cultural critique of research methodologies: A guide to Indigenist research methodology and its principles. Wicazo sa review, 14(2), 109--121.

37.

Rigney, L. I. (2017). A design and evaluation framework for Indigenisation of Australian universities. In J. Frawley, S. Larkin, & J. Smith (Eds), Indigenous pathways, transitions and participation in higher education (pp. 45--63). Singapore: Springer.

38.

Shay

(2016). Seeking new paradigms in Aboriginal education research: Methodological opportunities, challenges and aspirations. Social and Education History, 5(3), 273–296.

39.

Sherwood, J., & Russell-Mundine, G. (2017). How we do business: Setting the agenda for cultural competence at the University of Sydney. In J. Frawley, S. Larkin, & J. Smith (Eds), Indigenous pathways, transitions and participation in higher education (pp. 133--150). Singapore: Springer.

40.

Sherwood

McDaniel

McKenzie

(2013). Indigenous Graduate Attribute (IGA) discussion paper (Academic Senate Meeting 13/5). Sydney: University of Technology Sydney.

41.

Smallwood

(2015). Indigenist critical realism: Human rights and first Australians’ wellbeing. London, England: Routledge.

42.

Smith

L. T.

(2012). Decolonizing methodologies: Research and indigenous peoples. London, England: Zed Books Ltd.

43.

Snipp

C. M.

(2016). What does data sovereignty imply: What does it look like? In T. Kukutai & J. Taylor (Eds.), Indigenous data sovereignty: Toward an agenda (pp. 39–56). Acton, ACT: Anu Press.

44.

Tallbear

(2016). Dear indigenous studies, it’s not me, it’s you: Why I left and what needs to change. In A. Moreton-Robinson (Ed.), Critical indigenous studies: Engagement in First World locations (pp. 69–82). Tucson: University Arizona Press.

45.

United Nations General Assembly (2007). United Nations Declaration on the Rights of Indigenous Peoples. Adopted by the General Assembly, 2 October 2007, A/RES/61/295. Downloaded from https://www.refworld.org/docid/471355a82.html (accessed 13 July 2019).

46.

Universities Australia. (2017). Indigenous strategy 2017–2020. Canberra, Australia: Universities Australia.

47.

Universities Australia. (2011). National best practice framework for indigenous cultural competency in Australian universities. Canberra, Australia: Universities Australia.

48.

University of Technology Sydney [UTS] (2017). Indigenous Education and Employment Policy. Sydney, Australian: UTS. http://www.gsu.uts.edu.au/policies/documents/indigenous-education-employment-policy.pdf

49.

Virdun

Gray

Sherwood

Power

Phillips

Parker

Jackson

(2013). Working together to make Indigenous health care curricula everybody’s business: A graduate attribute teaching innovation report. Contemporary Nurse, 46(1), 97–104.

50.

Walter, M. (2016). Data politics and Indigenous representation in Australian statistics. In T. Kukutai & J. Taylor (Eds.), Indigenous data sovereignty: Toward an agenda (pp. 79--98). Acton, ACT: Anu Press.

51.

Walter

(2018). The voice of Indigenous data: Beyond the markers of disadvantage. Griffith Review, 60, 256–263.

52.

Walter

Andersen

(2013). Indigenous statistics: A quantitative research methodology. Walnut Creek, CA: Left Coast Press.