For more than 10 years, a program called Families And Schools Together (FAST) has been run in schools across the Northern Territory. These programs have always had an evaluation component built in. However, over time, the evaluations have changed. Initially, they were summative, built around a quantitative psychometric tool (with a positivist research paradigm). The intent of the summative evaluation was to demonstrate the effectiveness of the program. However, as the program was rolled out in remote contexts, the need for adaptation was recognised. Changes were made, but it was soon recognised that other outcomes, not captured or explored in the methodology, were emerging. After six years of working together, the evaluator and the program manager felt that it was time to explore effectiveness in a different way with a more qualitative evaluation process (based on naturalistic and pragmatic paradigms).
This article explores the six-year learning journey through evaluation that the evaluator and program manager have undertaken. It describes tensions between the need for reliable and generalisable objective quantitative data and the need for authentic and credible data based on participant experience.
AIATSIS2012, Guidelines for ethical research in Australian Indigenous studies, 2nd edn, rev. AIATSIS, Australian Institute of Aboriginal and Torres Strait Islander Studies, Canberra, viewed November 2012, http://www.aiatsis.gov.au/_files/research/GERAIS.pdf.
3.
BainbridgeR., WhitesideM., & McCalmanJ.2013, ‘Being, knowing, and doing: a phronetic approach to constructing grounded theory with Aboriginal Australian partners’, Qualitative Health Research, vol. 23, no. 2, pp. 275–288.
4.
BishopR.2011, Freeing ourselves, Sense Publishers, Rotterdam.
5.
BurtonP., GoodladR., & CroftJ.2006, ‘How would we know what works? Context and complexity in the evaluation of community involvement’, Evaluation, vol. 12, no. 3, pp. 294–312.
6.
CameronA., SalisburyC., LartR., StewartK., PeckhamS., CalnanM., PurdyS., & ThorpH.2011, ‘Policy makers’ perceptions on the use of evidence from evaluations’, Evidence and Policy: a Journal of Research, Debate and Practice, vol. 7, no. 4, pp. 429–447.
7.
CaracelliV.J.2006, ‘The evaluator-observer comes to the table: a moment of consequential decision making’, American Journal of Evaluation, vol. 27, no. 1, pp. 104–107.
8.
ChilisaB.2012, Indigenous research methodologies, Sage, Los Angeles.
9.
ChristieM.2013, ‘Generative and “ground-up” research in Aboriginal Australia’, Learning Communities: International Journal of Learning in Social Contexts, no. 13, May, pp. 3–12.
10.
Conley-TylerM.2005, ‘A fundamental choice: internal or external evaluation?’, Evaluation Journal of Australasia, vol. 5, nos. 1 and 2, pp. 3–11.
11.
DenzinN.K., & LincolnY.S.2008, ‘Introduction: critical methodologies and indigenous inquiry’, in DenzinN.K., LincolnY.S., & SmithL.T. (eds), Handbook of critical and indigenous methodologies, Sage, Thousand Oaks, California.
12.
DowneJ., MartinS., & BovairdT.2012, ‘Learning from complex policy evaluations’, Policy and Politics, vol. 40, no. 4, pp. 505–523.
13.
FosterD., WilliamsR., CampbellD., DavisV., & PepperillL.2006, ‘“Researching ourselves back to life”: new ways of conducting Aboriginal alcohol research’, Drug and Alcohol Review, vol. 25, no. 3, pp. 213–217.
14.
GillS.2010, Developing a learning culture in nonprofit organizations, Sage, Thousand Oaks, California.
GuentherJ., & BoonstraM.2009, ‘Adapting evaluation materials for remote Indigenous communities and low-literacy participants’, paper presented at the 12th Australasian Conference on Child Abuse and Neglect, Perth, 15-18 November 2009, viewed January 2010, http://www.catconatus.com.au/docs/091117_APCCAN_FAST.pdf.
19.
GuentherJ., & FalkI.2007, ‘The roles of the evaluator as objective observer and active participant. Are they mutually exclusive?’, paper presented at the AES International Conference, Melbourne, 3-7 September.
20.
GuentherJ., OsborneS., ArnottA., McRae-WilliamsE., & DisbrayS.2014, ‘Amplifying the voice of remote Aboriginal and Torres Strait Islander VET stakeholders using research methodologies: volume 1’, paper presented at the AVETRA 17th International Conference, Surfers Paradise, 22-24 April 2013, viewed June 2014, http://avetra.org.au/wp-content/uploads/2014/05/Abstract-33.pdf.
21.
KoernerC., TurS., & WilsonC.2009, ‘Bringing Indigenous sovereignties into community partnerships: unpacking racialisation to re-engage Indigenous Australian students in education’, in BairdB, & RiggsD (eds.), The racial politics of bodies, nations, and knowledges, Cambridge Scholars Publishing, Newcastle upon Tyne, England.
22.
LiamputtongP.2010, Performing qualitative cross-cultural research, Cambridge University Press, Cambridge.
MarkiewiczA.2008, ‘The political context of evaluation: what does this mean for independence and objectivity?’, Evaluation Journal of Australasia, vol. 8, no. 2, pp. 35–41.
25.
MartinK.2003, ‘Ways of knowing, ways of being and ways of doing: a theoretical framework and methods for Indigenous re-search and Indigenist research’, Journal of Australian Studies, vol. 27, no. 76, pp. 203–214.
26.
MartinK.2008, Please knock before you enter: Aboriginal regulation of outsiders and the implications for researchers, Post Pressed, Brisbane.
27.
MertensD.2009, Transformative research and evaluation, Guilford Press, New York.
28.
Moreton-RobinsonA., & WalterM.2009, ‘Indigenous methodologies in social research’, in WalterM (ed.), Social research methods, 2nd edn, Oxford University Press, South Melbourne, pp. 1–18.
29.
NakataM.2007, ‘The cultural interface’, Australian Journal of Indigenous Education, vol. 36S, supplement, pp. 7–14.
30.
NakataM., NakataV., KeechS., & BoltR.2012, ‘Decolonial goals and pedagogies for Indigenous studies’, Decolonization: Indigeneity, Education and Society, vol. 1, no. 1, pp. 120–140.
31.
NichollsR.2009, ‘Research and Indigenous participation: critical reflexive methods’, International Journal of Social Research Methodology, vol. 12, no. 2, pp. 117–126.
32.
OsborneS.2014, ‘Putuna kulini: the trouble with “hearing”’, AlterNative: an International Journal of Indigenous Peoples, vol. 10, no. 1, pp. 3–14.
RigneyL-I1999, ‘Internationalization of an Indigenous anticolonial cultural critique of research methodologies: a guide to Indigenist research methodology and its principles’, Wicazo Sa Review, vol. 14, no. 2, pp. 109–121.
37.
RogersP.J.2008, ‘Using programme theory to evaluate complicated and complex aspects of interventions’, Evaluation, vol. 14, no. 1, pp. 29–48.
38.
ScougallJ.2006, ‘Reconciling tension between principles and practice in Indigenous evaluation’, Evaluation Journal of Australasia, vol. 6, no. 2, pp. 49–55.
39.
SmithL.2012, Decolonizing methodologies: research and Indigenous peoples, 2nd edn, Zed Books, London.
40.
SullivanH.2011, ‘“Truth” junkies: using evaluation in UK public policy’, Policy and Politics, vol. 39, no. 4, pp. 499–512.
41.
TurS., BlanchF., & WilsonC.2010, ‘Developing a collaborative approach to standpoint in Indigenous Australian research’, The Australian Journal of Indigenous Education, vol. 39S, supplement, pp. 58–67.
42.
WilliamsE.2012, ‘Two-way accountability: improving ethical evaluation practice in Indigenous contexts’, paper presented at the AIC Occasional Seminar Series, Griffith, 19 April 2012, viewed July 2014, http://www.aic.gov.au/media_library/seminars/2012/emma_williams.pdf.
43.
WilliamsE., GuentherJ., & ArnottA.2011, ‘Beyond informed consent: how is it possible to ethically evaluate Indigenous programs?’, paper presented at the NARU Public Seminar Series, Darwin, 23 November 2011, viewed October 2013, http://covaluator.net/docs/S1.3_beyond_informed_consent.pdf.
44.
YangH., & ShenJ.2006, ‘When is an external evaluator no longer external? Reflections on some ethical issues’, American Journal of Evaluation, vol. 27, no. 3, pp. 378–382.
45.
YunkaportaT., & McGintyS.2009, ‘Reclaiming Aboriginal knowledge at the cultural interface’, Australian Educational Researcher, vol. 36, no. 2, pp. 55–72.
