This article charts the innovative evaluation journey an Australian nonprofit organisation, The Pyjama Foundation (PJF), has taken when designing an evaluation instrument to gain feedback from programme beneficiaries. PJF sought to develop a formal, targeted approach to hear the perspectives of children living in out-of-home care, who are involved with their Love of Learning educational programme. The design process included two focus group discussions with foster carers, programme volunteers, and child development experts. From this, an evaluation survey for children to use was developed. The survey’s underpinning conceptual framework, based on key protective factors influencing educational outcomes for children in out-of-home care, is a key contribution of this research. In addition, the design and implementation issues PJF encountered contribute insights for other nonprofit organisations and evaluators and academic knowledge towards evaluations involving children and vulnerable stakeholders. Hearing children’s views on programmes they are involved in is vital in helping to develop safe spaces for children to engage, where their thoughts are valued and opinions matter. As such, the processes detailed within this article support the development of evaluation practices that value children’s voices.
AlmqvistA.-L.AlmqvistL. (2015). Making oneself heard – children’s experiences of empowerment in Swedish preschools. Early Child Development and Care, 185(4), 578–593. https://doi.org/10.1080/03004430.2014.940931
2.
AndermanE. M.UrdanT.RoeserR. (2003). The patterns of adaptive learning survey: History, development, and psychometric properties. Paper prepared for the Indicators of Positive Development Conference, Washington, DC.
3.
AppletonJ. J.ChristensonS. L.KimD.ReschlyA. L. (2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology, 44(5), 427–445. https://doi.org/10.1016/j.jsp.2006.04.002
AzunreG. A.SowrirajanT. M. (2020). The mosaic approach as a tool to facilitate participatory planning with children: Insights from Milan, Italy. Education, 3(13), 1–16. https://doi.org/10.1080/03004279.2020.1739727
6.
BanduraA.BarbaranelliC.CapraraG. V.PastorelliC. (2001). Self-efficacy beliefs as shapers of children’s aspirations and career trajectories. Child Development, 72(1), 187–206. https://doi.org/10.1111/1467-8624.00273
7.
BaurV. E.AbmaT. A.WiddershovenG. A. M. (2010). Participation of marginalized groups in evaluation: Mission impossible?Evaluation and Program Planning, 33(3), 238–245. https://doi.org/10.1016/j.evalprogplan.2009.09.002
8.
BotsoglouK.BeazidouE.KougioumtzidouE.VlachouM. (2019). Listening to children: Using the ECERS-R and Mosaic approach to improve learning environments: A case study. Early Child Development and Care, 189(4), 635–649. https://doi.org/10.1080/03004430.2017.1337006
9.
BrandonP. R.NewtonB. J.HarmanJ. W. (1993). Enhancing validity through beneficiaries’ equitable involvement in identifying and prioritizing homeless children’s educational problems. Evaluation and Program Planning, 16(4), 287–293. https://doi.org/10.1016/0149-7189(93)90041-6
10.
CarmanJ. G.FredericksK. A. (2008). Nonprofits and evaluation: Empirical evidence from the field. New Directions for Evaluation, 2008(119), 51–71. https://doi.org/10.1002/ev.268
11.
CepukieneV.PakrosnisR.UlinskaiteG. (2018). Outcome of the solution-focused self-efficacy enhancement group intervention for adolescents in foster care setting. Children and Youth Services Review, 88, 81–87. https://doi.org/10.1016/j.childyouth.2018.03.004
12.
ChapmanM. V.WallA.BarthR. P. (2004). Children’s voices: The perceptions of children in foster care. American Journal of Orthopsychiatry, 74(3), 293–304. https://doi.org/10.1037/0002-9432.74.3.293
13.
ComanW.DevaneyJ. (2011). Reflecting on outcomes for looked-after children: An ecological perspective. Child Care in Practice, 17(1), 37–53. https://doi.org/10.1080/13575279.2010.522976
14.
CooperS. (2014). Transformative evaluation: Organisational learning through participative practice. The Learning Organization, 21(2), 146–157. https://doi.org/10.1108/TLO-03-2013-0003
15.
CooperS. (2017). Participatory evaluation in youth and community work: Theory and practice (1st ed.). Taylor & Francis.
16.
CunhaF.HeckmanJ. J.SchennachS. M. (2010). Estimating the technology of cognitive and noncognitive skill formation. Econometrica, 78(3), 883–931. https://doi.org/10.3982/ECTA6551
17.
DanseyD.ShberoD.JohnM. (2019). Keeping secrets: How children in foster care manage stigma. Adoption & Fostering, 43(1), 35–45. https://doi.org/10.1177/0308575918823436
18.
DhananiA.ConnollyC. (2012). Discharging not-for-profit accountability: UK charities and public discourse. Accounting, Auditing & Accountability Journal, 25(7), 1140–1169. https://doi.org/10.1108/09513571211263220
EvansD. (2020). Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation. Evaluation and Program Planning, 80, 101800. https://doi.org/10.1016/j.evalprogplan.2020.101800
21.
Evans-LockeK.HsuC. I. (2020). Using participatory methods to evaluate the impacts of an early intervention programme on children and young people (CYP). Evaluation Journal of Australasia, 20(3), 176–188. https://doi-org.ezp01.library.qut.edu.au/10.1177/1035719X20944020
22.
FattoreT.MasonJ.WatsonE. (2009). When children are asked about their well-being: Towards a framework for guiding policy. Child Indicators Research, 2(1), 57–77. https://doi.org/10.1007/s12187-008-9025-3
23.
FredricksJ. A.BlumenfeldP.FriedelJ.ParisA. (2005). School engagement. In MooreK. A.LippmanL. H. (Eds.), The Search Institute series on developmentally attentive community and society: What do children need to flourish: Conceptualizing and measuring indicators of positive development (pp. 305–321). Springer.
GillinghamP. (2018). Evaluation of practice frameworks for social work with children and families: Exploring the challenges. Journal of Public Child Welfare, 12(2), 190–203. https://doi.org/10.1080/15548732.2017.1392391
26.
GoldfarbD.TashjianS.GoodmanG.Bederian-GardnerD.HobbsS.CordónI.OgleC.BakanoskyR.ChaeY., & NYTD/CYTD Research Group. (2019). After child maltreatment: The importance of voice for youth in foster care. Journal of interpersonal violence. Advance online publication. https://doi.org/10.1177/0886260519825884
27.
GreeneJ. C. (1986, April). Participatory evaluation and the evaluation of social programs: Lessons learned from the field [Paper presentation]. Symposium on Participatory Evaluation at the Annual Meeting of the American Educational Research Association, San Francisco, CA, United States. https://eric.ed.gov/?id=ED291766
28.
GreenfieldC. (2011). Personal reflection on research process and tools: Effectiveness, highlights and challenges in using the Mosaic approach. Australasian Journal of Early Childhood, 36(3), 109–116. https://doi.org/10.1177/183693911103600314
29.
HarrisP.ManatakisH. (2013). Young children’s voices about their local communities. Australasian Journal of Early Childhood, 38(3), 68–76. https://doi.org/10.1177/183693911303800309
HouseE. R.HoweK. R. (2000). Deliberative democratic evaluation. New Directions for Evaluation, 2000(85), 3–12. https://doi.org/10.1002/ev.1157
32.
HreinsdottirA. M.DavidsdottirS. (2012). Deliberative democratic evaluation in preschools. Scandinavian Journal of Educational Research, 56(5), 519–537. https://doi.org/10.1080/00313831.2011.599426
33.
HydeJ.KammererN. (2009). Adolescents’ perspectives on placement moves and congregate settings: Complex and cumulative instabilities in out-of-home care. Children and Youth Services Review, 31(2), 265–273. https://doi.org/10.1016/j.childyouth.2008.07.019
34.
JacobsA.WilfordR. (2010). Listen first: A pilot system for managing downward accountability in NGOs. Development in Practice, 20(7), 797–811. https://doi.org/10.1080/09614524.2010.508113
35.
JacobsenH.MoeV.IvarssonT.Wentzel-LarsenT.SmithL. (2013). Cognitive development and social-emotional functioning in young foster children: A follow-up study from 2 to 3 years of age. Child Psychiatry & Human Development, 44(5), 666–677. https://doi.org/10.1007/s10578-013-0360-3
36.
JamesA. (2007). Giving voice to children’s voices: Practices and problems, pitfalls and potentials. American Anthropologist, 109(2), 261–272. https://doi.org/10.1525/aa.2007.109.2.261
37.
JonesR.Everson-HockE.PapaioannouD.GuillaumeL.GoyderE.ChilcottJ.CookeJ.PayneN.DuenasA.SheppardL. (2011). Factors associated with outcomes for looked-after children and young people: A correlates review of the literature. Child: Care, Health and Development, 37(5), 613–622. https://doi.org/10.1016/j.worlddev.2005.11.009
38.
KingstonK. L. (2016). Development of a child evaluation framework for early childhood services using deliberative democratic evaluation and the mosaic approach. Evaluation Journal of Australasia, 16(3), 25–34. https://doi.org/doi.org/10.1177/1035719X1601600304
39.
KnightR.RossiS. (2018). Children in out-of-home care and their educational outcomes: A literature review. September 2018. The Australian Centre for Philanthropy and Nonprofit Studies. https://eprints.qut.edu.au/122389/
LohmeierJ. H.LeeS. W. (2011). A school connectedness scale for use with adolescents. Educational Research and Evaluation, 17(2), 85–95. https://doi.org/10.1080/13803611.2011.597108
MendesP.MichellD.WilsonJ. Z. (2014). Young people transitioning from out-of-home care and access to higher education: A critical review of the literature. Children Australia, 39(4), 243–252. https://doi.org/10.1017/cha.2014.25
MişcaG. (2010). Challenges in evaluation of intervention programmes with children: Reflections from a research(er) perspective. Revista de Asistenţă Socială, 4, 41–47. http://search.proquest.com/docview/860232917/
46.
MitchellG. E.BerlanD. (2016). Evaluation and evaluative rigor in the nonprofit sector. Nonprofit Management and Leadership, 27(2), 237–250. https://doi.org/10.1002/nml.21236
NealD. (2017). Academic resilience and caring adults: The experiences of former foster youth. Children and Youth Services Review, 79, 242–248. http://doi.org/10.1016/j.childyouth.2017.06.005
49.
NewcomerB. (2016). Forging a strategic and comprehensive approach to evaluation within public and nonprofit organizations: Integrating measurement and analytics within evaluation. The American Journal of Evaluation, 37(1), 80–99. https://doi.org/10.1177/1098214014567144
50.
NorrisA. (2012). Interactive performance and focus groups with adolescents: The power of play. Research in Nursing & Health, 35(6), 671–679. https://doi.org/10.1002/nur.21509
51.
PurdueS.PetersonH.DengC. (2018). The case for greater youth participation in monitoring and evaluation in international development. Evaluation Journal of Australasia, 18(4), 206–221. https://doi.org/10.1177/1035719X18804401
ReevesJ.Le MareL. (2017). Supporting teachers in relational pedagogy and social emotional education: A qualitative exploration. International Journal of Emotional Education, 9(1), 85–98. http://search.proquest.com/docview/1890238225/
55.
Salahu-DinS. N.BollmanS. R. (1994). Identity development and self-esteem of young adolescents in foster care. Child and Adolescent Social Work Journal, 11(2), 123–135. https://doi.org/10.1007/BF01875771
56.
SalazarA. M.KellerT. E.GowenL. K.CourtneyM. E. (2013). Trauma exposure and PTSD among older adolescents in foster care. Social Psychiatry and Psychiatric Epidemiology, 48(4), 545–551. https://doi.org/10.1007/s00127-012-0563-0
57.
SchmitzH. P.RaggoP.Bruno-van VijfeijkenT. (2012). Accountability of transnational NGOs: Aspirations vs. practice. Nonprofit and Voluntary Sector Quarterly, 41(6), 1175–1194. https://doi.org/10.1177/0899764011431165
58.
SchwarzerR.JerusalemM. (1995). Generalized self-efficacy scale. In WeinmanJ.WrightS.JohnstonM. (Eds.), Measures in health psychology: A user’s portfolio. Causal and control beliefs (pp. 35–37). NFER-NELSON.
59.
SellersD. E.SmithE. G.IzzoC. V.McCabeL. A.NunnoM. A. (2020). Child feelings of safety in residential care: The supporting role of adult-child relationships. Residential Treatment for Children & Youth, 37(2), 136–155. https://doi.org/10.1080/0886571X.2020.1712576
60.
SkillingL. (2018). Educational outcomes and obstacles for children and youth in foster care in Hawai‘i [Doctoral dissertation]. University of Hawaiʻi at Mānoa.
61.
SmithA.DuncanJ.MarshallK. (2005). Children’s perspectives on their learning: Exploring methods. Early Child Development and Care, 175(6), 473–487. https://doi.org/10.1080/03004430500131270
62.
SnyderC. R.HozaB.PelhamW. E.RapoffM.WareL.DanovskyM.HighbergerL.RibinsteinH.StahlK. J. (1997). The development and validation of the Children’s Hope Scale. Journal of Pediatric Psychology, 22(3), 399–421. https://doi.org/10.1093/jpepsy/22.3.399
63.
StaffordL. (2017). ‘What about my voice’: Emancipating the voices of children with disabilities through participant-centred methods. Children’s Geographies, 15(5), 600–613. https://doi.org/10.1080/14733285.2017.1295134
64.
TautS. (2008). What have we learned about stakeholder involvement in program evaluation?Studies in Educational Evaluation, 34(4), 224–230. https://doi.org/10.1016/j.stueduc.2008.10.007
TheobaldM. (2019). UN Convention on the Rights of the Child: ‘Where are we at in recognising children’s rights in early childhood, three decades on . . . ?’International Journal of Early Childhood, 51, 251–257. https://doi.org/10.1007/s13158-019-00258-z
67.
ThompsonD. R.IachanR.OverpeckM.RossJ. G.GrossL. A. (2006). School connectedness in the health behavior in school-aged children study: The role of student, school, and school neighborhood characteristics. Journal of School Health, 76(7), 379–386. https://doi.org/10.1111/j.1746-1561.2006.00129.x
68.
TwerskyF.BuchananP.ThrelfallV. (2013). Listening to those who matter most, the beneficiaries. Stanford Social Innovation Review, 11(2), 40–45. http://search.proquest.com/docview/1319489436/
UsherH.-L.MillsR.MossP.TraceyF. (2018). Drawing the threads together: How emerging technologies can help integrate the health care needs of children and young people in out-of-home care. International Journal of Integrated Care, 18, 74. https://doi.org/10.5334/ijic.s1074
71.
WangM. C.HaertalG. D.WalbergH. J. (1994). Educational resilience in inner cities. In WangM. C.GordonE. W. (Eds.), Educational resilience in inner-city America: Challenges and prospects (pp. 45–72). Lawrence Erlbaum.
72.
WilliamsG. C.DeciE. L. (1996). Internalization of biopsychosocial values by medical students: A test of self-determination theory. Journal of Personality and Social Psychology, 70(4), 767–779. https://doi.org/10.1037/0022-3514.70.4.767
73.
WilsonA. M.MagareyA. M.DollmanJ.JonesM.MasterssonN. (2010). The challenges of quantitative evaluation of a multi-setting, multi-strategy community-based childhood obesity prevention programme: Lessons learnt from the eat well be active community programs in South Australia. Public Health Nutrition, 13(8), 1262–1270. https://doi.org/10.1017/S1368980009991807
74.
WoltersC. A.PintrichP. R.KarabenickS. A. (2005). Assessing academic self-regulated learning. In MooreA. W.LippmanL. H. (Eds.), What do children need to flourish? (Vol. 3, pp. 251–270). Springer. https://doi.org/org/10.1007/0-387-23823-9_16
