More evaluators have anchored their work in equity-focused, culturally responsive, and social justice ideals. Although we have a sense of approaches that guide evaluators as to how they should attend to culture, diversity, equity, and inclusion (DEI), we have not yet established an empirical understanding of how evaluators measure DEI. In this article, we report an examination of how evaluators and principal investigators (PIs) funded by the National Science Foundation's Advanced Technological Education (ATE) program define and measure DEI within their projects. Evaluators gathered the most evidence related to diversity and less evidence related to equity and inclusion. On average, PIs’ projects engaged in activities designed to increase DEI, with the highest focus on diversity. We believe there continues to be room for improvement and implore the movement of engagement with these important topics from the margins to the center of our field's education, theory, and practice.
American Society of Higher Education (2011). Racial and ethnic minority students’ success in STEM education [special issue]. ASHE Higher Education Report, 36(6).
4.
ArchibaldT.SharrockG.BuckleyJ.YoungS. (2018). Every practitioner a “knowledge worker”: Promoting evaluative thinking to enhance learning and adaptive management in international development. New Directions for Evaluation, 2018(158), 73–91. https://doi.org/10.1002/ev.20323
5.
AugusteE.PackardB. W.-L.KeepA. (2018). Nontraditional women students’ experiences of identity recognition and marginalization during advising. NACADA Journal, 38(2), 45–60. https://doi.org/10.12930/nacada-17-046
6.
AvendanoL.RenteriaJ.KwonS.HamdanK. (2019). Bringing equity to underserved communities through STEM education: Implications for leadership development. Journal of Educational Administration and History, 51(1), 66–82.
7.
AventC. (2020). Investigations of social justice conceptualizations: The capacity of language to influence stakeholder and evaluator actions. Unpublished doctoral dissertation, University of North Carolina, Greensboro.
8.
BoyceA. S. (2017). Lessons learned using a values-engaged approach to attend to culture, diversity, and equity in a STEM program evaluation. Evaluation and Program Planning, 64, 33–43. https://doi.org/10.1016/j.evalprogplan.2017.05.018
9.
BoyceA. S. (2019). A re-imagining of evaluation as social justice: A discussion of the education justice project. Critical Education, 10(1), 1–19.
10.
BoyceA. S.ChouinardJ. A. (2017). Moving beyond the buzzword: A framework for teaching culturally responsive approaches to evaluation. Canadian Journal of Program Evaluation, 32(2), 266–279.
11.
BraunV.ClarkeV. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
12.
BuckleyJ.ArchibaldT.HargravesM.TrochimW. M. (2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36(3), 375–388.
13.
CatsambasT.SegoneM.De SilvaS.SaundersM. (2013). EvalPartners: An international partnership to strengthen civil society’s evaluation capacities and promote equity. Evaluation and Civil Society. UNICEF EWP Series, pp. 43–51. New York, NY.
14.
CharlestonL. J. (2012). A qualitative investigation of African Americans’ decision to pursue computing science degrees: Implications for cultivating career choice and aspiration. Journal of Diversity in Higher Education, 5(4), 222–243. https://doi.org/10.1037/a0028918
15.
ChouinardJ. A.CousinsJ. B. (2009). A review and synthesis of current research on cross-cultural evaluation. American Journal of Evaluation, 30(4), 457–494.
16.
ChouinardJ. A.CramF. (2020). Culturally responsive approaches to evaluation: Empirical implications for theory and practice (vol. 4). Sage Publications.
17.
ClevelandD. (Ed.). (2004). A long way to go: Conversations about race by African American faculty and graduate students (vol. 14). Peter Lang.
DaviesR.MacKayK. (2014). Evaluator training: Content and topic valuation in university evaluation courses. American Journal of Evaluation, 35(3), 419–429.
20.
Dean-CoffeyJ. (2018). What’s race got to do with it? Equity and philanthropic evaluation practice. American Journal of Evaluation, 39(4), 527–542.
21.
EricsonD. P. (1990). Social justice, evaluation, and the educational system. New Directions for Program Evaluation, 1990(45), 5–21.
22.
FreireP. (1996). Pedagogy of the oppressed (revised). Continuum.
23.
FriersonH.HoodS.HughesG.ThomasV. (2010). A guide to conducting culturally responsive evaluation. In Frechling WestatJ. (Ed.), The 2010 user-friendly handbook for project evaluation. National science foundation, directorate for education and human resources, division of research, evaluation, and communication. REC 99-12175 (pp. 75–96). National Science Foundation. Retrieved from: The 2010 User-Friendly Handbook for Project Evaluation (rrf.org).
24.
GreeneJ. C.BoyceA. S.AhnJ. (2011). Values-engaged, educative evaluation guidebook. University of Illinois, Urbana-Champaign. Created and produced with funds from the National Science Foundation.
25.
GreeneJ. C.DeStefanoL.BurgonH.HallJ. (2006). An educative, values-engaged approach to evaluating STEM educational programs. New Directions for Evaluation, 2006(109), 53–71.
26.
HallJ. N. (2020). The other side of inequality: Using standpoint theories to examine the privilege of the evaluation profession and individual evaluators. American Journal of Evaluation, 41(1), 20–33.
27.
HallM. E. (2018). Evaluation’s race problem in the United States: A call to action for the profession and the American journal of evaluation. American Journal of Evaluation, 39(4), 569–583.
28.
HoodS. (1998). Responsive evaluation amistad style: Perspectives of one African American evaluator. In DavisR. (Ed.), Proceedings of the stake symposium on educational evaluation (pp. 101–112). University of Illinois.
29.
HoodS.HopsonR. K.FriersonH. T. (Eds.) (2015a). Continuing the journey to reposition culture and cultural context in evaluation theory and practice. InfoAge Publishing Co.
30.
HoodS.HopsonR. K.KirkhartK. (2015b). Culturally responsive evaluation. In NewcomerK. E.HatryH. P.WholeyJ. S. (Eds.), Handbook of practical program evaluation (4th ed, pp. 281–317). John Wiley & Sons.
31.
HouseE. (1980). Evaluation with validity. Sage.
32.
HouseE.HoweK. R. (1999). Values in evaluation and social research. Sage Publications.
33.
HouseE. R. (1991). Evaluation and social justice: Where are we? In McLaughlinM.PhillipsD. (Eds.), Evaluation and education at quarter century, national society for the study of education yearbook (pp. 233–246). University of Chicago Press.
34.
KlenkN. L.MeehanK.PinelS. L.MendezF.LimaP. T.KammenD. M. (2015). Stakeholders in climate science: Beyond lip service?Science (New York, N.Y.), 350(6262), 743–744.
35.
KulikC. T.RobersonL. (2008). Diversity initiative effectiveness: What organizations can (and cannot) expect from diversity recruitment, diversity training, and formal mentoring programs. In BriefA. P. (Ed.), Cambridge companions to management. Diversity at work (pp. 265–317). Cambridge University Press. https://doi.org/10.1017/CBO9780511753725.010.
36.
LeeA. (2015). An investigation of the linkage between technology-based activities and STEM major selection in 4-year postsecondary institutions in the United States: Multilevel structural equation modelling. Educational Research and Evaluation, 21(5–6), 439–465.
37.
LincolnP. Z. (2015). Educational equality or educational equity. NUCB Journal of Economics and Information Science, 60(1), 187–203.
38.
MacDonaldB. (1976). Evaluation and the control of education. In TawneyD. (Ed.), Curriculum evaluation today: Trends and implications (pp. 125–136). Macmillan.
39.
MadisonA. (2007). New directions for evaluation coverage of cultural issues and issues of significance to underrepresented groups. New Directions for Evaluation, 2007(114), 107–114.
40.
MadisonA. M. (1991). Minority issues in program evaluation. New Directions for Program Evaluation, 53, 1–92.
41.
MapedzahamaV.RudgeT.WestS.PerronA. (2012). Black nurse in white space? Rethinking the in/visibility of race within the Australian nursing workplace. Nursing Inquiry, 19(2), 153–164.
42.
MarraM. (2015). Cooperating for a more egalitarian society: Complexity theory to evaluate gender equity. Evaluation, 21(1), 32–46.
43.
McCabeJ. (2009). Racial and gender microaggressions on a predominantly-White campus: Experiences of Black, Latina/o and White undergraduates. Race. Gender & Class, 16(1/2), 133–151.
44.
McGeeE. O. (2021). Black, brown, bruised: How racialized STEM education stifles innovation. Harvard Education Press.
45.
MertensD. M.HopsonR. K. (2006). Advancing evaluation of STEM efforts through attention to diversity and culture. New Directions for Evaluation, 2006(109), 35–51. https://doi.org/10.1002/ev.177
46.
MertensD. M.WilsonA. T. (2018). Program evaluation theory and practice. Guilford Publications.
47.
National Academies of Sciences, Engineering, and Medicine (2018). Indicators for monitoring undergraduate STEM education. The National Academies Press. https://doi.org/10.17226/24943
48.
National Center for Educational Statistics (2009). The condition of education. US Department of Education, Office of Educational Research and Improvement.
Osei-KofiN.TorresL. E. (2015). College admissions viewbooks and the grammar of gender, race, and STEM. Cultural Studies of Science Education, 10(2), 527–544.
53.
PackardB. W. L. (2015). Successful STEM mentoring initiatives for underrepresented students: A research-based guide for faculty and administrators. Stylus Publishing, LLC.
PurittyC.StricklandL. R.AliaE.BlonderB.KleinE.KohlM. T.GerberL. R. (2017). Without inclusion, diversity initiatives may not be enough. Science (New York, N.Y.), 357(6356), 1101–1102. https://science.sciencemag.org/content/357/6356/1101
56.
ReidA. M.BoyceA. S.AdetogunA.MollerJ.AventC. (2020). If not us, then who? Evaluators of color and social change. In L. C. Neubauer, D. McBride, A. D. Guajardo, W. D. Casillas, & M. E. Hall (eds.), examining issues facing communities of color today: The role of evaluation to incite change. New Directions for Evaluation, 166, 23–36.
57.
SamuelsM.RyanK. (2011). Grounding evaluations in culture. American Journal of Evaluation, 32(2), 183–198.
58.
SmithC.WingateL. (2016). Strategies for broadening participation in advanced technological education programs: Practice and perceptions. Community College Journal of Research and Practice, 40(9), 779–796.
59.
SmithT. L.BarlowP. B.SkolitsG. J.PetersJ. M. (2015). Demystifying reflective practice: Using the DATA model to enhance evaluators’ professional activities. Evaluation and Program Planning, 52(2015), 142–147.
60.
Stickl HaugenJ.ChouinardJ. A. (2019). Transparent, translucent, opaque: Exploring the dimensions of power in culturally responsive evaluation contexts. American Journal of Evaluation, 40(3), 376–394. https://doi.org/10.1177/1098214018796342
61.
ThomasV. G.MadisonA. (2010). Integration of social justice into the teaching of evaluation. American Journal of Evaluation, 31(4), 570–583.
62.
ToveyT. L. S.ArchibaldT. (2022, in press). The relationship between reflective practice, evaluative thinking, and practical wisdom. In DonaldsonS.HurteauM. (Eds.), Practical wisdom for an ethical evaluation practice. Information Age Publishing.
63.
ToveyT. L. S.SkolitsG. J. (2021). Conceptualizing and engaging in reflective practice: Experienced evaluators’ perspectives. American Journal of Evaluation, 43(1), 5–25.
64.
TurnerC.GrauerholzL. (2017). Introducing the invisible man: Black male professionals in higher education. Humboldt Journal of Social Relations, 39, 212–227.
65.
VossoughiS.HooperP. K.EscudéM. (2016). Making through the lens of culture and power: Toward transformative visions for educational equity. Harvard Educational Review, 86(2), 206–232. NUCB journal of economics and information science=名古屋商科大学論集.
66.
WeinerG. (1990). Ethical practice in an unjust world: Educational evaluation and social justice. Gender and Education, 2(2), 231–238. https://doi.org/10.1080/0954025900020207
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
