Exploring NSF-Funded Evaluators’ and Principal Investigators’ Definitions and Measurement of Diversity,Equity,and Inclusion

Abstract

More evaluators have anchored their work in equity-focused, culturally responsive, and social justice ideals. Although we have a sense of approaches that guide evaluators as to how they should attend to culture, diversity, equity, and inclusion (DEI), we have not yet established an empirical understanding of how evaluators measure DEI. In this article, we report an examination of how evaluators and principal investigators (PIs) funded by the National Science Foundation's Advanced Technological Education (ATE) program define and measure DEI within their projects. Evaluators gathered the most evidence related to diversity and less evidence related to equity and inclusion. On average, PIs’ projects engaged in activities designed to increase DEI, with the highest focus on diversity. We believe there continues to be room for improvement and implore the movement of engagement with these important topics from the margins to the center of our field's education, theory, and practice.

Keywords

diversity equity inclusion cultural responsiveness

Get full access to this article

View all access options for this article.

References

Advanced Technological Education Central (2021). Retrieved from https://atecentral.net/

Advanced Technological Education Impacts (2020). Retrieved from https://ateimpacts.net/book

American Society of Higher Education (2011). Racial and ethnic minority students’ success in STEM education [special issue]. ASHE Higher Education Report, 36(6).

Archibald

Sharrock

Buckley

Young

(2018). Every practitioner a “knowledge worker”: Promoting evaluative thinking to enhance learning and adaptive management in international development. New Directions for Evaluation, 2018(158), 73–91. https://doi.org/10.1002/ev.20323

Auguste

Packard

B. W.-L.

Keep

(2018). Nontraditional women students’ experiences of identity recognition and marginalization during advising. NACADA Journal, 38(2), 45–60. https://doi.org/10.12930/nacada-17-046

Avendano

Renteria

Kwon

Hamdan

(2019). Bringing equity to underserved communities through STEM education: Implications for leadership development. Journal of Educational Administration and History, 51(1), 66–82.

Avent

(2020). Investigations of social justice conceptualizations: The capacity of language to influence stakeholder and evaluator actions. Unpublished doctoral dissertation, University of North Carolina, Greensboro.

Boyce

A. S.

(2017). Lessons learned using a values-engaged approach to attend to culture, diversity, and equity in a STEM program evaluation. Evaluation and Program Planning, 64, 33–43. https://doi.org/10.1016/j.evalprogplan.2017.05.018

Boyce

A. S.

(2019). A re-imagining of evaluation as social justice: A discussion of the education justice project. Critical Education, 10(1), 1–19.

10.

Boyce

A. S.

Chouinard

J. A.

(2017). Moving beyond the buzzword: A framework for teaching culturally responsive approaches to evaluation. Canadian Journal of Program Evaluation, 32(2), 266–279.

11.

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.

12.

Buckley

Archibald

Hargraves

Trochim

W. M.

(2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36(3), 375–388.

13.

Catsambas

Segone

De Silva

Saunders

(2013). EvalPartners: An international partnership to strengthen civil society’s evaluation capacities and promote equity. Evaluation and Civil Society. UNICEF EWP Series, pp. 43–51. New York, NY.

14.

Charleston

L. J.

(2012). A qualitative investigation of African Americans’ decision to pursue computing science degrees: Implications for cultivating career choice and aspiration. Journal of Diversity in Higher Education, 5(4), 222–243. https://doi.org/10.1037/a0028918

15.

Chouinard

J. A.

Cousins

J. B.

(2009). A review and synthesis of current research on cross-cultural evaluation. American Journal of Evaluation, 30(4), 457–494.

16.

Chouinard

J. A.

Cram

(2020). Culturally responsive approaches to evaluation: Empirical implications for theory and practice (vol. 4). Sage Publications.

17.

Cleveland

(Ed.). (2004). A long way to go: Conversations about race by African American faculty and graduate students (vol. 14). Peter Lang.

18.

Committee on Equal Opportunities in Science and Engineering (2017). Biennial Report to Congress 2017–2018: Investing in Diverse Community Voices. Retrieved from https://www.nsf.gov/od/oia/activities/ceose/reports/CEOSE_ReportToCongress_RP_FVmp_508.pdf

19.

Davies

MacKay

(2014). Evaluator training: Content and topic valuation in university evaluation courses. American Journal of Evaluation, 35(3), 419–429.

20.

Dean-Coffey

(2018). What’s race got to do with it? Equity and philanthropic evaluation practice. American Journal of Evaluation, 39(4), 527–542.

21.

Ericson

D. P.

(1990). Social justice, evaluation, and the educational system. New Directions for Program Evaluation, 1990(45), 5–21.

22.

Freire

(1996). Pedagogy of the oppressed (revised). Continuum.

23.

Frierson

Hood

Hughes

Thomas

(2010). A guide to conducting culturally responsive evaluation. In Frechling Westat

(Ed.), The 2010 user-friendly handbook for project evaluation. National science foundation, directorate for education and human resources, division of research, evaluation, and communication. REC 99-12175 (pp. 75–96). National Science Foundation. Retrieved from: The 2010 User-Friendly Handbook for Project Evaluation (rrf.org).

24.

Greene

J. C.

Boyce

A. S.

Ahn

(2011). Values-engaged, educative evaluation guidebook. University of Illinois, Urbana-Champaign. Created and produced with funds from the National Science Foundation.

25.

Greene

J. C.

DeStefano

Burgon

Hall

(2006). An educative, values-engaged approach to evaluating STEM educational programs. New Directions for Evaluation, 2006(109), 53–71.

26.

Hall

J. N.

(2020). The other side of inequality: Using standpoint theories to examine the privilege of the evaluation profession and individual evaluators. American Journal of Evaluation, 41(1), 20–33.

27.

Hall

M. E.

(2018). Evaluation’s race problem in the United States: A call to action for the profession and the American journal of evaluation. American Journal of Evaluation, 39(4), 569–583.

28.

Hood

(1998). Responsive evaluation amistad style: Perspectives of one African American evaluator. In Davis

(Ed.), Proceedings of the stake symposium on educational evaluation (pp. 101–112). University of Illinois.

29.

Hood

Hopson

R. K.

Frierson

H. T.

(Eds.) (2015a). Continuing the journey to reposition culture and cultural context in evaluation theory and practice. InfoAge Publishing Co.

30.

Hood

Hopson

R. K.

Kirkhart

(2015b). Culturally responsive evaluation. In Newcomer

K. E.

Hatry

H. P.

Wholey

J. S.

(Eds.), Handbook of practical program evaluation (4th ed, pp. 281–317). John Wiley & Sons.

31.

House

(1980). Evaluation with validity. Sage.

32.

House

Howe

K. R.

(1999). Values in evaluation and social research. Sage Publications.

33.

House

E. R.

(1991). Evaluation and social justice: Where are we? In McLaughlin

Phillips

(Eds.), Evaluation and education at quarter century, national society for the study of education yearbook (pp. 233–246). University of Chicago Press.

34.

Klenk

N. L.

Meehan

Pinel

S. L.

Mendez

Lima

P. T.

Kammen

D. M.

(2015). Stakeholders in climate science: Beyond lip service? Science (New York, N.Y.), 350(6262), 743–744.

35.

Kulik

C. T.

Roberson

(2008). Diversity initiative effectiveness: What organizations can (and cannot) expect from diversity recruitment, diversity training, and formal mentoring programs. In Brief

A. P.

(Ed.), Cambridge companions to management. Diversity at work (pp. 265–317). Cambridge University Press. https://doi.org/10.1017/CBO9780511753725.010.

36.

Lee

(2015). An investigation of the linkage between technology-based activities and STEM major selection in 4-year postsecondary institutions in the United States: Multilevel structural equation modelling. Educational Research and Evaluation, 21(5–6), 439–465.

37.

Lincoln

P. Z.

(2015). Educational equality or educational equity. NUCB Journal of Economics and Information Science, 60(1), 187–203.

38.

MacDonald

(1976). Evaluation and the control of education. In Tawney

(Ed.), Curriculum evaluation today: Trends and implications (pp. 125–136). Macmillan.

39.

Madison

(2007). New directions for evaluation coverage of cultural issues and issues of significance to underrepresented groups. New Directions for Evaluation, 2007(114), 107–114.

40.

Madison

A. M.

(1991). Minority issues in program evaluation. New Directions for Program Evaluation, 53, 1–92.

41.

Mapedzahama

Rudge

West

Perron

(2012). Black nurse in white space? Rethinking the in/visibility of race within the Australian nursing workplace. Nursing Inquiry, 19(2), 153–164.

42.

Marra

(2015). Cooperating for a more egalitarian society: Complexity theory to evaluate gender equity. Evaluation, 21(1), 32–46.

43.

McCabe

(2009). Racial and gender microaggressions on a predominantly-White campus: Experiences of Black, Latina/o and White undergraduates. Race. Gender & Class, 16(1/2), 133–151.

44.

McGee

E. O.

(2021). Black, brown, bruised: How racialized STEM education stifles innovation. Harvard Education Press.

45.

Mertens

D. M.

Hopson

R. K.

(2006). Advancing evaluation of STEM efforts through attention to diversity and culture. New Directions for Evaluation, 2006(109), 35–51. https://doi.org/10.1002/ev.177

46.

Mertens

D. M.

Wilson

A. T.

(2018). Program evaluation theory and practice. Guilford Publications.

47.

National Academies of Sciences, Engineering, and Medicine (2018). Indicators for monitoring undergraduate STEM education. The National Academies Press. https://doi.org/10.17226/24943

48.

National Center for Educational Statistics (2009). The condition of education. US Department of Education, Office of Educational Research and Improvement.

49.

National Science Board (2018). Science & Engineering Indicators. Retrieved from https://www.nsf.gov/statistics/2018/nsb20181/assets/1407/digest.pdf

50.

National Science Foundation (2008). Broadening participation at the National Science Foundation: A framework for action. https://www.nsf.gov/od/broadeningparticipation/nsf_frameworkforaction_0808.pdf

51.

National Science Foundation (2018). Advanced Technological Education Program. https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5464

52.

Osei-Kofi

Torres

L. E.

(2015). College admissions viewbooks and the grammar of gender, race, and STEM. Cultural Studies of Science Education, 10(2), 527–544.

53.

Packard

B. W. L.

(2015). Successful STEM mentoring initiatives for underrepresented students: A research-based guide for faculty and administrators. Stylus Publishing, LLC.

54.

President’s Council of Advisors on Science & Technology (2020). Recommendations for strengthening American leadership in industries of the future. Retrieved from https://science.osti.gov/-/media/_/pdf/about/pcast/202006/PCAST_June_2020_Report.pdf?la=en&hash=019A4F17C79FDEE5005C51D3D6CAC81FB31E3ABC

55.

Puritty

Strickland

L. R.

Alia

Blonder

Klein

Kohl

M. T.

Gerber

L. R.

(2017). Without inclusion, diversity initiatives may not be enough. Science (New York, N.Y.), 357(6356), 1101–1102. https://science.sciencemag.org/content/357/6356/1101

56.

Reid

A. M.

Boyce

A. S.

Adetogun

Moller

Avent

(2020). If not us, then who? Evaluators of color and social change. In L. C. Neubauer, D. McBride, A. D. Guajardo, W. D. Casillas, & M. E. Hall (eds.), examining issues facing communities of color today: The role of evaluation to incite change. New Directions for Evaluation, 166, 23–36.

57.

Samuels

Ryan

(2011). Grounding evaluations in culture. American Journal of Evaluation, 32(2), 183–198.

58.

Smith

Wingate

(2016). Strategies for broadening participation in advanced technological education programs: Practice and perceptions. Community College Journal of Research and Practice, 40(9), 779–796.

59.

Smith

T. L.

Barlow

P. B.

Skolits

G. J.

Peters

J. M.

(2015). Demystifying reflective practice: Using the DATA model to enhance evaluators’ professional activities. Evaluation and Program Planning, 52(2015), 142–147.

60.

Stickl Haugen

Chouinard

J. A.

(2019). Transparent, translucent, opaque: Exploring the dimensions of power in culturally responsive evaluation contexts. American Journal of Evaluation, 40(3), 376–394. https://doi.org/10.1177/1098214018796342

61.

Thomas

V. G.

Madison

(2010). Integration of social justice into the teaching of evaluation. American Journal of Evaluation, 31(4), 570–583.

62.

Tovey

T. L. S.

Archibald

(2022, in press). The relationship between reflective practice, evaluative thinking, and practical wisdom. In Donaldson

Hurteau

(Eds.), Practical wisdom for an ethical evaluation practice. Information Age Publishing.

63.

Tovey

T. L. S.

Skolits

G. J.

(2021). Conceptualizing and engaging in reflective practice: Experienced evaluators’ perspectives. American Journal of Evaluation, 43(1), 5–25.

64.

Turner

Grauerholz

(2017). Introducing the invisible man: Black male professionals in higher education. Humboldt Journal of Social Relations, 39, 212–227.

65.

Vossoughi

Hooper

P. K.

Escudé

(2016). Making through the lens of culture and power: Toward transformative visions for educational equity. Harvard Educational Review, 86(2), 206–232. NUCB journal of economics and information science = 名古屋商科大学論集.

66.

Weiner

(1990). Ethical practice in an unjust world: Educational evaluation and social justice. Gender and Education, 2(2), 231–238. https://doi.org/10.1080/0954025900020207

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.13 MB

0.16 MB