Implementing measurable goals for diversity,equity,and inclusion in Clinical and Translational Science Awards leadership

Abstract

Charting a benchmarking strategy is recommended to measure improvements in equal leadership opportunities for minorities and women in the Clinical Translational Science Awards Consortium (CTSAs). Academic institutions that support diversity, equity, and inclusion (DEI) initiatives should demonstrate a willingness to track their progress with established metrics. In the fall of 2020, the CTSAs convened a virtual conference, which started a dialogue on developing measurable objectives to ensure accountability for DEI goals. Using qualitative and quantitative data from breakout sessions, the authors analyze participant responses to the following recommendation, “develop a common metric and dashboard with regular reporting on diversity in CTSA leadership, with an emphasis on increasing female and Black Indigenous, People Of Color (BIPOC) representation to 30% each,” to understand the impact and challenges associated with implementing metrics within CTSAs. Thematic analysis revealed that white supremacist culture and demographic composition are obstacles to establishing metrics. Participants expressed uncertainty about the perception of token roles. Additionally, participants believed that DEI targets can increase diversity in perspectives and approaches to translational science. Implications for CTSAs include establishing CTSA-wide benchmarks for DEI initiatives, which includes a baseline of the existing DEI climate to assess institutional norms and measurable objectives to track progress.

Introduction

Diversity in the scientific workforce adds creativity and brings innovative ideas in problem solving (National Institutes of Health, 2019). Despite such evidence, minorities and women lack equal representation in academic leadership roles. A 2016 status report from the American Council of Education reported that 21.1% of faculty members at institutions were faculty of color, 16.8% of university presidencies were held by people of color, and 5.1% of university presidencies were held by women of color (Espinosa et al., 2019). In Fall 2020, the National Center for Education Statistics reported the demographic make-up of full-time faculty at institutions of higher education in the US as 67.4% white, 10.9% Asian/Pacific Islander, 5.7% Black, and 5.3% Hispanic (National Center for Education Statistics, 2021). A 2017 meta-analysis by Quillian et al. found that discrimination against populations of color and women has not significantly declined over the past 25 years across industries (Quillian et al., 2017). Additional studies examined hiring practices of minoritized and female candidates and found that racism, discrimination, and white supremacy culture persist as key barriers to hiring (Quillian et al., 2017; Antwi-Boasiako, 2008; Zschirnt and Ruedin, 2016).

In 2020, the American Council on Education reported that fewer than half of higher education institutions felt like they had adequate resources and tools to focus on and achieve DEI efforts (Cheung et al., 2020: 2). In order to best address this gap, the Council recommends that academic institutions implement accountability measures, such as measurable DEI goals and hiring metrics (Cheung et al., 2020: 3–4). Additional literature suggests that many organizations’ DEI efforts do not lead to structural change (Anand, 2019; Melaku and Winkler, 2022; Singleton et al., 2021). DEI efforts are often viewed as “diversity checkboxes,” lacking accountability and ignoring underlying barriers to progress (Anand, 2019). Melaka and Winkler (2022) stress the importance of measuring the progress, or lack thereof, of DEI efforts through accountability metrics. They suggest that metrics need to include requirements, for example, hiring metrics that connect the organizational structure to DEI progress. In addition, Boulware et al. (2022) recommend that metrics be used to assess the state of DEI in clinical and translational research and ensure accountability in DEI efforts and improvements.

Many scientific organizations, including the NIH, public health agencies, national medical organizations, and academic institutions, continue to bring awareness of the need to improve DEI in biomedical, clinical, and translational research and have established committees, training programs, altered recruitment strategies, and funded programs to support women and URM, junior faculty and investigators (National Institutes of Health, 2021; Ofili et al., 2021; UC Berkeley, n.d.). While this demonstrates some advancements from knowledge to practice, operationalizing DEI efforts and tracking performance using realistic metrics is needed to ensure progress. Benchmarking represents another pathway scientific fields can use to monitor progress in their DEI efforts (O’Mara et al., 2021). With different ideologies of what inclusive excellence means, benchmarking serves an important role in setting standards and reaching agreement upon the desired state of DEI, identifying the current state, engaging employees, determining short and long-term goals, and then tracking progress toward those goals (O’Mara et al., 2021; Moriarty, 2011).

The evidence in the scientific literature is clear cut on the value of identifying metrics and discrete objectives for DEI goals to monitor progression (Abdel-Rahman et al., 2021). With this in mind, the Fall 2020 virtual Clinical and Translational Science Awards (CTSA) Consortium Meeting prioritized recommendations to advance DEI at CTSA hubs and identify benchmarks to examine success in achieving those priority goals (Boulware et al., 2022). This analysis focuses on the recommendation: Define a target of 30% of leaders (Principal Investigators/Core Directors) of translational science awards are women and 30% of the leaders are Black Indigenous and People of Color (BIPOC). Develop a common metric and dashboard with regular reporting on diversity in CTSA leadership (henceforth referred to as “targets, metrics, and dashboards”). Chats and polling data from this session were analyzed using thematic analysis. The results of this analysis provides recommendations to CTSAs on the critical importance of establishing measurable and sustainable DEI objectives to foster equity, accountability, and an inclusive culture that prioritizes minority and female leadership opportunities through the voices of those currently engaged at the CTSAs.

Methods

Participants and setting

During the virtual annual meeting of the CTSA Consortium, the steering committee chose to use the meeting to identify ways to improve diversity, equity, and inclusion in four strategic areas: leadership diversity, workforce development, research in health equity/disparities, and clinical trials recruitment. Registrants of the virtual Fall 2020 CTSA Consortium Meeting included: Executive Director/Administrators (n = 35), UL1 Principal Investigators (n = 30), KL2 Directors (n = 22), CTSA Program Hub Steering Committee Members (n = 18), TL1 Directors (n = 15), NCATS Program Officers or NIH Representatives (n = 7), and participants holding other positions (n = 117). Participants were asked to self-select to attend one of the four strategic areas and each breakout room included discussions about DEI in the respective topic areas, drawing on stakeholder participation to develop recommendations for moving forward. The data from the leadership breakout room constitutes the focus of this analysis, specifically responses to the “targets, metrics, and dashboards” recommendation. The leadership breakout room consisted of 67 participants from the overall group. Participants were informed of a consent clause during online registration and this research adhered to UNC’s IRB and ethical procedures.

Data collection and analysis

This study analyzed qualitative and quantitative data of Zoom chat and PollEverywhere poll result transcripts from the leadership diversity breakout session during the 2020 virtual national CTSA meeting. Participants were prompted to send in short, written responses to six recommendations, with the “targets, metrics, and dashboards” recommendation being one of them, which were aimed at advancing DEI efforts in CTSAs and the translational science field. Participants were asked one question about potential risks and one question about potential opportunities for pursuing the “targets, metrics, and dashboards” recommendation and voluntarily responded in the Zoom chat function. After the recommendation was presented and commented on, participants responded to three polls, using PollEverywhere, on the potential feasibility, impact, and priority of “targets, metrics and dashboards”. On each of the three polls, participants ranked the six recommendations from first place to sixth place, according to the three domains.

Two investigators independently coded the chat transcript data to look for emerging themes and cross-cutting themes across participant responses in order to identify main challenges and opportunities identified around “targets, metrics and dashboards.” After each coder identified preliminary themes, the two met to reach consensus and create an overall codebook of primary themes. The chat responses were inductively coded in NVivo software, and the tallies of each code were used to identify the most common themes (see Table 1). Polling data from the three PollEverywhere polls was also analyzed by examining the average ranking position of “targets, metrics, and dashboards” across polling results in order to identify respondents' perceptions of the feasibility, impact, and priority of this recommendation.

Table 1.

Emerging themes, definitions, and example quotes in response to a common metric and dashboard for tracking DEI in CTSAs.

Emerging theme	Definition	Example quotes
Whiteness and racism	White supremacy culture and divisions among white members of CTSAs and institutions	“White supremacy culture is deepened and BIPOC people and communities are further harmed.”	“Is it equitable? Was the most qualified individual selected or to meet this criteria a qualified individual is left out.”
Structural change	Large scale shift in the practices of CTSAs and their broader institutions. The need for large-scale change in order to achieve these priorities-change beyond the CTSA	“Would take a paradigm shift, but there would be buy-in at some point and impact positively.”	“No structural change; just gaming the system.”
Superficial change	Fear that the changes resulting from these priorities may not lead to structural change	“Risk is that this will be isolated, rather than seen as paradigmatic for an institution, where these larger biases may also exist- so it might be an island, instead of the water.”	“People will be placed into token roles without any power.”
New perspectives	Expanding DEI in CTSA leadership could result in new, broader, more diverse perspectives	“Diverse ways of thinking/approaching translational science.”	“Impact on leadership: Improved diversity of perspective and models.”

Results

Chat results

Table 1 presents Zoom chat quotes from participants in the leadership diversity breakout session by theme from individuals in CTSAs to the recommendation of developing “targets, metrics and dashboards.” Question 2A asked about potential risks in carrying out a common metric and dashboard for DEI efforts in CTSAs. Participants described concerns about isolated impact, white supremacy culture, and demographic makeup at different locations serving as barriers hindering the utility of “targets, metrics, and dashboards.” Additionally, respondents questioned if creating this benchmark would be equitable and if it risks placing individuals into “token roles.” Question 2B asked about potential impacts “targets, metrics, and dashboards” could have on leadership in CTSAs. In response, participants expressed feelings of excitement over new perspectives, ideas, and creativity that enacting such strategies could bring. Others talked about how “targets, metrics, and dashboards” would require a paradigm shift in current leadership and institutions, but improved DEI leadership inside CTSAs would lead to increased diversity in the broader community. Four primary themes emerged from the analysis of the “targets, metrics, and dashboards” recommendation: whiteness and racism (n = 5 references, 10.20% of responses), structural change (n = 3 references, 6.12% of responses), new perspectives (n = 7 references, 14.29% of responses), and superficial change (n = 3 references, 6.12% of responses). The greatest opportunity “targets, metrics, and dashboards” presented, as determined by respondents, was the increase in new perspectives which could result in broader, more diverse ideas, projects, etc. Whiteness, racism, and discrimination that are embedded in institutional culture was identified as a key barrier to moving forward with a common “targets, metrics and dashboard” system. Additionally, the debate over whether recommending “targets, metrics, and dashboards” would result in structural or superficial change was present throughout many responses.

Polling results

The subsequent poll asked participants to upvote or downvote the recommendation of “targets, metrics, and dashboards”, along with the five other recommendations presented. Looking at the polling data to determine the averaged ranking position, participants ranked the recommendation of “targets, metrics and dashboards” overall as the fifth of the six possible to recommend future action. Poll One questioned the feasibility of the six recommendations, and garnered 53 responses (79% engagement), with the results placing “targets, metrics and dashboards” in fifth place as compared to the other issues. Poll Two asked participants to rank the six issues on the greatest impact, which garnered 58 responses (88% engagement) and left “targets, metrics and dashboards” in fourth place. Poll Three asked about which recommendation should receive the highest priority in implementation. This poll had 88% engagement (59 responses) and placed “targets, metrics and dashboards” in fifth place. The recommendations that ranked above “targets, metric and dashboards” discussed developing transparent hiring policies, identifying cross-institutional work groups, defining a clear succession plan for BIPOC leaders, and making room for structural change through leaders giving over space, influence, and resources (Fernandez et al., 2022). For more detailed findings from the polls and overall results, refer to complementary article Fernandez (2022).

Discussion

Beyond published work, CTSA’s can take actionable steps to eliminate systemic barriers that perpetuate inequities in DEI initiatives within their walls. The CTSA virtual conference in 2020 gave CTSA members a space to authentically dialogue on challenges that hinder academic institutions from establishing metrics to monitor DEI progress. The findings from this qualitative work are intended to highlight the voices of these participants so that academic institutions can learn what behaviors to target to improve equity and move towards the creation of defined, transparent, and measurable DEI goals to evaluate success and opportunities for correction. The findings from this study shows workforce of the CTSAs felt less optimistic about the feasibility, the impact, or the priority of using “targets, metrics and dashboards” as a key strategy to advance DEI goals in CTSA leadership, when compared to five other recommendations described in our other published work (Fernandez et al., 2022). Respondents expressed concerns that lack of institutional buy-in and backlash from white leadership and staff could serve as barriers to measuring DEI efforts. Consistent with the academic discourse on barriers to measuring DEI efforts in academia, the CTSA workforce in this study also identified racism and discrimination as impediments towards a commitment to DEI performance measures. (Hinton and Lambert, 2022; Grindstaff, 2022). Many of the participants also noted that recommendations to set metrics for DEI goals would be superficial and the historical roots of exclusionary practices in CTSAs would still exist. As a result, the participants ranked other recommendations such as transparency in selection processes, cross institutional action learning strategies, and intentional succession planning as having greater feasibility, impact, and priority for recommendations.

Interestingly, such goal setting has been demonstrated to be useful in achieving sustainable goals with DEI initiatives (Sweeney and Bothwick, 2016) and could be similarly useful in facilitating measurable change in CTSAs. For example, organizations can follow Sweeney and Bothwick’s STAR framework to create and achieve sustainable goals with metrics in their DEI initiatives (Sweeney and Bothwick, 2016). The acronym STAR stands for “starting out, taking the leap, achieving change, and reaping the rewards” and allows for organizations to guide DEI efforts and track change (Sweeney and Bothwick, 2016). Objectives for DEI plans and programs should also incorporate the SMART criteria (Specific, Measurable, Assigned, Realistic, and Timed), considered the gold standard for the development of goals and objectives (Bjerke and Renger, 2017). SMART objectives provide a framework for creating specific and outcome-oriented goals that allow for more effective goal setting, planning, and implementation (Ogbeiwi, 2017: 328). Ogbeiwi stresses the importance of creating SMART objectives that specify outcome, indicator, target-level, and timeframe in order to best create results-oriented action (Ogbeiwi, 2017: 329–330). The Centers for Disease Control and Prevention, the United Way, the W.K. Kellogg Foundation, and the United States Department of Education all recommend using SMART criteria to create objectives, as SMART objectives allow for greater and accessible evaluation (Bjerke and Renger, 2017).

While participants seemed less optimistic about the feasibility, impact, and priority of “targets, metrics, and dashboards”, establishing benchmarks with targeted SMART objectives that include increasing women and minority representation in leadership in a realistic timeframe could help guide the efforts aligning with the other higher ranking priorities, such as creating action-learning workgroups specifically populated with women and under-represented scientists or applying the “targets, metrics, and dashboard” models to intentional mentoring and succession planning efforts. Additionally, implementing a consortium-wide dashboard can monitor progress towards achieving SMART objectives and help illuminate which other strategies help move the needle in effective ways. Honest efforts to accomplish equitable leadership representation requires institutional leaders to introspectively, reflect on the culture of whiteness and acknowledge the loss of ideas and opportunities that result from subtle and overt racism, in order to achieve an inclusive culture and impact systemic change. Implementing some form of metrics and dashboards can be a useful tool to keeping on track with these larger aims.

Limitations

The Consortium meeting had broad representation of the CTSA workforce and 29% of the virtual conference participants self-selected to attend the leadership breakout session. While this participation is a slight over-representation of the entire group, the composition of the roles and responsibilities of the breakout group participants is unknowable. It is possible that the group represented a cluster rather than a broad range of roles and responsibilities, which theoretically could result in a particular or non-representative viewpoint. Another limitation is that conference participants typically use their real names for their zoom identity, which shows up on the screen and is visible in the chat box. Additionally, although chat responses were gathered in a “waterfall” format, where the group simultaneously submits responses making individual comments challenging to peruse, nevertheless the knowledge that others could see one’s comment might have affected participation. It is possible that when asked to respond to sensitive, politically delicate, or emotionally tense issues that some hesitated to participate if their comments might be perceived as different from the whole or as non-supportive of DEI efforts.

Conclusions

Diversity in the scientific workforce offers benefits of creativity and innovation as well as promotes cultural sensitivity, which can be reflected in the experiences of research teams as well as the communities served. Yet it can be difficult to track the success of DEI initiatives, particularly in large and complex organizations such as research intensive universities. In this paper, we recommend academic institutions who support DEI initiatives set clearly defined benchmarks that are based on the science and the voices of their constituents. Lessons learned for CTSA’s include engaging with their respective member constituents to understand race and discriminatory perceptions, practices, and power dynamics within their institutions. Additionally, they should move beyond diversity, equity and inclusion surveys and invest in resources in qualitative research to get a better understanding of the breadth and depth of inequitable practices and beliefs. Following this, CTSA’s can create more realistic and targeted DEI benchmarks (and more effective implementation) if they develop strategies to achieve such benchmarks based on the outcomes from qualitative studies, which should include a bi-directional dialogue of DEI concerns with their CTSA members. While not seen as the most important priority by CTSA workforce members in this study, employing strategies around targets, metrics, and dashboards can be helpful in demonstrating a willingness to track progress and can facilitate advancement of other identified priorities.