Sage Journals: Discover world-class research

Abstract

Background

Research studies in multiple sclerosis (MS) lack diversity.

Objective

To gain consensus on priorities about how future research in MS in Canada addresses equity, diversity, and inclusion (EDI) using a modified e-Delphi technique.

Methods

We recruited people with lived experience (people with MS (PwMS) or family members); EDI researchers; and clinicians with experience caring for PwMS. The first survey was developed based on a scoping review, EDI literature, and advisory group meetings and included items for five domains: measuring and reporting diversity characteristics, recruiting diverse research populations, role of funders, role of publishers/editors, and training. All items were rated on a 7-point scale with anchors of 1 (aspirational), 4 (achievable), and 7 (core). We undertook three rounds. New items were added after the first round; consensus was defined as a standard deviation ≤1.0.

Results

Ultimately, 87 people (36 with lived experience, 28 EDI researchers, 23 MS clinicians) completed the first survey, of whom 82 (94.2%) completed the second survey and 81 (93.1%) completed the third. Forty-five items reached consensus across the five domains. Nearly all these items were rated as achievable.

Conclusions

This e-Delphi identified priorities for how future MS research in Canada addresses EDI.

Keywords

multiple sclerosis Delphi consensus equity diversity inclusion Canada

Introduction

Social determinants of health such as education, as well as age, sex, gender, sexual orientation, ethnicity are dimensions of diversity. Diversity is strongly associated with health outcomes in people with multiple sclerosis (PwMS). Immigrants to Ontario, Canada with multiple sclerosis (MS) have more comorbidities¹ and are hospitalized more often in the year of diagnosis than long-term Canadian residents.² Lower as compared to higher socioeconomic status is associated with increased hospitalizations³ and greater disability progression⁴ in PwMS. Yet, a scoping review of the Canadian MS literature found that most studies did not report diversity characteristics beyond age and sex and that racialized PwMS were likely underrepresented.⁵

Recent observations indicate that research studies in MS lack diversity regarding leadership, interest holder involvement, and participant characteristics.⁶ In a review of Phase 3 clinical trials of disease-modifying therapies 38% (17) of trials did not report race or ethnicity⁷; the median percentage of participants who did not identify as White was only 6%. Similarly, trials of rehabilitation interventions in MS did not enroll diverse populations and underrepresent individuals who are racialized, rural residents, more disabled, less literate or lacking access to digital tools.^8,9 Barriers to research participation, including system factors such as lack of community engagement and financial burden, and individual factors such as language are discussed elsewhere.^10,11 Action is needed by researchers to address deficits in understanding of the diversity characteristics of PwMS, enhance measurement of these characteristics in intervention studies, and to improve the diversity of trial populations to ensure that findings generalize to all PwMS. Collectively, these actions can guide policy and care to optimize outcomes for all PwMS.

We sought to gain consensus on priorities regarding how future research in MS in Canada should address equity, diversity, and inclusion (EDI) using a modified e-Delphi process. A Delphi process is a structured method to achieve consensus by considering the opinion of all participants regarding large complex problems.¹² We chose a consensus approach over other methods such as surveys, interviews, or focus groups to facilitate engagement of participants with different expertise, support critical reflection on priorities via the iterative process, and ensure that all perspectives were considered.

Methods

Study design

We conducted a three-round modified e-Delphi process^13,14; the protocol was not registered. We report our findings according to ACcurate COnsensus Reporting Document (ACCORD).¹⁵

Advisory group

An advisory group provided input regarding the items included in the e-Delphi via three meetings conducted using Microsoft Teams. The advisory group (n = 7) included people with lived experience of MS from underrepresented groups (e.g., racialized, disabled, male), clinicians, researchers with expertise in MS and epidemiology, and representatives of advocacy organizations including MS Canada, and the Association of Black Social Workers.

Respondent panel

We sought a heterogeneous sample with a variety of expert perspectives, based on recommendations and evidence from the Child Health and Nutrition Research Initiative.¹⁶ Experts were: (i) People with ≥2 years of lived experience (either PwMS or family members); (ii) researchers with expertise in EDI issues in any chronic disease; and (iii) clinicians from any discipline (e.g., neurologist, nurse, physical therapist) with experience in the care of PwMS. Participants were required to be comfortable communicating in English, commit to the full Delphi process, and respond within 2 weeks for each round. The investigators and advisory group members did not participate in the panel.

Perspectives regarding optimal sample sizes for Delphi studies are varied. While some authors recommend 10–18 people per Delphi panel,¹⁷ results from very small groups are not reliable. Recent work indicates that multi-interest holder Delphi surveys stabilize (replicability >80%) with n = 60–80,¹⁸ and that subgroup analyses are acceptable with 20–30 participants per group. Therefore, we targeted 60–80 respondents for the panel, with up to 20–30 participants in each expert group.

Recruitment of panelists

We used a multi-pronged approach to recruit panel participants, aiming for participants from different Canadian provinces/territories and settings. Specifically, we advertised the study on the MS Canada Research Portal and MS Canada shared study information via their social media accounts. We distributed email invitations to members of the Canadian Network of MS Clinics and to Medical Directors of MS Clinics and via our personal networks. Invitations indicated that the invitation could be forwarded to other relevant individuals, including non-neurologist clinicians who care for PwMS (i.e., snowball recruitment). The invitations included a uniform resource locator and quick response code that directed individuals to information about the study and the e-consent, which embedded the eligibility criteria. The e-consent process captured participant names and preferred email addresses to allow contact for the iterative rounds of the Delphi process. At the end of the study, participants were offered electronic gift cards in the amount of $25 per round completed as a thank you for their time.

Preparatory research and survey development

A modified Delphi begins with a pre-populated survey rather than an item generation round, which can improve efficiency of the process.¹⁹ Pre-populated surveys can be informed by systematic or scoping reviews, review of existing conceptual frameworks, expert opinion, and focus groups. We generated our pre-populated survey based on findings from a prior scoping review about the characteristics of PwMS in Canada,⁵ the PROGRESS-Plus equity framework,^20,21 other literature examining diversity issues in research,⁶ and discussions held during virtual meetings with our advisory group. We held three virtual meetings, each with a different subgroup of advisors. Changes to the list of domains and items were made following each meeting; no changes were made following the third meeting thus no further meetings were scheduled.

The facilitator guided the advisory group through a discussion organized according to five domains: (i) Measurement and reporting of diversity characteristics in research studies, (ii) including diverse research populations, (iii) role of funders (e.g., MS Canada), (iv) role of publishers/editors, and (v) building capacity (training). The first two domains were derived from findings of a prior scoping review that most MS research studies in Canada reported only age and/or sex,⁵ and that prior research had not been sufficiently inclusive. The remaining domains were derived by considering the individuals/organizations who could effect change in existing approaches.

First, the facilitator reviewed the findings of the scoping review, then explained the Delphi process. Second, the facilitator described the domains and the rationale for their inclusion. Advisors were asked to identify domains that should be added or removed; no domains were added or removed. Next, potential items within each domain were discussed, including potential language that could be used in a survey. The potential items for measurement and reporting of diversity characteristics in research studies began with the PROGRESS-Plus equity framework. PROGRESS-Plus refers to Place of residence, Race/ethnicity/culture/language (e.g., Racial background, Ethnic background), Occupation (e.g., occupation), Gender/Sex (gender identity, sex assigned at birth, sexual orientation), Religion (e.g., religious affiliation), Education, Socioeconomic status (e.g., annual household income), Social capital (e.g., marital status, family size, social connectedness), and Plus characteristics such as age, ability, sexual orientation and immigrant status.^20,21 The advisory group recommended clarifying terminology, particularly related to social capital by adding marital status, family size, and social connectedness as modifiers. They also suggested adding residence on or off reserve (land set aside by federal government for use by a First Nation group) as a place of residence in addition to urban/rural, and community/nursing home setting. The term ability rather than disability was suggested to frame this characteristic from a positive perspective.

With respect to including diverse populations in research, the discussion began with strategies proposed in a workshop aimed at improving inclusivity of clinical trial populations in MS.⁶ These items included use of formal diversity plans, engagement with community groups, training of research teams regarding: cultural sensitivity and bias, use of tailored materials, addressing financial and logistical barriers to participation of underrepresented groups, and minimizing participant burden.⁶

With respect to the role of funders domain, the advisory group agreed that funders, particularly MS Canada as a core funder of MS research in Canada, had an important role in setting priorities related to EDI in MS research. Apart from setting clear guidelines for addressing diversity issues in research proposals (provided as an example), all other items were proposed by the advisory group including providing funding incentives for projects that prioritize diversity and inclusion, allowing research budgets to include funds to enhance recruitment and retention of diverse populations, supporting community engagement including co-design, requiring researchers to report diversity characteristics of their study populations in annual reports and publications, and funding training programs for researchers on the importance of diversity and how to recruit and retain diverse participants. The advisory group emphasized the importance of funders supporting engagement with the community using co-design principles; thus, this was included in the definition of community engagement (engagement with community groups including co-design).

With respect to the role of publishers/editors, we included these items: enforce guidelines that require authors to report diversity characteristics, provide training for authors, reviewers, editors on importance of reporting diversity characteristics, require authors to describe limitations in their data collection related to diversity characteristics, ensure diversity in editorial boards and peer reviewers, and use double blind peer review. The advisory group did not recommend additions or changes to these items.

The advisory group felt strongly that training current and future researchers was important to build capacity to improve how EDI is addressed in MS research. The facilitator provided the following terms or examples of items to begin the discussion: unconscious bias, sex and gender-based analysis, and EDI. Ultimately, the topics identified as relevant for training included: implicit bias, sex and gender-based analysis, cultural competency, community engagement, ethical implications of diversity in research, legislation and policy, data governance, development of methods and recruitment strategies to engage diverse populations, and best practices for measuring and reporting findings related to diversity and intersectionality.

Upon the conclusion of the advisory group discussion sessions, the research team finalized the statements for inclusion in the Delphi survey. Five people pilot tested the Round 1 survey before distribution (see Table e1, first column for the list of items included). The people who pilot tested the survey were chosen to reflect the three expert groups (people with lived experience, EDI experts, clinicians, and the lay public); they were not members of the advisory panel nor study participants.

Statements

For each priority statement, the panel members indicated the degree of importance or agreement on a seven-point linear (numeric) scale (with three anchors: 1 = aspirational, 4 = achievable, 7 = core). While Likert-type items are often used in Delphi processes, linear scales are easier to interpret and have better mathematical properties because the distance between responses is equal. Definitions for core, achievable and aspirational were adapted from those used in an e-Delphi process for MS care.²² Specifically, core was defined as should readily be achieved by most MS research teams/ organizations in Canada and will provide a minimum standard. Achievable was defined as a realistic target for most MS research teams/organizations and reflects a good approach to EDI. Aspirational was defined as achievable by only a few MS research teams but should set the standard for high-quality research.

Round 1 was open for two weeks. Participants could propose new items in each domain of the survey using an open-ended text field. Although all participants received the same survey, we used the eligibility criteria from the consent process to assign people in expert groups for the summary process.

In Round 2, we added items recommended in Round 1 comments and retained all items that had not reached consensus across all three expert groups in Round 1 (see Table e2 for the list of items included). Each participant received the summarized responses of the three expert groups and their own prior responses. This round was open for two weeks.

Items that had reached consensus in all 3 expert groups in Round 2 were removed from Round 3 (see Table e3 for the list of items included). Each participant received the summarized responses of each the three expert groups, the summarized combined responses of the three groups, and their own prior responses. This round was open for two weeks.

We undertook 3 rounds. Additional rounds were not included because attrition rises when there are more rounds,¹⁹ and progress toward consensus slows with continued iterations.

Defining consensus

A recent methodological study suggested that items using a 7-point linear scale with a standard deviation (SD) ≤ 1.0 provides a probability of incorrectly concluding consensus of 7.4% if the sample size is at least 10²³; thus we chose an SD ≤1.0. Items with an SD of >1.0 to 1.5 in any subgroup were retained for inclusion in the next Delphi round. Stability indicates the consistency of responses between successive rounds,²⁴ and was defined as a concordance statistic between rounds of ≥0.75 (good agreement) across all participants and little shift in central tendency as shown by kernel density plots.

Data management

The Nova Scotia Health Research Ethics Board approved this study. All participants provided electronic consent prior to initiation of study procedures using the Research Elecronic Data Capture (REDCap) E-consent framework. We collected and stored survey data through REDCap, a secure web application.²⁵ To maximize response rates, we sent two reminders for each survey. The first reminder was sent after seven days, and the second was sent 24 h before the survey closure.

Analysis

After each Delphi round, we used descriptive statistics (means, medians, SD and interquartile ranges) to generate the summary feedback data for inclusion in the subsequent rounds.²⁶ Mean and median values provide a group aggregate rank of importance, and converge as consensus improves. SD and interquartile range will decrease. We used a Shrout–Fleiss intraclass correlation coefficient (ICC) for within-participant agreement in ratings between rounds.²⁶ ICC are interpreted as poor (<0.50), moderate (0.50–0.75), good (>0.75–0.90), and excellent (>0.90).²⁷ We used data from the final round to illustrate which priorities or statements achieved consensus. We also report priorities or statements that nearly reached consensus (SD ≤1.3).

Results

Overall, 97 people consented to participate, of whom 87 (90.0%) completed the first survey. Of those 87, 82 (94.2%) completed the second, and 81 (93.1%) completed the third. The 87 participants included 36 people with lived experience, 28 researchers with EDI expertise, and 23 clinicians with experience in MS.

Based on the threshold of an SD ≤1.0, no Round 1 items reached consensus in any expert group (Table e1). Figure 1 shows the distribution of responses for each item by group. Based on participant recommendations in the Round 1 survey, we added 11 items to the Round 2 survey (Table e4) and split ability into physical and mental ability.

Figure 1.

Violin plots showing response distributions by expert group for (A) measurement and reporting of diversity characteristic domain (B) all other domains Y axis: Consensus level 1 = aspirational, 4 = achievable, 7 = core. Violin plot: Combines features of box plot and kernel density plot. Width of the plot indicates frequency of data points. The middle of each density curve is a box blot showing the median, first and third quartiles.

After Round 2, four items were dropped based on reaching consensus in all expert groups including two for funders (setting clear guidelines for addressing diversity issues in research proposals; funding training programs for researchers on the importance of diversity and effective recruitment and retention of diverse participants); one for publishers/editors (ensuring diversity in editorial boards/peer reviewers); and one for training (training in best practices for measuring and reporting findings related to diversity and intersectionality) (Table e2). Several other items reached consensus in the EDI researcher and clinician expert groups but not in the lived experience group in the measuring and reporting diversity domain, recruiting diverse populations domain, role of funders domain, role of publishers/editors domain, and training domain (Table e2).

After Round 3, 19 of 29 items related to measuring and reporting diversity characteristics reached consensus (Figure 2), of which 4 reached consensus in all groups, 12 reached consensus in two groups, and 3 reached consensus in 1 group (Table e3). The remaining items nearly reached consensus (SD 1.1–1.3). The consensus level (score) did not differ across expert groups for most items. Mean ratings were statistically significantly different for ethnicity, gender identity, sexual orientation, number of dependent children, and immigrant status with the EDI expert group giving higher ratings, but the magnitude of these differences was less than 1 point on the 7-point scale.

Figure 2.

Consensus level for priorities for measuring and reporting characteristics of study populations*. Boxplot shows minimum, first quartile, median, third quartile and maximum values. *Items that reached consensus in only 2 of 3 expert groups: Residence (on reserve vs. off reserve [where reserve refers to land set aside by federal government for use of a First Nation group), occupation, employment status, gender identity, social capital (family size), number of dependent children, physical ability level, multiple sclerosis (MS) severity level, home ownership, food security, availability of insurance (indicated by green color). Items that reached consensus in only 1 expert group: Social capital (marital status), age, and mental ability level (indicated by gold color). See Table e3 for details.

With respect to enhancing inclusion of diverse populations in research, all five items reached consensus overall (Figure 3) with two reaching consensus in all groups (Table e3), generally at the achievable level. Home ownership and family size were rated closer to aspirational (3.0). Urban/rural residence, race, ethnicity, age, and insurance availability were rated closer to core (≥5.0). No statistically significant differences across groups were observed.

Figure 3.

Consensus level for priorities for recruiting diverse populations*. Boxplot shows minimum, first quartile, median, third quartile and maximum values. *Items that reached consensus in only 2 of 3 expert groups: Develop formal diversity plan for participant recruitment & retention, address barriers to support participation in a study, develop study materials appropriate to different populations. See Table e3 for details.

For funders, all seven items had reached consensus overall by the end of Round 3 (Figure 4), generally in the achievable range. Three of the items reached consensus in all expert groups while the remainder reached consensus among the lived experienced and EDI researcher groups (Table e3). Clinicians rated funding research about health inequities as achievable (4.0), while the lived experience and EDI experts rated this item in the achievable range but closer to core (5.0).

Figure 4.

Consensus level for priorities for funders*. Boxplot shows minimum, first quartile, median, third quartile and maximum values. *Items that reached consensus in only 2 of 3 expert groups: support community engagement including co-design, fund research about health inequities in multiple sclerosis (MS). See Table e3 for details.

With respect to the role of publishers/editors, all items reached consensus overall by the end of Round 3, all at the achievable level (Figure 5). Three items reached consensus in all expert groups, and three reached consensus in two groups. Clinicians rated reporting of diversity of editorial boards (mean 4.1) and reporting how well a journal's articles reported diversity lower (mean 4.0) than the lived experience and EDI expert groups for which mean ratings were 0.7–0.8 points higher.

Figure 5.

Consensus level for publisher/editor priorities*. Boxplot shows minimum, first quartile, median, third quartile and maximum values. * Items that reached consensus in only 2 of 3 expert groups: use double blind peer review, summarize how well the journal's articles report diversity characteristics. See Table e3 for details.

With respect to training, 7 of 10 items reached consensus by the end of Round 3, generally at the achievable level (Figure 6); the remaining items nearly reached consensus (SD = 1.1). Four items reached consensus in all expert group, two items reached consensus in two of the three groups, and three items reached consensus among clinicians.

Figure 6.

Consensus level for training priorities*. Boxplot shows minimum, first quartile, median, third quartile and maximum values. * Items that reached consensus in only 2 of 3 expert groups: cultural competency, how to decolonize research methods. See Table e3 for details.

Using graphical methods and ICC, responses to most items that achieved consensus were stable with ICC ≥0.75 (Table 1 shows all items, Figures e1–e46 show items that reached consensus), although we noted some exceptions where the ICC fell in the moderate (0.50 to <0.74) range within each domain despite little change in central tendency (Figures e1–e45). One ICC fell below the moderate range, for training in best practices in measuring and reporting diversity, despite little change in central tendency (Figure e46). The ICC varied more across the three groups.

Table 1.

Intraclass correlation coefficients (ICC) between rounds across all expert groups.

Variable	ICC
Measurement, Reporting & Analysis Of Diversity Characteristics
^Place of residence (region) urban vs. rural	0.66
^Place of residence (reserve) on-reserve vs. off-reserve	0.80
^Place of residence (setting) community vs. home care vs. long-term care	0.71
^Racial background	0.68
^Ethnic background (cultural heritage & ancestry)	0.78
^Occupation	0.77
^Employment status	0.75
^Gender identity	0.84
^Education	0.74
^Social capital—Marital status	0.82
^Social capital—family size	0.77
^Number of dependent children	0.83
^Age	0.88
^Physical ability level	0.77
^Mental ability level	0.72
^MS Severity	0.69
^Home ownership	0.61
^Food security	0.70
^Availability of private insurance	0.77
Language	0.80
Sex assigned at birth	0.77
Religion	0.84
Socioeconomic status	0.87
Social capital—social connectedness	0.87
Coexisting (comorbid) health conditions	0.80
Sexual orientation	0.72
Immigrant status (country of birth, citizenship status)	0.74
Time since immigration	0.81
Including Diverse Research Populations
^Develop formal diversity plan for participant recruitment & retention	0.70
^Engage with community	0.75
^Address barriers to support participants to be in a study	0.68
^Minimize burden in study procedures	0.67
^Develop study materials appropriate to different populations	0.67
Role Of Funders
^#Set clear guidelines for addressing diversity in research proposals	0.55
^Provide funding incentives	0.67
^Allow research budgets to include funds to enhance recruitment/retention of diverse populations	0.80
^Support community engagement including co-design	0.80
^Require researchers to report diversity characteristics of their study populations in annual reports/ publications	0.87
^Fund research about health inequities in MS	0.81
^#Fund training programs for researchers on the importance of diversity	0.47
Role Of Publishers & Journal Editors
^Enforce guidelines that require authors to report diversity characteristics	0.81
^Provide training for authors, reviewers, editors on importance of reporting diversity characteristics	0.68
^#Ensure diversity in editorial boards and peer reviewers	0.52
^Require authors to describe limitations in their data collection related to diversity characteristics	0.80
^Use double blind peer review	0.89
^Report diversity characteristics of editorial boards and peer reviewers	0.85
^Summarize how well the journal's articles report diversity characteristics	0.78
Training
^Cultural competency	0.91
^Community engagement	0.80
^Ethical implications of diversity in research	0.79
^Legislation and policy (legal and policy frameworks related to diversity, anti-discrimination)	0.86
^Development of methods and recruitment strategies to engage diverse populations	0.69
^#Best practices for measuring and reporting findings related to diversity and intersectionality	0.46
^How to decolonize research methods (recognize and address biases that prioritize Western perspectives)	0.79
Implicit bias	0.83
Sex and gender-based analysis	0.65
Data governance (engagement, ownership, control, access, possession, protection of data collected from a community)	0.37

EDI = equity, diversity and inclusion, MS = multiple sclerosis.

#For items that reached consensus after round 2, the intraclass correlation coefficient is for Rounds 1 and 2. For items that reached or nearly consensus after Round 3, the intraclass correlation coefficient is for Rounds 2 and 3. ^Item reached consensus.

Discussion

This national modified e-Delphi study engaged people with lived experience of MS, clinicians, and researchers with EDI expertise to prioritize how future research in MS in Canada addresses EDI. Overall, 45 items reached consensus including 19 in the measurement and reporting domain, 5 for including diverse research populations, and 7 each for the role of funders, role of publishers/editors, and training. Nearly all items that reached consensus were considered achievable priorities. Only age was considered a core characteristic for measurement reporting. We do not know why only age was rated as a core (minimum standard) characteristic and several other characteristics (urban/rural residence, race, ethnic background, gender identity, availability of private insurance and setting) were rated between achievable and core, although closer to achievable. Based on a prior scoping review we can speculate that the frequent use of secondary data sources in MS research in Canada played a role⁵; these data sources often lack information about personal characteristics such as race, ethnicity, gender identity and availability of private insurance. Cultural biases and concerns regarding resources required could also play a role.

Multiple studies have emphasized the lack of diversity in research populations in the MS field and other fields, and insufficient reporting of participant characteristics related to health equity.^5,8,9,28 The PROGRESS and later PROGRESS-Plus framework was proposed to assist with the collection and reporting of information relevant to health equity. A recent scoping review of MS studies that reported characteristics related to PROGRESS-Plus found that 92% reported age, and 78% reported sex or gender.²⁹ The least reported characteristics were race (14%) and religion (9%).²⁹ That review emphasized the need to identify a standardized, minimum dataset of equity-relevant data. Our work takes a novel step toward achieving consensus regarding the minimum dataset in the MS field. We found that most diversity characteristics were considered achievable to measure and report; only age was considered core.

A working group of the National Institute for Health and Care Research's Clinical Research Network developed a consensus-based framework to integrate EDI into clinical research design to enhance participation of underrepresented groups.³⁰ Some of their recommendations overlapped with priorities that we identified, including collecting and analyzing diversity data of study participants, community engagement, use of strategies to enhance recruitment of underrepresented groups by mitigating barriers, and enhancing the competency of research staff regarding EDI.³⁰ The identified priorities are also consistent with those recommended by the International Advisory Committee on Clinical Trials in MS to enhance diversity in MS trials,⁶ but provide formal consensus-informed support for those recommendations which was lacking.

A 2020 commentary emphasized the need to embed anti-racism in Canadian Schools of Public Health.³¹ Multiple gaps were identified within existing curricula and several recommendations for change were made.³¹ Several of these overlapped with priorities that emerged from our study including the role of colonialism in health inequities, and the importance of teaching anti-colonial and anti-racist approaches, and the principles Ownership, Control, Access and Possession (OCAP, created by First Nations) and Engagement Governance Access and Protection Framework (EGAP, created by the Black Health Equity Working Group).³² The importance of community partnership when engaging in these efforts was underscored.

Respondents clearly indicated a role for publishers and journal editors in supporting EDI through diversity in editorial boards, accountability measures, and enforcing guidelines regarding reporting of participant characteristics and study limitations. Some of these recommendations are consistent with the most recent version of the “Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly Work in Medical Journals,³³ which suggest that editors should engage a “diverse array of authors, reviewers, editorial staff, editorial board members, and readers.” The inclusion of representative populations is encouraged, along with reporting of age, sex, ethnicity and other relevant (undefined) demographic characteristics. Accountability measures were not discussed although our study suggests these are an achievable addition to those recommendations. Initial implementation could include measuring and publicly reporting the diversity of editorial boards.

Strengths of this study included the use of an advisory group with participants from underrepresented groups in MS research, multi-interest holder panel, high response rates, and broad set of domains considered. However, limitations should also be considered. To preserve anonymity in a small research community, we did not capture characteristics of study participants beyond their classification into one of three expert groups, so we do not know how representative the panel was, although we sought participants across Canada. The use of an e-Delphi enhanced geographic reach but may have excluded individuals without reliable internet access or the technical skills to complete an online survey. We did not include all potential priorities to address EDI³⁴ to limit participant burden and minimize attrition. We focused this study on research but many of these priorities, such as community engagement and training about EDI, are also relevant to quality improvement initiatives for health care inequities.³⁵ The study design precludes an understanding of the reasons, such as sensitivity of the item, between the slight differences observed between groups for some items. That most items reached consensus at or near anchor of the achievable level may reflect a cognitive bias, specifically central tendency bias, related to adding a third central anchor to a linear numeric scale. However, the use of a heterogeneous, multi-interest holder panel can mitigate such anchoring effects. Including participants with high expertise and anonymity are also intended to mitigate cognitive bias. Since most items reached consensus at the achievable level, further work is needed to determine which items should be addressed first, and to identify barriers and facilitators to implementation.

The findings of this study provided consensus priorities regarding how future MS research in Canada addresses EDI that can be adopted by researchers, funders, publishers and educators. We hope that adoption of these priorities will begin to address limitations in current approaches being used.

Supplemental Material

sj-docx-1-mso-10.1177_20552173251398975 - Supplemental material for Setting priorities for how future research in multiple sclerosis in Canada considers equity, diversity and inclusion: A modified e-Delphi study

Supplemental material, sj-docx-1-mso-10.1177_20552173251398975 for Setting priorities for how future research in multiple sclerosis in Canada considers equity, diversity and inclusion: A modified e-Delphi study by Ruth Ann Marrie, Afolasade Fakolade, Colleen J Maxwell, Dalia L Rotstein, Helen Tremlett, E. Ann Yeh and Marcia Finlayson in Multiple Sclerosis Journal – Experimental, Translational and Clinical

Footnotes

Acknowledgments

The authors thank the advisory group for their time and input (Christina Wolfson, Yvonne/Tade Oni, Julie Petrin, Rheanna Robinson, John Fisk, Rajean Willis for the Association of Black Social Workers).

Consent to participate

All participants completed an e-consent form via REDCap before participation.

Data availability

Data are held at Nova Scotia Health. Data may be available with appropriate ethical approvals and data sharing agreements.

Declaration of conflicting interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Ruth Ann Marrie receives research funding from: CIHR, MS Canada, National Multiple Sclerosis Society, CMSC, The Arthritis Society, US Department of Defense, UK MS Society, Public Health Agency of Canada, Brain Canada, MMSF, and CHRIM and is a co-investigator on studies receiving funding from Biogen Idec, Roche Canada and the Pfizer Foundation. She is supported by the Multiple Sclerosis Clinical Research Chair (Dalhousie University). Afolasade Fakolade is an investigator on projects funded by MS Canada, Social Sciences and Humanities Research Council, Queen's University, National Research Council of Canada, and Government of Canada's New Frontiers in Research Fund. She has received consulting or teaching honoraria from Novartis. Colleen J Maxwell receives research funding from a University of Waterloo Research Chair, CIHR, MS Canada, National Multiple Sclerosis Society, CMSC and the Public Health Agency of Canada. Dalia L Rotstein receives research funding from MS Canada, the National MS Society, Canada's Drug Agency, Li Ka Shing Knowledge Institute of Unity Health Toronto, the University of Toronto Division of Neurology, Alexion, Roche Canada, and Peter and Susan Gordon. She has received consulting or speaker fees from Alexion, Amgen, Biogen, EMD Serono, Novartis, Roche, Sanofi Aventis and Touch International Medical Education. Helen Tremlett has in the last five years, received research support from the Canada Research Chair Program, the National Multiple Sclerosis Society, the Canadian Institutes of Health Research, MS Canada, the Multiple Sclerosis Scientific Research Foundation and the EDMUS Foundation (`Fondation EDMUS contre la sclérose en plaques'). In addition, in the last five years, has had travel expenses or registration fees prepaid or reimbursed to present at CME conferences or attend meetings (e.g., as a member of the International Advisory Committee on Clinical Trials in Multiple Sclerosis) from the Consortium of MS Centres (2023), the Canadian Neurological Sciences Federation (2023), National MS Society (2022-25), ECTRIMS/ ACTRIMS (2017-2025), MS Canada (2023, 2025) and the American Academy of Neurology (2019). Speaker honoraria are either declined or donated to an MS charity or to an unrestricted grant for use by HT's research group. E. Ann Yeh has received research funding from NMSS, CMSC, CIHR, NIH, OIRM, SCN, CBMH Chase an Idea, SickKids Foundation, Rare Diseases Foundation, MS Scientific Foundation, McLaughlin Centre, Leong Center, Peterson Foundation, Garry Hurvitz Centre for Brain and Mental Health, Canada's Drug Agency. Investigator initiated research funding from Biogen. Scientific advisory: Hoffman-LaRoche, Alexion. DSMB: Pipeline Therapeutics.  Speaker honoraria: Biogen, JHU, Saudi Epilepsy Society, NYU, MS-ATL; ACRS, PRIME, CNPS. Co-Editor in Chief, MSARD. Governing Council: CANTRAIN. Steering Committee: Rare-Kids CAN. Marcia Finlayson has research support from MS Canada and the Women's Giving Circle of University Hospitals Kingston Foundation and is a co-investigator on studies funded by the Canadian Institutes of Health Research, the Social Sciences and Humanities Research Council, MS Canada, the Patient Centered Outcomes Research Institute, and the International Progressive Multiple Sclerosis Alliance.

Ethical considerations

The Nova Scotia Health REB approved the study (ROMEO # 1031247).

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by MS Canada (EGID 1186492).

ORCID iDs

Ruth Ann Marrie

Afolasade Fakolade

Helen Tremlett

E. Ann Yeh

Marcia Finlayson

Supplemental material

Supplemental material for this article is available online.

References

Rotstein

Maxwell

, et al. High prevalence of comorbidities at diagnosis in immigrants with multiple sclerosis. Multiple Sclerosis J 2021; 27: 1902–1913.

Rotstein

Marrie

, et al. Health service utilization in immigrants with multiple sclerosis. PLoS One 2020; 15: e0234876.

Marrie

Elliott

Marriott

, et al. Dramatically changing rates and reasons for hospitalization in multiple sclerosis. Neurology 2014; 83: 929–937.

Harding

Wardle

Carruthers

, et al. Socioeconomic status and disability progression in multiple sclerosis: A multinational study. Neurology 2019; 92: e1497–ee506.

Marrie

Fakolade

Linton

, et al. Characterizing the diversity of the multiple sclerosis population in Canada: A scoping review. Mult Scler J Exp Transl Clin 2025; 11: 20552173251321814.

Marrie

Chataway

Bierer

, et al. Enhancing diversity of clinical trial populations in multiple sclerosis. Multiple Sclerosis J 2023; 29: 1174–1185.

Onuorah

H-M

Charron

Meltzer

, et al. Enrollment of non-white participants and reporting of race and ethnicity in phase III trials of multiple sclerosis DMTs. A Syst Rev 2022; 98: e880–ee92.

Finlayson

Al-Mashita

Sandhu

. Participant diversity in clinical trials of rehabilitation interventions for people with multiple sclerosis: A scoping review. Mult Scler 2023; 29: 1149–1157.

Fakolade

Akbar

Mehelay

, et al. Mapping two decades of multiple sclerosis rehabilitation trials: A systematic scoping review and call to action to advance the study of race and ethnicity in rehabilitation research. Mult Scler Relat Disord 2023; 72: 104606.

10.

George

Duran

Norris

. A systematic review of barriers and facilitators to minority research participation among African Americans, Latinos, Asian Americans, and Pacific Islanders. Am J Public Health 2014; 104: e16–e31.

11.

Hamel

Penner

Albrecht

, et al. Barriers to clinical trial enrollment in racial and ethnic minority patients with cancer. Cancer Control 2016; 23: 327–337.

12.

Barrett

Heale

. What are Delphi studies? Evidence Based Nurs 2020; 23: 68–69.

13.

Toronto

. Considerations when conducting e-Delphi research: A case study. Nurse Res 2017; 25: 10–15.

14.

Shang

. Use of Delphi in health sciences research: A narrative review. Medicine (Baltimore) 2023; 102: e32829.

15.

Gattrell

Logullo

van Zuuren

, et al. ACCORD (ACcurate COnsensus reporting document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLoS Med 2024; 21: e1004326.

16.

Rudan

Yoshida

Wazny

, et al. Setting health research priorities using the CHNRI method: v. quantitative properties of human collective knowledge. J Glob Health 2016; 6: 010502.

17.

Okoli

Pawlowski

. The Delphi method as a research tool: An example, design considerations and applications. Inf Manage 2004; 42: 15–29.

18.

Manyara

Purvis

Ciani

, et al.

Sample size in multistakeholder Delphi surveys: At what minimum sample size do replicability of results stabilise?

J Clin Epidemiol 2024; 174: 111485.

19.

Keeney

Hasson

McKenna

. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud 2001; 38: 195–200.

20.

O'Neill

Tabish

Welch

, et al. Applying an equity lens to interventions: Using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. J Clin Epidemiol 2014; 67: 56–64.

21.

Kavanaugh

Oliver

Lorenc

. Reflections on developing and using PROGRESS-Plus. 2008;2:1–3.

22.

Hobart

Bowen

Pepper

, et al. International consensus on quality standards for brain health-focused care in multiple sclerosis. Mult Scler 2019; 25: 1809–1818.

23.

Franc

Hung

KKC

Pirisi

, et al. Analysis of Delphi study 7-point linear scale data by parametric methods: Use of the mean and standard deviation. Methodol Innovations 2023; 16: 226–233.

24.

Dajani

Sincoff

Talley

. Stability and agreement criteria for the termination of Delphi studies. Technol Forecast Soc Change 1979; 13: 83–90.

25.

Harris

Taylor

Thielke

, et al. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42: 377–381.

26.

Holey

Feeley

Dixon

, et al. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med Res Methodol 2007; 7: 52.

27.

Sim

Wright

. The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Phys Ther 2005; 85: 257–268.

28.

Green

Trivedi

Hsu

, et al. Despite the FDA’s five-year plan, black patients remain inadequately represented in clinical trials for drugs. Health Aff 2022; 41: 368–374.

29.

Karran

Cashin

Barker

, et al. Using PROGRESS-plus to identify current approaches to the collection and reporting of equity-relevant data: A scoping review. J Clin Epidemiol 2023; 163: 70–78.

30.

Gedela

Wong

Balendra

, et al. Embedding equity, diversity and inclusion processes within clinical trials and health and social care research. BMJ Open 2025; 15: e091807.

31.

Banerjee

Tan

Boston-Fisher

, et al. Embedding anti-racism in schools of public health: A pathway to accountability for progress towards equity. Can J Public Health 2023; 114: 872–877.

32.

Black Health Equity Working Group . Engagement, governance, access, and protection (EGAP): A data governance framework for health data collected from Black communities. blackhealthequity.ca 2021.

33.

International Committee of Medical Journal Editors . Recommendations for the conduct, reporting, editing and publication of scholarly work in Medical Journals ICMJE, 2025.

34.

Ahmad

Sabat

Trump-Steele

, et al. Evidence-Based strategies for improving diversity and inclusion in undergraduate research labs. Front Psychol 2019; 10: 2019.

35.

Burra

Auguste

, et al. Equity in action: A scoping review and meta-framework for embedding equity in quality improvement. BMJ Qual Saf. 2025; Online ahead of print.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.49 MB

Setting priorities for how future research in multiple sclerosis in Canada considers equity,diversity and inclusion: A modified e-Delphi study

Abstract

Background

Objective

Methods

Results

Conclusions

Keywords

Introduction

Methods

Study design

Advisory group

Respondent panel

Recruitment of panelists

Preparatory research and survey development

Statements

Defining consensus

Data management

Analysis

Results

Discussion

Supplemental Material

sj-docx-1-mso-10.1177_20552173251398975 - Supplemental material for Setting priorities for how future research in multiple sclerosis in Canada considers equity, diversity and inclusion: A modified e-Delphi study

Footnotes

Acknowledgments

Consent to participate

Data availability

Declaration of conflicting interests

Ethical considerations

Funding

ORCID iDs

Supplemental material

References

Supplementary Material