Abstract
We used engagement marketing and human-centered design principles to cocreate a digital decision support tool for research participation with LGBTQIA+ community members to help them make an informed decision about joining the All of Us Research Program. Building on results from the research phase, we conducted eight problem validation and solutioning workshops with 48 LGBTQIA+ community members. Community members validated barriers to engagement with All of Us and brainstormed 47 potential digital solutions. We developed potential solutions into 27 concepts (descriptive text and visual storyboards) and assessed acceptability, appropriateness, feasibility, and engagement in a set of 10 concept testing workshops with 57 community members. We developed one of the highest rated concepts, the “Decide Later Tool,” into a prototype and tested it with 45 LGBTQIA+ community members and 14 community advisory group members to assess acceptability, appropriateness, feasibility, usability, and engagement. Prototype testing participants indicated that the tool provides information to help with decision making, provides a clear value or benefit to them, was designed for someone like them, provides the right amount of information, and is easy to use; they also offered constructive feedback to improve it. Across the design and development phases, community members indicated that the process of engaging them demonstrated integrity, competence, dependability, trust, and collaboration; fostered a sense of connection to All of Us; and will enhance future engagement with All of Us. Our next steps are to develop the prototype into a fully functioning web tool and pilot test it in community and health care settings.
Introduction
The National Institutes of Health’s
To address barriers to research participation and increase representation of LGBTQIA+ people in All of Us, researchers need to take an equity-centered approach and engage community members in the research process. Engagement marketing involves active and intentional collaboration with community members to foster connections, build trust, and facilitate long-term relationships, which is in contrast to a traditional research model that implies a hierarchical power relationship between researchers and participants. 5 Human-centered design focuses on designing solutions to address complex problems by engaging end users throughout the research, design, development, and delivery of the solution. 6
Our primary goal was to engage LGBTQIA+ community members as cocreators, drawing on engagement marketing and human-centered design principles, in the design and development of a culturally appropriate, 7 acceptable, 7 feasible, 7 and usable 8 digital decision support tool for making an informed decision about enrolling in All of Us. We also sought to determine whether community members viewed the approach of engaging them in the design and development process as one that demonstrated integrity, trust, collaboration, and fostered a sense of connection to All of Us. 9 There is a body of research and documentation illustrating the ways that LGBTQIA+ communities have built digital cultures 10 and used digital spaces to organize socially and politically.11–15 This context underscores the importance of centering LGBTQIA+ community members’ input throughout the design and development of digital products to draw on experiences and existing practices and to reduce the risk of the tool not being culturally relevant or responsive.
Materials and Methods
The study was reviewed and determined to be exempt from Institutional Review Board review by the National Institutes of Health’s
We engaged adult LGBTQIA+ community members at three time points across the design and development phases: (1) problem validation and solutioning, (2) concept testing, and (3) prototype testing (Figure 1). 1 Each time, we observed, recorded and measured cocreation, user testing, and engagement constructs both qualitatively and quantitatively (Table 1).
Inclusive Approach to Engaging Community Members in the Design and Development of a Digital Decision Support Tool. Dev, Development; C1–C6, Development Cycles. Source: Adapted from Lewis 2022. 5
Definitions of Cocreation, User Testing and Engagement Constructs Measured Across the Design and Development Phases and Activities
Problem validation and solutioning
We conducted eight virtual 2-hour workshops with 48 community members in June 2022 to validate barriers to engaging with All of Us and to generate potential solutions to address those barriers. We presented workshop participants with a list of barriers that were identified in the earlier research phase (Figure 1), asked them which, if any, resonated with them and why, and to add other barriers to the list. Each workshop participant chose the barrier they felt was the most important to address, independently brainstormed six ideas to address that barrier, and then discussed their idea with the group. Finally, they completed an online survey to assess their engagement with the process and with All of Us and were compensated with a digital gift card ($100).
Concept testing
Using input from the problem validation and solutioning workshops, we developed concepts (descriptive text and visual storyboards, see Figure 2 for example) for potential digital solutions to address barriers to joining All of Us. We then conducted a set of 10 virtual 2-hour workshops with 57 community members in July and August 2022 to test the concepts. In each concept testing workshop, we shared five concepts with the group (one at a time) and asked open-ended questions to elicit feedback. After discussing the concept as a group, we used the polling feature on Zoom to measure acceptability, appropriateness, feasibility, and potential for the idea to promote engagement with All of Us. We repeated this process for all five concepts. Workshop participants completed an online survey to assess their engagement with the process and with All of Us and were compensated with a digital gift card ($100).
The Decide Later Tool Concept and Prototype.
Prototype testing
We used input from the concept testing workshops to develop the “Decide Later Tool” concept into a prototype and conducted prototype testing (feedback sessions, content testing, and usability testing) in August 2023.
Feedback sessions
We sent mockups of the prototype to PRIDEnet’s community advisory groups 2 to review prior to holding 30-minute virtual feedback sessions with the group members (two sessions with 14 individuals across both sessions). We demonstrated the prototype and asked open-ended questions to elicit feedback. 3
Content testing
We conducted two 60-minute virtual content testing workshops with community members (12 individuals across both groups). We demonstrated the prototype and then showed its content in a series of slides. After each slide, we asked open-ended questions to elicit feedback. Finally, content testing participants completed an online survey to assess acceptability, appropriateness, and feasibility of the content overall, their engagement with the process and with All of Us, and were compensated with a digital gift card ($50).
Usability testing
We conducted 33 intercept interviews (20–25 minutes each) with community members to test the prototype’s usability on tablet computers during a 2-day Pride event (an environment similar to the tool’s intended use). We obtained verbal consent and confirmed that potential interviewees were 18 years of age or older and identified as a member of the LGBTQIA+ community before proceeding with the interviews. We briefly described the purpose of the prototype, showed the prototype’s homepage, and assessed first impressions. Next, we asked users to select either option at the bottom of the homepage (“Sign Up Today” or “Decide Later”) and to think aloud as they used the tool (repeating the process for whichever path they did not select initially). After they completed both branches, we asked a set of closed-ended questions to assess the tool’s overall acceptability, appropriateness, feasibility, and usability. Users completed an online survey on the tablet to assess engagement with the usability testing process and with All of Us, provide their individual characteristics, and were compensated with a digital gift card ($20).
Data analysis
We primarily used a deductive approach to code the qualitative data. We coded the problem validation and solutioning data for barriers/challenges and solutions identified in the prior research phase and added inductive codes for new barriers/challenges and solutions identified during the workshops. In the concept testing and prototype testing phases, we coded the data for acceptability, appropriateness, feasibility, usability (prototype testing only), and engagement, and iteratively and collaboratively added inductive subcodes as they emerged from the data. Two lead coders reviewed a small random sample of transcripts coded by each member of the coding team to confirm that codes were applied accurately and consistently (according to the agreed upon definitions), looked for redundant inductive subcodes, and identified opportunities to combine and/or consolidate subcodes. Once the data were coded, we conducted thematic analyses to identify patterns and consistencies, 16 which allowed us to quickly discern key findings across the coding constructs. We calculated descriptive statistics for the quantitative data collected in each round (Zoom polls; post-workshop surveys, post-content testing surveys, and post-usability testing surveys) and computed composite scores when appropriate.
Results
Problem validation and solutioning, concept testing, and prototype testing participant characteristics
Problem validation and solutioning, concept testing, and prototype testing participants were diverse in age, racial/ethnic identity, gender identity, and sexual orientation. 4 Most were not affiliated with All of Us, a majority reported living in the West or South, and few reported having less than some college education (Table 2).
Problem Validation and Solutioning, Concept Testing, and Prototype Testing Participant Characteristics
Some individuals participated in more than one activity (n = 34). Characteristics of these individuals are included in the columns for each activity but are only included once in the combined column.
The Combined column only includes unique individuals (n = 81 across problem validation and solutioning, concept testing, and content testing sessions); individuals who participated in more than one activity are only included once.
One individual who engaged in the problem validation and solutioning and the concept testing activities did not complete all screener questions, the ns for these calculations are reduced by one for the problem validation and solutioning, concept testing, and combined columns.
Question was “select all that apply,” percentages will not add to 100.
Individuals were asked if they were affiliated with the All of Us Research Program, including as a participant, on an advisory board, or in other capacities.
Problem validation and solutioning
Workshop participants validated concerns about personal data security and privacy, lack of awareness (of All of Us), lack of intentional outreach to the LGBTQIA+ community, misperceptions of inclusion criteria, accessibility challenges, distrust of research and/or government organizations, concerns about short- and long-term time commitment, and unclear value or benefit of participation as barriers to joining All of Us. Workshop participants also mentioned concern about signing up for All of Us in a crowded public space and wanted the opportunity to decide later (Table 3).
Qualitative Themes and Illustrative Quotes from Community Members by Activity
Direct quotes were only available from the feedback session and content testing participants; usability testing interviews were conducted at an event and unable to be audio-recorded.
Workshop participants brainstormed 47 potential digital solutions to address the barriers and challenges raised, including synchronous options (live chats, kiosks/virtual booths, and virtual events); asynchronous options (chatbots, websites, apps, Q&A, interactive FAQs, and virtual assistants); and digital media (educational videos, testimonial videos, podcasts, online games, transit ads, social media contests, and QR codes). They recommended that all digital solutions demonstrate cultural competence, include information about participant data privacy and security, include an overview of All of Us, and acknowledge past harms committed or caused by research.
Concept testing
We created 27 concepts from the 47 potential solutions generated in the problem validation and solutioning workshops. One of those concepts was the “Decide Later Tool,” designed to increase transparency, build awareness of and trust in All of Us, facilitate intentional outreach to the LGBTQIA+ community, and provide an alternative to joining All of Us onsite at a crowded, public event. The Decide Later Tool concept would enable community members to select topics about which they want to receive additional information, enter their contact information, and receive a tailored email with brief information about the topics they selected and links to a website with more detailed information about the topics. The goal is to provide community members with the information they want (tailored to their needs), the opportunity to review the information outside the context of a crowded, public event, and to make an informed decision about enrolling in the program when they are ready (Figure 2).
Mean composite scores for the 27 concepts tested in this phase ranged from 2.4 to 4.4 on a 5-point scale (data not shown); the Decide Later Tool concept had one of the highest composite scores (4.2) of all the concepts tested. Concept testing participants indicated that the Decide Later Tool idea was acceptable, appropriate, feasible, and would promote engagement with All of Us (Table 4).
Concept Testing Participants’ Evaluation of the Decide Later Tool Concept
Mean composite score on a scale of 1 (strongly disagree) to 5 (strongly agree) included the following items: I liked this idea overall; this idea provides a clear value or benefit to me; this idea was designed for someone like me; this idea will help people understand, connect or interact with All of Us; this idea would be easy to use; this idea would take too much time to use (reverse coded).
Positive feedback
Workshop participants liked the option to get more information to look at later, requesting information on a tablet rather than writing down their email or phone number, and getting the information digitally versus in a paper handout. They also thought using the tool at a public event would be feasible. Some workshop participants thought that people may be more likely to join All of Us because the tool would help them avoid feeling “forced” or “put on the spot” at an event. They also liked that it could provide information about topics such as data security, privacy, and trust (Table 3).
Constructive feedback
Some workshop participants were concerned about entering their contact information into the tool; they wanted assurance that their information would be kept secure and private. Some workshop participants worried that people would forget or lose interest in signing up for All of Us after an event. Suggestions for improvements or additions to the concept included (1) adding a QR code for users who do not want to use or touch the tablet; (2) minimizing the information users need to provide (e.g., email/phone number and first name); (3) including an option to receive information via text or email; and (4) making people aware of the tool at events so that it does not get overlooked (Table 3).
Prototype testing 5
The process to develop the Decide Later Tool concept into a prototype began in spring 2023 in an effort to address the challenges described previously and to support PRIDEnet’s engagement and enrollment efforts by automating the process for providing additional information to potential participants at All of Us community engagement events. Currently, if event attendees want more information on a specific topic that is not already covered in a general All of Us brochure, staff write down name(s) and email address(es) and follow-up individually via email, which can be a laborious process. The Decide Later Tool prototype allows community members to elect to enroll in All of Us immediately or decide later. If they elect to decide later, they can select up to six topics about which they want additional information. They will then receive a tailored email with information on those topics and a link to a webpage with additional details (Figure 2). The six topics included in the prototype were informed by results gathered during the research and design phases (Figure 1), PRIDEnet’s prior work, and PRIDEnet’s direct experience engaging with community members at All of Us events.
Positive feedback
Community members who participated in content and usability testing indicated that the Decide Later Tool prototype was acceptable, appropriate, feasible, and usable (Table 5). Prototype testing participants liked that the information was organized into distinct topics, which made the information easier to understand. Content testing participants liked that the content acknowledged that some people may be skeptical or have questions, reported that the information answered their questions, and thought the content was “straightforward” and “easy to understand.” Usability testing participants noted that the user interface was straightforward and visually appealing. They liked the layout, color scheme, and the diversity of the people shown in the images (Table 3).
Prototype Testing Participants’ Evaluation of the Decide Later Tool Prototype by Activity
Response options on 5-point Likert-type scale (1 = strongly disagree to 5 = strongly agree).
Two participants did not respond to these questions after usability testing.
Constructive feedback
Some prototype testing participants reported that the homepage of the web application did not have enough information about the overarching goal of All of Us; some were uncertain about whether All of Us is only for the LGBTQIA+ community or open to everyone; and some wanted more information about privacy and data protection to foster trust. They wanted to know that All of Us understands the risks for the community, considering the current unsafe political climate for LGBTQIA+ people. Content testing participants made several specific suggestions for changes to words or phrases in the email to better clarify meaning and improve understanding. Some usability testing participants did not like that the “Sign Up Today” and “Decide Later” buttons on the homepage were not visible on a tablet without scrolling down. On the page that allows users to choose which All of Us topics they wanted more information about, many usability testing participants did not realize they could click on the topics before moving to the next page and some had trouble clicking on the check boxes for the topics. A few usability testing participants noted that it may be less feasible for older or less digitally literate people to use the tool (Table 3).
Engagement with the design and development process and with All of Us
Across all activities in the design and development phases, community members indicated that the process of engaging them demonstrated integrity, competence, dependability, trust, and collaboration; fostered a sense of connection to All of Us; and will enhance community members’ engagement with All of Us (Table 6).
Community Members’ Engagement with the Design and Development Process and with All of Us by Phase and Activity
In each activity except for content testing, some individuals did not respond to the engagement questions, leaving a smaller n than overall group participation. Ranges are provided where differing numbers of individuals responded to each question.
Ratings for individuals who participated in more than one activity (e.g., problem validation and solutioning, concept testing, and content testing) are included in the respective columns.
Scores are reversed such that higher scores indicate individuals do think the group process is helpful.
Discussion
Drawing on engagement marketing and human-centered design principles, we engaged community members as cocreators to design and develop a digital decision support tool prototype to help LGBTQIA+ people make informed decisions about whether to enroll in All of Us. The prototype addressed barriers identified by members of the community while also addressing a challenge that engagement staff faced in the field.
We created an environment of cocreation by valuing the contributions of community members as experts, recognizing their efforts by compensating them, and providing “collaboration updates” (brief, plain language summaries of key findings and recommendations resulting from their input). 17 In turn, community members generated nearly 50 ideas for potential digital solutions to address the barriers; we developed these into 27 concepts for community input in the concept testing phase. One of the highest rated concepts was the “Decide Later Tool,” a tablet-based app designed to increase transparency, build awareness of and trust in All of Us, facilitate intentional outreach to the LGBTQIA+ community, and provide an alternative to joining All of Us onsite at a crowded, public event. We developed the Decide Later Tool concept into a prototype for further testing with the community. Community members’ feedback indicated that the prototype was culturally appropriate, acceptable, feasible, and usable. Across all activities in the design and development phases, community members indicated that the process of engaging them demonstrated integrity, trust, collaboration, and fostered a sense of connection to All of Us.
The positive results support taking an equity-centered approach that engages community members throughout the research process. This is particularly important in research involving genomics given past abuses by researchers.17,18 Our process provided a structural change to the researcher and participant relationship that fostered greater inclusion and a bidirectional relationship. 19 Cocreation with communities that have experienced systematic disadvantage addresses justice concerns raised by bioethicists concerned with genomic translational research. 20 This may be why there are increasing examples of cocreation in studies related to genomics. 21 In addition to engaging community members in the cocreation and testing process, we recommend that other research teams routinely evaluate the extent to which community members perceive their engagement in the process as a positive experience (e.g., measure the extent to which the process of engaging them demonstrates integrity, trust, collaboration, and fosters a sense of connection with the overall program or cause).
Our study had a few limitations. We recruited members of the LGBTQIA+ community to collaborate with us in the cocreation and testing process. Individuals who engaged with us may have a stronger interest in or motivation related to the topic. Therefore, they may not be representative of the larger LGBTQIA+ community, and results are not generalizable. Furthermore, we were unable to assess the extent to which the Decide Later Tool achieves short- and long-term outcomes during the design and development phases. However, we will evaluate these outcomes in the delivery phase through pilot testing. We hypothesize that digital solutions that are cocreated with community members will be more effective in achieving short- and long-term outcomes, such as raising awareness, motivating information seeking, addressing gaps in knowledge, building trust, and facilitating informed decision-making about enrollment and retention.
Conclusions
Taking an engagement marketing and human-centered design approach resulted in a culturally appropriate digital decision support tool that addresses barriers to enrollment in All of Us identified by the community while also addressing a challenge that engagement staff faced in the field. Our next steps are to develop the Decide Later Tool into a fully functioning web tool and pilot test it in community and health care settings. Finally, we heard some of the same barriers to enrollment raised in problem validation and solutioning workshops held with members of other communities underrepresented in biomedical research. Therefore, we plan to explore the possibility of adapting the Decide Later Tool for other communities by collaborating with members of those communities in a similar cocreation and testing process.
Footnotes
Acknowledgments
This work would not have been possible without the community members who collaborated with us in the design and development phases. The authors acknowledge the important contributions of the RTI International All of Us Engagement and Retention Innovator Team: Stephen Brown, Yujiemi Chisholm, Julia Chistiakova, Astrid Flores, Christel Gianci, Ivonne Headley, Sula Hood, Dustin Kingsmill, Mark Koyanagi, Gabriel Madson, Ryan Paquin, Ye Pogue, Alex Rabre, Amy Sanders, Cindy Soloe, Hadya Sow, Cheryl Velez, and Teddy Weathersbee.
Authors’ Contributions
Conceptualization: J.U. and M.L.; formal analysis: A.C., J.B., J.G.-N., and A.O.; investigation: M.F., A.J., K.B., S.D., A.C., J.B., and C.G.; methodology: J.U., M.L., A.C., J.B., J.G.-N., A.O., D.M., and A.R.; project administration: J.U. and M.L.; supervision: J.U. and M.L.; writing—original draft: J.U., M.L., A.C., and J.B.; writing—review and editing: K.B., M.F., A.J., D.M., A.R., C.G., J.G.-N., A.O., and S.D.
Human Subjects Protection
This study was reviewed by the All of Us Research Program's Research Compliance Branch and determined to be exempt from requiring IRB review. All procedures performed were in accordance with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Author Disclosure Statement
None to declare by any author.
Funding Information
This work was supported by the Division of Engagement and Outreach, The All of Us Research Program, National Institutes of Health (OT2OD028395, M. Lewis & J. Uhrig, MPIs). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.