Sage Journals: Discover world-class research

Abstract

Community participation in research involving stigmatized populations has been sub-optimal, and digital tools could potentially increase participation in qualitative research. This study aims to describe the implementation of an online chat-based FGD (Focus Group Discussion) with men who have sex with men (MSM) in China as part of formative research for the PIONEER project, determine the advantages and limitations associated with the approach, and assess the feasibility of deepening community participation in STI research. Participants were involved in four days of asynchronous FGDs on sexually transmitted diseases and answered questions about the online FGD method. Online FGDs allowed us to deepen participant engagement through bidirectional communication channels. Data from online FGDs directly informed recruitment strategies and community participation for a clinical trial. Overall, 63% (29/46) of men who had never participated in offline LGBTQ + activities joined online FGDs. Many participants (89%, 41/46) noted that online FGDs were more convenient, less socially awkward, and more anonymous than in-person qualitative research. We highlighted potential risks as well as mitigation strategies when using online FGDs. Online FGDs were feasible among this group of sexual minorities and may be particularly useful in many cities where stigma limits in-person research participation.

Keywords

men who have sex with men WeChat digital FGDs qualitative research participatory research

Introduction

Health services research studies are essential for identifying gaps in sexually transmitted infections (STI) service delivery and barriers to service acceptability and utilization. Strong community participation in the health research process is vital for yielding generalizable findings that inform interventions development and successful implementation, like identifying relevant research topics and generating people-centered insight into addressing barriers to participation (Johnson et al., 2015; Manta et al., 2020; Nyika et al., 2010). However, the participation of minority groups in health research, especially in lower and middle-income countries (LMICs), is hindered by various sociocultural factors (like stigma, discrimination and criminalizing local laws).

The need for more social science research among sexual minorities to understand their healthcare-seeking behaviours and care needs to inform interventions is still urgent to inform service scale-up in LMICs. But the social stigma attached to these groups, the fear of provider-instigated discrimination, and the local criminalization of these groups in the settings often discourage services uptake and deter them from joining in-person health research studies like traditional in-person qualitative research methods (Hollander, 2004). In-person research activities are also associated with risks like unintentional disclosure (of sexual or gender identity or health status); being identified or recognized by an acquaintance or family member; being tagged by a comment or an opinion shared (people can put a face to your comments); and losing privacy (other participants can recognize you in public), amongst others. Such risks threaten the safety and privacy of sexual and gender minority individuals and further deter their participation. This leads to less quality work with limited context, research mistrust, and lower acceptability of research findings in community settings (Fung, 2006; Holzer et al., 2014; Reza-Paul et al., 2019; Shaffer et al., 2018).

Digital strategies provide one way to decrease the risks of participating in social science research compared to conventional in-person methods. The World Health Organization and others have noted that research with sexual minorities must respect privacy and ensure that people are not outed as part of the process. Digital approaches can do this by allowing anonymous participation, decreasing the number of people accessing sensitive information, and so on (Tan et al., 2022). Regardless of risks like data breaches associated with digital tools, asynchronous and synchronous digital strategies to collect qualitative data are increasingly used. This is because of the more widespread use of digital devices and platforms that facilitate anonymous peer interactions and ease convenience (Inan et al., 2020).

For example, digital approaches provide opportunities to deepen community participation among marginalized populations at high risk of STI by offering anonymity and privacy and eliminating the need for travel to in-person study venues (Chen & Neo, 2019). This approach is especially important in the era of infectious disease pandemics like the COVID-19 outbreak, during which in-person events are primarily limited. As chat-based apps and modalities require less bandwidth than phone calls or video conferencing software, it may also help overcome issues of intermittent internet access or the lack of bandwidth in areas with weaker digital infrastructures. Using social media messaging apps (like WhatsApp and WeChat) for conducting online chat-based qualitative FGDs (Focus Group Discussion) (Chen & Neo, 2019; Colom, 2021; Neo et al., 2022), but few studies have assessed their benefits as a research method.

This study aims to describe the implementation of an online chat-based FGD with MSM in China conducted as part of formative research for t the PIONEER project, determine the advantages and limitations associated with the approach, and assess the feasibility of deepening community participation in STI research.

Methodology

Study Setting and the PIONEER Project

The PIONEER project is an ongoing ‘pay-it-forward’ gonorrhea and chlamydia testing randomized controlled trial (PIONEER) that aims to evaluate the effectiveness of two pay-it-forward strategies in promoting testing uptake compared to the standard of care (in which men pay for their tests out-of-pocket) among MSM and male STD patients in China [ClinicalTrials.gov Id: NCT05723263]. Gonorrhea and chlamydia are the most common STIs among men who have sex with men (MSM) in China. Guangzhou is China’s fourth-largest city, with an estimated population of 13 million. Estimates show that about one in 10 MSM and one in six MSM in Guangzhou have gonorrhea and chlamydia, respectively (Yang et al., 2018). However, gonorrhea and chlamydia testing rates remain low among Chinese MSM (Lin et al., 2014; Wu et al., 2019) as infection are often asymptomatic at extragenital sites. (Detels et al., 2011; Lutz, 2015). These low testing rates pose a public health concern since gonorrhea and chlamydia infection are known to increase the risk of HIV acquisition (World Health Organization, 2022).

Previous studies have shown pay-it-forward (PIF) interventions to be associated with a substantial increase in gonorrhea and chlamydia test uptake compared to standard-of-care. In the trial, Chinese MSM would receive free gonorrhea and chlamydia testing from other MSM and then be asked to donate any amount to fund subsequent tests for other MSM in the community (paying it forward). A previous quasi-experimental study and a small RCT found evidence that this helped increase testing among MSM and resulted in cost savings (Li et al., 2019; Yang et al., 2020).

Study Design

The PIONEER project’s implementation approaches and recruitment procedure were developed through co-creation to ensure adequate community engagement and ownership of the intervention. Co-creation is a community engagement methodology that aims to create a shared leadership role of program beneficiaries in the development and design of programs (Slattery et al., 2020). In co-creation, program beneficiaries are regarded as experts of their lived experience and are empowered by researchers to serve as equal design team members (Robert et al., 2021; Sanders & Stappers, 2008). The PIONEER co-creation model was guided by Leask’s principles and recommendations for co-creating public health intervention (Leask et al., 2019). See Supplementary Material 1 for more details on the PIONEER co-creation group, its formation, and their activities. As part of formative research to inform the PIONEER project design, the co-creation group recommended a focus group discussion with target beneficiaries of the project, MSM in Guangzhou, to explore their STI care needs and opinions and solicit their inputs into the project design. An online approach was selected because Covid-19 outbreaks were still sporadic, group gatherings were inconvenient, and the literature review showed chat-based FGDs were feasible. The co-creation group designed the FGD topic guide, and the self-administered survey assessed the feasibility and acceptability among participants as a secondary outcome. WeChat was preferred to other potential chat-based apps in China, given its popularity and relevance in the daily lives of Chinese individuals (Tu, 2016). WeChat is the most popular chat-based messenger and social media app in China, and by the end of 2021 recorded about 1.27 billion daily active users from around the world (Statista, 2022). The choice to use WeChat was also informed by feedback from representatives of MSM community-based organizations, who cited using the platform for activity planning and group discussions related to their services provisions and suggested it would be a good platform for these FGDs.

Study Participants

As the target beneficiaries of the PIONEER project, only MSM residing in the implementation cities were recruited for participation in the online FGDs. Eligible participants were 18 years old and above, assigned male sex at birth, had anal sex with a man at least once, spoke Mandarin Chinese or Cantonese, and lived in Guangzhou or neighboring cities such as Shenzhen, Zhuhai, Jiangmen, Foshan, or Dongguan. Given the reported varying usage patterns for social media and mobile phone applications across age groups (Rosenfeld et al., 2018), we stratified participants by age into six FGDs with eight participants per FGD, resulting in 48 participants recruited. Of these FGDs, two involved discussions among participants aged 18 to 29, two were those aged 30 to 39, and two were those aged 40 and above. FGD participants were purposively recruited based on the above demographic and behavioural characteristics to ensure diverse backgrounds.

Participants were recruited through the University of North Carolina (UNC) Project-China team’s WeChat public account. A recruitment poster (See Supplementary Material 2) was published on the WeChat public account and disseminated to other community members by sharing and forwarding the message within individual networks and also promoted with the assistance of our partner MSM-led CBO in Guangzhou. The post invited interested individuals to a link below the poster, leading them to an eligibility assessment questionnaire, an informed consent form, and a baseline survey. The research coordinator then contacted eligible participants for recruitment into the FGDs. Each participant received 200 yuan (∼30 USD) as reimbursement for full participation in all four days of the FGD.

FGD Process

The WeChat FGDs took place asynchronously over four days. Past studies using chat-based apps with a mix of synchronous and asynchronous chat sessions over five days recommend that asynchronous approaches over a shorter period would optimize participation in these chat-based FGDs (Neo et al., 2022). Discussions were informed by the co-created topic guide, which focused on a single topic for each day which we developed in consultation with individual MSM community members and a co-creation group. The discussion topics (see Table 1) included questions about defining and characterizing the MSM community, digital spaces, health services for MSM, perspectives on HIV and other STIs, and feedback on the pay-it-forward intervention.

Table 1.

Abbreviated Topic Guide for WeChat Focus Group Discussions.

Day of Discussion	Topics
Day 1	Defining the Chinese men who have sex with men (MSM) community.
	o Defining the MSM community
	o Past experiences at events for the MSM community
	o Spaces, activities, and businesses associated with the MSM community.
	Digital spaces for Chinese MSM.
	o Examples of digital spaces for Chinese MSM (e.g., mobile phone applications, social media, websites)
	o Importance of digital spaces to MSM
Day 2	Knowledge on sexually transmitted infections (STI) testing.
	o Preferences for testing
	o Spaces associated with testing.
	Barriers and facilitators to STI testing.
	o Motivations to test for STI
	o Barriers to testing for varying STI.
	o Differences between HIV and other STIs
Day 3	Perceptions of public STI clinics.
	o Past experiences with public (government-run) STI clinics
	o Barriers and facilitators for MSM to attend public STI clinics.
	o Advantages and disadvantages of public STI clinics relative to other modes of testing (e.g., self-testing or MSM-led clinics)
	o Recommendations for public STI clinics to improve engagement of MSM
Day 4	Feedback on pay-it-forward testing.
	o Perspectives on pay-it-forward as an intervention.
	o Developing community engagement strategies
	o Improving clinic implementation processes associated with pay-it-forward.
	o Recommendations to promote pay-it-forward testing and to nurture a sense of community

Participants’ acceptability and opinions of the online chat-based approach and using WeChat as the platform for online FGD were assessed using exit interviews.

Two facilitators (TW and RKJT) moderated the FGDs and were present in all FGDs, with one assuming the main facilitator role while the other assumed a co-facilitator role. We also created an observer role (CL) for another research team member in each FGD, who served to take field notes and provide feedback during and at the end of the FGD. RKJT and CL are trained qualitative researchers with social sciences backgrounds and public health and utilized this opportunity to train TW, the other co-facilitator with a social work background. TW and CL identified as Chinese, cisgender, heterosexual women, while RKJT identified as a Singaporean, cisgender, gay man.

The facilitators piloted the online FGD approach within the project team to help the facilitators familiarize themselves with the topic guide and operational aspects of setting up the FGDs. The pilot team’s feedback on the phrasing of questions, tips to build participant rapport over chat, as well as prompts, were collected and subsequently discussed by the facilitators. The facilitators created a separate chat group called “FGD Field Notes,” where the facilitators and observers held reflexive discussions around the substantive nature of discussions and issues around building rapport throughout the FGDs. At the end of the FGD, each participant completed an exit survey (see Supplementary Material 3) that assessed their WeChat usage behaviors, perspectives on the use of WeChat for FGDs, their overall experience as participants, as well as feedback on positive and challenging aspects of participation in the FGD. Participants were also encouraged to submit additional feedback directly to the research coordinator via WeChat private messaging.

Data Management

At the end of the four days session, we had a bit over 40 transcripts. Data from the FGDs were extracted verbatim using a copy-and-paste function that allowed the transfer of text and emojis from the chat room into a text-processing document (e.g., Microsoft Word). Images and media shared in the chat groups were extracted separately by downloading the media and indicating where such media would appear in the transcript. This allowed the research team to generate a transcript for each FGD for analysis. All FGDs were conducted in Mandarin Chinese but were transcribed verbatim and subsequently translated by a bilingual native Chinese speaker (TW). The transcripts were then verified by a bilingual native English speaker (RKJT) before analysis.

Data Analysis

Data from the focus groups were analyzed using the framework analysis method (Gale et al., 2013). TW, RT, GM, MH, and CL met as a group to develop the coding framework tree using one transcript for each day for each FGD group. Before the meeting, TW and RT and MH and CL worked as a team to code transcripts for two days each, and GM served as a tiebreaker in case of discrepancies. Two independent members double-coded transcripts for each day and highlighted discrepancies for group discussion. This involved seven pre-defined steps for transcription, familiarization, independent coding by research team members, developing a working analytical framework, applying the framework to subsequent transcripts, iterative framework revision as coding proceeded, and interpreting the data.

The final framework was generated at the team meeting [Supplementary Table 1], after which all members worked individually to apply the codes to the transcripts. We also examined the field notes/reflections recorded by the observers within each group at the end of each day to help improve the chat-based discussion process. Data were managed using an online qualitative data management cloud software, Dedoose (Socio-cultural Research Consultants, LLC). Data from the baseline and exit surveys were analyzed through descriptive analysis (for quantitative data) as well as framework analysis (for qualitative, open-ended data) (Gale et al., 2013). Specifically, while such data were collected through separate mechanisms, we opted to integrate thematic insights from our qualitative analysis and descriptive statistics from our quantitative data to detail a ‘significant whole’ (Bazeley & Kemp, 2012). The team debriefed each other to refine the codes, assess intercoder agreement, and address discrepancies in a weekly discussion team meeting.

Ethical Approval

The internal review board of the Dermatology Hospital of xx Medical University and the ethical committee of the University of xx approved the study. All participant identifier details like contact numbers and real names were not made known to others in the group unless explicitly requested to ensure anonymity, privacy, and confidentiality. All participants were required to provide informed consent to strict confidentiality to ensure sensitive information shared during the discussion sections (like personal experiences of STIs and navigating one’s sexuality) would not be leaked.

Results

Overall, participants found the online chat-based FGD approach more convenient, less awkward, and allowed them to speak more freely. The strategy also highlighted some vital ethical factors that warrant consideration when designing similar studies in future.

Ethical Considerations

Our literature review and MSM community members flagged several ethical considerations during our pilot testing phase, and steps were taken to mitigate any risks posed to participants. First, the local institutional review board expressed concern over how individuals can contact one another directly through the chat group. In an in-person FGD, details such as contact numbers will not be made known to others unless explicitly requested. Second, MSM community leaders expressed concern about anonymity and confidentiality of participation in online research, given that it was the first time they had witnessed a WeChat FGD being conducted. Third, as WeChat does not offer end-to-end encryption, our research team members also raised a potential issue around fears of sensitive information (e.g., personal experiences of STI, navigating one’s sexuality, etc.) being leaked.

To mitigate these issues, the team made it mandatory for participants only to use a pseudonym in the group chat and not use a profile picture with their faces. A special feature in WeChat allows users to change their alias for a group chat, facilitating this process. Furthermore, participants were asked to adhere to strict confidentiality, anonymity, respect, consideration, and safety (see Supplementary Material 4) as part of FGD ground rules in the informed consent form. These were adapted from other online chat-based FGDs using WhatsApp (Neo et al., 2022).

Deepening Engagement Through Bidirectional Communication

Online FGDs allowed us to deepen engagement with participants in several ways. First, using WeChat increased bidirectional communication between participants and the researcher. For example, in terms of content provided by participants, we found that WeChat communication platforms flattened communication hierarchies and democratized participation by allowing participants to share content in diverse formats (e.g., text, videos, websites, or mobile phone applications). For example, in FGD1 (18- to 29-year-old participants), when participants were asked for online websites that were important to Chinese MSM, participants were able to share names and direct links to websites that others could access as the FGD proceeded. In another FGD (FGD3, 30- to 39-year-old participants), participants were asked whether they had encountered any heartwarming experiences within the MSM community. One participant shared a video he had personally recorded from a previous event he had attended – allowing him to share a memory he had captured with the group. Examples showing these bi-directional communications are displayed in Supplementary Figure 2.

Mitigating Implementation Challenges

The first challenge we encountered was low levels of participation by several participants. While participants were aware and encouraged to actively participate in all the WeChat FGDs sessions, maintaining enthusiastic participation throughout the discussions took time due to the asynchronous format. Participants’ feedback also highlighted why questions needed to be expanded to provide more depth in the discussion and the need to relax guidelines to enable points raised by members who missed peak discussion hours to be captured. Therefore, the previous minimum word count requirement for responses to questions and their related subjects that may arise was exempted. And to engage participants who could not respond directly to certain comments promptly, our facilitator helped by reposting numbered summarizations of the questions asked during that morning or afternoon’s discussion periodically (at 8:30 am, 11:30 am, 2:30 pm, and 5:30 pm). That helped participants save time in scrolling back to identify questions asked by the facilitator:

Facilitator: Some of us may be busy in the afternoon. To help everyone catch up with our questions, I have summarized the questions we asked this afternoon. Please reply to the unanswered questions after seeing them if you still need to answer.

Facilitator: [List of questions in the past few hours]

Facilitator: I’m here to remind you that some of our friends here have not participated much in our discussion today. If you did not participate much, please reply to today’s questions before 9:00 am tomorrow ☺.

While less interactive, this allowed participants to contribute meaningfully and in-depth to these questions. It also allowed participants who responded earlier to add more insights into what they might have missed earlier in the day. Participants provided responses that illustrate this point [see Table 2].

Table 2.

Summary of Illustrative Quotes for Participant Feedback to the WeChat Focus Group Discussions.

Category	Themes	Quotes From Participants
Positive aspects of participation	Convenience of WeChat	“I can respond as and when I prefer.”
	Convenience of WeChat	“I participated in responding to basic questions, but I can always return to add more to my previous answers.”
	Less social awkwardness	“It is not awkward.”
	Less social awkwardness	“You can speak freely on WeChat without feeling restrained.”
	Being able to speak freely due to greater anonymity	“Privacy allows you to express all your opinions safely.”
Challenging aspects of participation	Lack of in-depth exploration	“Questions can be expanded on a little more.”
	Lack of in-depth exploration	“While convenient, discussion feels a little thin, and some points emerge after a group of people discussed.”
	Lack of perceived enthusiasm due to the online nature	“Mobilize all members to participate or increase the number of members in a single group, and everyone expresses their own opinions to lead to more opinions and topics.”
	Lack of perceived enthusiasm due to the online nature	“The discussion was not intense enough.”

Dealing With Conflicts

The second challenge that arose was dealing with conflicts in the FGDs. While participants felt less restricted and safer sharing due to the anonymity of their participation, on the flip side, it created more opportunities for interpersonal conflict among participants. However, interpersonal conflict was rare, and we observed only one event throughout the FGDs. Another participant’s response to a comment on sexual practices was deemed insensitive and disrespectful by the original poster, which led to an argument on how and why each other’s contributions should not be less valued. A prompt intervention in the group chat and a private conversation with each participant by the group facilitator within a minute of the conflict onset helped resolve issues and restored harmony on the page with apologies from each participant. Our experience was that such situations needed to be monitored closely as this occurred regardless of the ground rules established on the first day, which emphasized respect for different experiences and no use of teasing/insulting language. While this was a minor occurrence, group facilitators needed to consistently observe interactions between participants to ensure that such conflicts were identified and resolved promptly when using this approach. The advantage of WeChat FGDs was that private messaging could be initiated on the side. At the same time, the group chat was ongoing, allowing facilitators to chat privately with individual participants who may wish to convey their concerns.

Participants’ Experience

Feasibility and Acceptability

Feasibility was inferred from the overall uptake rate, defined as the number of recruited individuals that fully participated in the FGD for the duration of the study. Thus, our observed uptake rate of 95.8% (n = 46/48) showed that the approach was feasible. To assess acceptability participants were asked if this research approach was acceptable to them in the open-ended exit survey. Overall, participants felt that WeChat FGDs were highly suitable for FGDs and had a positive experience. When asked to rate their experience participating in the WeChat FGDs, 91.3% (n = 42) of participants responded that they found it good or very good, respectively. When asked to rate the suitability of WeChat for FGDs, 93.5% (n = 43) of participants responded that they found it appropriate and very appropriate, respectively.

For example:

“[WeChat] was a convenient way of participating in the FGD” (FGD Group 5 participant, 40 years old and above with below college educational attainment).

“[The WeChat format] allowed me to answer questions and share my personal views” (FGD Group 5 participant, 40 years old and above, below college educational attainment)

Table 3 summarizes illustrative quotes on participants’ positive feedback and challenges faced. Participants highlighted convenience, the opportunity to respond as and when they could or preferred, and the option to return to previous questions that had already passed and used the ‘quote’ function that allows an individual to respond to a specific message and the positive aspects of the chat-based FGD. Participants also highlighted how the lack of in-person interaction made it less socially awkward, allowing them to “speak freely on WeChat without feeling restrained.” Finally, the anonymity that WeChat accorded allowed them to express themselves safely.

Table 3.

Sociodemographic Characteristics of Participants (n = 48).

Question	Responses	n	%
How would you describe your sexual preferences?	Mainly attracted to men	43	90.0
	Mainly attracted to women	1	2.1
	Attracted to both men and women	4	8.3
What is your highest educational attainment?	Elementary school	3	2.7
	Junior high school	6	5.3
	High school	23	20.4
	College	5	10.4
	Postgraduate	11	9.7
What is your monthly income?	1500 RMB and below	4	8.3
	1501 – 3000 RMB	5	10.4
	3001 – 5000 RMB	13	27.1
	5001 – 8000 RMB	9	18.8
	8000 RMB and above	17	35.4
Have you disclosed your sexual orientation to:	Your straight/heterosexual friends	15	31.3
	Colleagues	5	10.4
	Parents	7	14.6
	Other relatives	7	14.6
	Healthcare providers	10	20.8
Have you ever participated in any offline LGBTQ-related activities or events?	Yes	19	39.6
	No	29	60.4
Where do you usually go for HIV testing?	Self-testing	20	42.6
	Voluntary counseling and testing clinic	16	34.0
	Public hospital or clinic	11	23.4
Where do you usually go for STI testing?	Self-testing	20	42.6
	Voluntary counseling and testing clinic	16	34.0
	Public hospital or clinic	11	23.4
In the past 12 months, were you diagnosed with either gonorrhea, syphilis, or chlamydia?	Yes	36	75.0
	No	12	25.0

Question	Responses	Median	Interquartile range
How many times did you test for HIV in the past 12 months?		3.0	2.5
How many times did you test for HIV in the past 12 months?		2.0	2.5

Enhanced Inclusivity

Inclusivity was assessed through the sociodemographic attributes of our participants. Of the 48 participants, 95.8% (n = 46) completed the four days of WeChat FGDs, and two were exempted due to inactivity after the first day of the discussions. Most participants (93.5%, n = 43) identified as having a sexual attraction to other men only, had a university level of education and above (65.2%, n = 30), and had a monthly income of ∼1125 USD and above (37.0%, n = 17). Most participants had never attended any in-person LGBT activities (63.0%, n = 29), but just a few had disclosed their attraction to other men to medical professionals (21.7%, n = 10). The median number of tests done for HIV and other STIs was 3.0 (Interquartile Range [IQR] = 2.5, Minimum = 0, Maximum = 6), most preferred HIV self-testing (n = 20, 43.5%), and testing for other STIs at public hospitals (n = 20, 43.5%). Table 3 summarizes the sociodemographic attributes of the participants who commenced the study.

Deepening Participants’ Influence Over Key Research Workflows and Processes

We found that online FGDs helped to deepen engagement related to influence and inclusivity. Regarding influence, the WeChat FGD sought to get direct feedback from participants on the PIONEER RCT, which involved questions on the nature of the intervention, implementation workflows, and crowdsourcing recommendations for intervention elements through the FGD. With this, participants could have direct feedback on the research team’s workflows and processes.

For example, on the fourth day of the FGD, the facilitator started the day by sharing details of the pay-it-forward intervention through an image and a video, alongside brief study details. Participants provided feedback as well as recommendations on how to improve clinical implementation processes for the PIONEER study. For example, one participant (FGD1, 18- to 29-year-old participants) provided additional feedback on several aspects of the clinic workflow:

‘First, strengthen medical staff’s awareness of sexual minorities so that they treat everyone equally, without personal judgement; second, strengthen the privacy of patient information among medical staff, and avoid leakage of private information; third, ensure a reasonable flow of people – when you make an appointment online, ensure that patients can see the number of available doctors at a specific time of the day.’

Discussion

We found that online FGDs enabled MSM to engage deeper in research in several ways. First, the WeChat platform allowed for collaborative discussions that flattened communication hierarchies and enabled participants to participate in a more bidirectional form of information sharing. Second, using WeChat allowed a diverse and heterogeneous group of MSM to participate in this study. Third, WeChat FGDs were suitable for formative research processes that involved discussing implementation aspects of clinical trial studies, allowing MSM in our study to influence research processes for a clinical trial directly. In the context of deepening community engagement (Fung, 2006; US CDC, 2011), the platform allowed us to heighten the collaborative aspect of the research process, as well as deepen the influence and inclusivity of our formative research - based on the continuum of community engagement defined by NIH (7) and the dimensions of participation conceptualized by Fung (12). Our findings showed that using social media chat-based apps for online qualitative research is feasible and allows for deeper participant engagement for vulnerable sub-populations and discussing sensitive topics like sexual behaviors compared to in-person methods. The online chat-based FGDs approach allowed for stronger bidirectional content-sharing than in-person FGDs [see Table 4]. This observation aligns with a previous study that assessed the acceptability and feasibility of using other platforms like WhatsApp for online FGDs. In this study, the chat-based apps democratized and enhanced the participatory nature of FGDs compared to traditional in-person FGDs (Neo et al., 2022). Due to anonymity, possible feelings of shyness and anxiety hindering bidirectional sharing in in-person FGDs were eliminated in online chat-based FGDs. Another study found that using chat-based apps like WhatsApp for online FGDs provides high ecological validity as individuals respond to the questions in their everyday lives (Colom, 2021). This increased ecological validity was also reflected in our finding that participants were able to share their own content (e.g., web links, apps, videos, photos), some of which were even recorded by themselves, with others in the group, as though it were a regular WeChat group chat that took place in the context of their daily lives.

Table 4.

The Level and Depth of Participant Engagement in Online Versus Offline Focus Group Discussions Differ.

In-Person Focus Group Discussions	Online Focus Group Discussions
Time and convenience
All participants must be available at the specified time to be able to contribute to the discussion.	Participants with busy schedules who miss peak active discussion moments can still submit their opinions on already discussed topics.
Inconvenient for engaging vulnerable populations (like gender and sexual minorities) at risk of social stigma and discrimination and those who wish to stay hidden within the community.	Convenient for engaging vulnerable populations, including those who wish to remain hidden
Depth of engagement
Discussions are mainly unidirectional (occur between the facilitator and the participants).	Discussions are more bidirectional and occur between facilitators and participants and between participants.
Participants solely use verbal means to communicate their opinions and views. Facilitators may take note of body language during sessions, but their meanings could be difficult to discern.	Alternative communication tools like videos, stickers and emojis help participants express themselves more deeply.
	These become additional data sources for researchers and are easy to decipher.
More responses and the total number of words spoken as participants generally took turns to allow one another to talk about each topic.	Fewer total number of words spoken/typed as participants are not required to take turns answering each topic. However, the length of the responses varies across groups and is elaborate.
Dynamics
The presence of 'dominant' individuals (a person whose responses make up more than 40% of all the answers in the focus group session) intimidates other participants and prevents them from speaking.	The presence of dominant individuals is less noticeable or disruptive. Thus, they are less likely to overwhelm the conversation or prevent others from speaking; their relatively frequent responses provided additional stimuli for other participants to respond to questions.
Participants may have difficulty elaborating in response to questions on sensitive topics like sexual behaviors and beliefs due to shyness or fear of being ridiculed or judged.	Participants elaborate more on their responses to questions on sensitive topics due to the anonymity and privacy afforded by the chat app.

Note. Features of in-person focus group discussions were obtained from the findings of a comparative study conducted by Chen and Neo (Chen & Neo, 2019).

Our data suggests that online FGDs allow participation from MSM who do not disclose their sexual orientation to physicians or participate in conventional community activities. Hence, online FGDs could be an essential tool to engage closeted MSM in research since a national cross-sectional study in China reported that about 80% of MSM participants had never disclosed their sexual identity to others, including healthcare workers (Tang et al., 2017). While legal persecution of sexual minorities was repealed in 1997 (Wang et al., 2019), stigma manifests in the lives of Chinese MSM in multiple ways, including stigma towards MSM among the general public and healthcare workers (Neilands et al., 2008; Ong et al., 2017), as well as self-stigma among Chinese MSM (Chi et al., 2021). These remain barriers to engaging MSM in participatory health research that guide the development of tailored strategies to address challenges to sexual health services delivery (Tang et al., 2016, 2017; Zhao et al., 2016). Therefore, online FGDs provide an opportunity to circumvent these barriers to deepen the engagement of more hidden key populations in health research and the diversity of outcomes from such participatory processes.

The acceptability of the experience and appropriateness of online chat-based app for FGDs, the participatory nature of discussions, and the inclusion of underserved MSM makes chat-based FGDs a valuable tool for formative clinical trial research. The approach could complement the existing range of qualitative methods appropriate for addressing important implementation questions in the formative phases of clinical trial processes (Hamilton & Finley, 2019). Notably, sociodemographic and behavioral characteristics of target populations and local settings factor vital to designing online chat-based FGDs. For example, work routines, online chat-based app use frequency, age, and accessibility to data bundles influence key aspects of online FGDs designs. Like which chat-based app is most suitable, the time or day to initiate online FGD sessions, and how participants should be categorized (like age group or workgroup). Therefore, variations in target population characteristics and local context settings should be keenly considered when adapting online FGDs to ensure optimized outcomes.

Implementation Challenges

We are also mindful that several challenges and limitations arose with WeChat FGDs. The lack of in-depth explorations and potential lack of enthusiasm in participation has implications for the depth of discussions and interaction among participants. These echo the findings of multiple studies conducted on WhatsApp FGDs and text-based qualitative research in general. For example, a study directly comparing in-person and WhatsApp FGDs found that data richness and detail in the WhatsApp FGDs did not match that of in-person groups, especially among older participants who may be less technologically savvy (Chen & Neo, 2019). Other studies using chat-based interviews or FGDs also generally report lower word counts, shorter responses, and a lack of detail and richness (Abrams et al., 2015; Brüggen & Willems, 2009; Woodyatt et al., 2016). Regardless, participant responses in online chat-based FGDs could be equally expressive and descriptively rich in the contextual use of visuals like emojis, images, and videos which adds depth in a way that in-person interactions cannot. Therefore, online chat-based FGDs remain suitable for more descriptive forms of qualitative analysis that focus more strongly on a text’s semantic meaning (Sandelowski, 2000).

Implications

Our observations and experience with online chat-based FGDs using a messaging app have several implications for future participatory health research. Firstly, the data collected through the online FGD approach was in-depth, and visuals (like videos/pictures) yielded rich contextual data. Therefore, online FGDs could be used to improve self-expression among participants in formative research on sensitive subjects (like sexual behaviors) that people may feel uncomfortable or embarrassed discussing in person. Secondly, the approach enabled formative research during a pandemic where participants’ social/physical distancing and safety were paramount and could be adaptable for settings with high phone-based online communication platform use. This provides new opportunities for how research could be expanded with more community participation during pandemics to provide timely information for data-driven prevention interventions in the future. Third, the privacy of participant contacts was safeguarded in our study by privacy features of the online chat-based app adopted (WeChat), which prevents others from adding participant contacts directly from chat groups. However, future research in this area should recognize differences in technology, app openness, and privacy features when choosing messaging apps for online FGDs and establish additional protocols to safeguard the privacy of participant contacts. Fourth, the online FGD approach facilitated deeper engagement of closeted MSM due to its anonymous nature. Thus, online chat-based formative research could be a complementary strategy to engage highly marginalized populations in participatory health research processes, especially in settings with high prejudice, stigma, and local legislature barriers.

Finally, future studies should consider how the disembodied nature of online spaces affects data quality and balance between the need for high-quality data and potentially stressing participants, especially during a pandemic where mental health may be fragile.

Lessons and Recommendations

Adopting an asynchronous four-day FGD, compared to a mixed (asynchronous and synchronous) five-day FGD reported by Neo and colleagues, helped ensure participants were well-engaged with low attrition rates (16). With the asynchronous approach, participants could contribute to sessions missed during peak active discussion moments, ensuring that most participants’ opinions were captured. Future chat-based FGDs should consider a maximum of three or four days to maximize the retention and engagement of participants. Also, future studies should attempt to conceptualize and measure chat participation across multiple days to validate this recommendation.

Secondly, being in a chat group with unknown strangers still comes with some discomfort that could hinder the smooth flow of conversation and limit discussion activities, especially on the first day. An interactive ice-breaker approach during self-introductions at the start of the online FGD sessions eased tensions and facilitated familiarity among participants in this study. Hence, we recommend that researchers and facilitators develop approaches and tools that encourage interaction, such as interactive introductory or ice-breaking activities and soliciting responses to certain research questions.

Our study has some limitations. First, there was a potential lack of depth and interaction in responses when using chat-based online apps, a limitation documented by studies evaluating the use of such platforms for FGDs (Neo et al., 2022). However, this was balanced by the ability of participants to share direct links and content online, which may not have been achievable through in-person focus group discussions. Secondly, gay, bisexual and other MSM who utilize online spaces are not homogenous, and researchers have to further explore nuances of intracommunity stigma and issues of how subcommunities of Chinese GBMSM may experience such online spaces differently. Third, the anonymity online spaces provide make it difficult to verify the identity of participants. Also, all data generated through the online approach are self-reported and subject to social desirability and recall bias. Nonetheless, the distinction between online versus offline/general physical spaces keeps blurring with time and creates broader opportunities for engagement (Miles, 2021).

Conclusion

Online FGDs using chat-based apps show great promise in engaging underserved populations, especially in settings where prevailing stigma and local factors hinder research participation. The approach is convenient and flexible for participants and inexpensive for researchers to engage wider communities since most online messaging apps are freely available for download and use. Additionally, chat-based online FGDs could be adapted to complement other traditional health research strategies to safely collect relevant real-time data during a pandemic. However, further research is needed to circumvent ethical, privacy and confidentiality concerns and verify the cost-effectiveness of the approach in other health research contexts outside of pandemics.

Supplemental Material

Supplemental Material - Online Focus Group Discussions to Engage Stigmatized Populations in Qualitative Health Research: Lessons Learned

Supplemental Material for Online Focus Group Discussions to Engage Stigmatized Populations in Qualitative Health Research: Lessons Learned by Gifty Marley, Rayner Kay Jin Tan, Tong Wang, Chunyan Li, Margaret E. Byrne, Dan Wu, Cheng Wang, Weiming Tang, Rohit Ramaswamy, Danyang Luo, Sean S. Sylvia, and Joseph D. Tucker in International Journal of Qualitative Methods

Footnotes

Acknowledgments

We appreciate the staff of the University of North Carolina at Chapel Hill and the Dermatology Hospital of Southern Medical University for their collaboration and assistance. We also wish to thank MSM community volunteers in Guangzhou for participating the discussions and the staff of SESH Global for additional assistance during the pretest.

Author Contributions

JT, WT, RT, and WC conceived the study idea. JT, WT, RT, DL, and TW designed the study. RT, GM, TW, MH, DL and CL facilitated the implementation. GM, RT, CL, and TW drafted the initial manuscript. JT, WT, TW, MH, RR, SS and WC critically revised the manuscript for important intellectual content. All authors approved the final manuscript version for publication.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institutes of Health (NIAID R01AI158826, K24AI143471). The funder(s) played no role in the writing of this protocol and will play no role nor have authority over any of the following study-related activities: study design, implementation, data collection, management, data analysis, interpretation of data, writing of reports and the decision to submit the report for publication.

Ethical Statement

ORCID iD

Gifty Marley

Supplemental Material

Supplemental material for this article is available online.

References

Abrams

K. M.

Wang

Song

Y. J.

Galindo-Gonzalez

(2015). Data richness trade-offs between face-to-face, online audiovisual, and online text-only focus groups. Social Science Computer Review, 33(1), 80–96. https://doi.org/10.1177/0894439313519733

Bazeley

Kemp

(2012). Mosaics, triangles, and DNA: Metaphors for integrated analysis in mixed methods research. Journal of Mixed Methods Research, 6(1), 55–72. https://doi.org/10.1177/1558689811419514

Brüggen

Willems

(2009). A critical comparison of offline focus groups, online focus groups and e-Delphi. International Journal of Market Research, 51(3), 1–15. https://doi.org/10.1177/147078530905100301

Chen

Neo

(2019). Texting the waters: An assessment of focus groups conducted via the WhatsApp smartphone messaging application. Methodological Innovations, 12(3), 205979911988427. https://doi.org/10.1177/2059799119884276

Chi

Huang

Pachankis

Valimaki

Shen

(2021). Internalized sexual minority stigma is associated with HIV testing behavior among Chinese men who have sex with men: A cross-sectional study. Journal of the Association of Nurses in AIDS Care: Journal of the Association of Nurses in AIDS Care, 32(5), 578–588. https://doi.org/10.1097/jnc.0000000000000205

Colom

(2021). Using WhatsApp for focus group discussions: Ecological validity, inclusion and deliberation. Qualitative Research, 22(3), 452–467. https://doi.org/10.1177/1468794120986074

Detels

Green

A. M.

Klausner

J. D.

Katzenstein

Gaydos

Handsfield

H. H.

Pequegnat

Mayer

Hartwell

T. D.

Quinn

T. C.

(2011). The incidence and correlates of symptomatic and asymptomatic Chlamydia trachomatis and Neisseria gonorrhoeae infections in selected populations in five countries. Sexually Transmitted Diseases, 38(6), 503–509. https://doi.org/10.1097/olq.0b013e318206c288, http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3408314/

Fung

(2006). Varieties of participation in complex governance. Public Administration Review, 66(s1), 66–75. https://doi.org/10.1111/j.1540-6210.2006.00667.x

Gale

N. K.

Heath

Cameron

Rashid

Redwood

(2013). Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Medical Research Methodology, 13(1), 117. https://doi.org/10.1186/1471-2288-13-117

10.

Hamilton

A. B.

Finley

E. P.

(2019). Qualitative methods in implementation research: An introduction. Psychiatry Research, 280, 112516. https://doi.org/10.1016/j.psychres.2019.112516

11.

Hollander

J. A.

(2004). The social contexts of focus groups. Journal of Contemporary Ethnography, 33(5), 602–637. https://doi.org/10.1177/0891241604266988

12.

Holzer

J. K.

Ellis

Merritt

M. W.

(2014). Why we need community engagement in medical research. Journal of Investigative Medicine: The Official Publication of the American Federation for Clinical Research, 62(6), 851–855. https://doi.org/10.1097/JIM.0000000000000097

13.

Inan

O. T.

Tenaerts

Prindiville

S. A.

Reynolds

H. R.

Dizon

D. S.

Cooper-Arnold

Turakhia

Pletcher

M. J.

Preston

K. L.

Krumholz

H. M.

Marlin

B. M.

Mandl

K. D.

Klasnja

Spring

Iturriaga

Campo

Desvigne-Nickens

Rosenberg

Steinhubl

S. R.

Califf

R. M.

(2020). Digitizing clinical trials. Npj Digital Medicine, 3(1), 101. https://doi.org/10.1038/s41746-020-0302-y

14.

Johnson

D. A.

Joosten

Y. A.

Wilkins

C. H.

Shibao

C. A.

(2015). Case study: Community engagement and clinical trial success: Outreach to African American women. Clinical and Translational Science, 8(4), 388–390. https://doi.org/10.1111/cts.12264

15.

Leask

C. F.

Sandlund

Skelton

D. A.

Altenburg

T. M.

Cardon

Chinapaw

M. J. M.

De Bourdeaudhuij

Verloigne

Chastin

S. F. M.

on behalf of the GrandStand, Safe Step and Teenage Girls on the Move Research Groups (2019). Framework, principles and recommendations for utilising participatory methodologies in the co-creation and evaluation of public health interventions. Research Involvement and Engagement, 5(1), 2. https://doi.org/10.1186/s40900-018-0136-9

16.

K. T.

Tang

Huang

Lee

Feng

Pan

S. W.

Han

Mak

Yang

Tucker

J. D.

(2019). Pay-it-forward strategy to enhance uptake of dual gonorrhea and chlamydia testing among men who have sex with men in China: A pragmatic, quasi-experimental study. The Lancet Infectious Diseases, 19(1), 76–82. https://doi.org/10.1016/s1473-3099(18)30556-5

17.

Lin

Nehl

E. J.

Tran

Zheng

Wong

F. Y.

(2014). Sexually transmitted infection testing practices among ‘money boys’ and general men who have sex with men in Shanghai, China: Objective versus self-reported status. Sexual Health, 11(1), 94–96. https://doi.org/10.1071/SH13199

18.

Lutz

A. R.

(2015). Screening for asymptomatic extragenital gonorrhea and Chlamydia in men who have sex with men: Significance, recommendations, and options for overcoming barriers to testing. LGBT Health, 2(1), 27–34. https://doi.org/10.1089/lgbt.2014.0056

19.

Manta

Patrick-Lake

Goldsack

J. C.

(2020). Digital measures that matter to patients: A framework to guide the selection and development of digital measures of health. Digital Biomarkers, 4(3), 69–77. https://doi.org/10.1159/000509725

20.

Miles

(2021). Let’s (not) go outside: Grindr, hybrid space, and digital queer neighborhoods. In Bitterman

Hess

D. B.

(Eds.), The life and afterlife of gay neighborhoods: Renaissance and resurgence (pp. 203–220). Springer International Publishing. https://doi.org/10.1007/978-3-030-66073-4_9

21.

Neilands

T. B.

Steward

W. T.

Choi

K. H.

(2008). Assessment of stigma towards homosexuality in China: A study of men who have sex with men. Archives of Sexual Behavior, 37(5), 838–844. https://doi.org/10.1007/s10508-007-9305-x

22.

Neo

P. H.

Lim

J. M.

Tan

R. K.

Ong

S. E.

(2022). Using WhatsApp focus group discussions to collect qualitative data collection during a pandemic: Exploring knowledge, attitudes, and perceptions of COVID-19 in Singapore. International Journal of Qualitative Methods, 21, 16094069221090355. https://doi.org/10.1177/16094069221090355

23.

Nyika

Chilengi

Ishengoma

Mtenga

Thera

M. A.

Sissoko

M. S.

Lusingu

Tiono

A. B.

Doumbo

Sirima

S. B.

Lemnge

Kilama

W. L.

(2010). Engaging diverse communities participating in clinical trials: Case examples from across africa. Malaria Journal, 9(1), 86. https://doi.org/10.1186/1475-2875-9-86

24.

Ong

J. J.

Peng

Zhu

Y. R. J.

Fairley

C. K.

Kidd

M. R.

Roland

Jiang

Wong

W. C. W.

(2017). Opportunities and barriers to STI testing in community health centres in China: A nationwide survey. Sexually Transmitted Infections, 93(8), 566-571. https://doi.org/10.1136/sextrans-2017-053196

25.

Reza-Paul

Lazarus

Jana

Ray

Mugo

Ngure

Folayan

Durueke

Idoko

Béhanzin

Alary

Gueye

Sarr

Mukoma

Kyongo

Bothma

Eakle

Dallabetta

Presley

Lorway

(2019). Community inclusion in PrEP demonstration projects: Lessons for scaling up. Gates Open Research, 3(1504), 1504. https://doi.org/10.12688/gatesopenres.13042.2

26.

Robert

Donetto

Williams

(2021). Co-designing healthcare services with patients. The Palgrave handbook of Co-production of public services and outcomes (pp. 313–333). Springer.

27.

Rosenfeld

Sina

Sarne

Avidov

Kraus

(2018). WhatsApp usage patterns and prediction of demographic characteristics without access to message content. Demographic Research, 39, 647–670. https://doi.org/10.4054/DemRes.2018.39.22

28.

Sandelowski

(2000). Whatever happened to qualitative description? Research in Nursing & Health, 23(4), 334–340. https://doi.org/10.1002/1098-240x(200008)23:4<334::aid-nur9>3.0.co;2-g(200008)23:4<334::aid-nur9>3.0.co;2-g

29.

Sanders

E. B. N.

Stappers

P. J.

(2008). Co-creation and the new landscapes of design. CoDesign, 4(1), 5–18. https://doi.org/10.1080/15710880701875068

30.

Shaffer

K. M.

Nelson

C. J.

DuHamel

K. N.

(2018). Barriers to participation in a sexual health intervention for men following treatment for rectal and anal cancer. Psycho-Oncology, 27(3), 1082–1085. https://doi.org/10.1002/pon.4593

31.

Slattery

Saeri

A. K.

Bragge

(2020). Research co-design in health: A rapid overview of reviews. Health Research Policy and Systems, 18(1), 17. https://doi.org/10.1186/s12961-020-0528-9

32.

Statista (2022). Number of active WeChat messenger accounts Q2 2011-Q4 2021. Retrieved May 17 from. https://www.statista.com/statistics/255778/number-of-active-wechat-messenger-accounts/

33.

Tan

R. K. J.

Day

Zhao

Larson

H. J.

Sylvia

Tang

Tucker

J. D.

(2022). Digital approaches to enhancing community engagement in clinical trials. Npj Digital Medicine, 5(1), 37. https://doi.org/10.1038/s41746-022-00581-1

34.

Tang

Mao

Tang

Liu

Mollan

Cao

Wong

Zhang

Hudgens

Qin

Han

Yang

Wei

Tucker

J. D.

Sesh Study Group (2017). Disclosure of sexual orientation to health professionals in China: Results from an online cross-sectional study. Journal of the International AIDS Society, 20(1), 21416. https://doi.org/10.7448/ias.20.1.21416

35.

Tang

Mao

Tang

Liu

Mollan

Wong

Zhang

Hudgens

Qin

Liao

Yang

Kang

Wei

Tucker

J. D.

(2016). Disclosure of sexual orientation in men who have sex with men in China: A cross-sectional survey. The Lancet, 388, S68. https://doi.org/10.1016/S0140-6736(16)31995-X

36.

(2016). WeChat and civil society in China. Communication and the Public, 1(3), 343–350. https://doi.org/10.1177/2057047316667518

37.

US CDC . (2011). Principles of community engagement. US NIH. https://stacks.cdc.gov/view/cdc/11699

38.

Wang

Peng

Xin

Yang

Drescher

Chen

(2019). Discrimination against LGBT populations in China. The Lancet Public Health, 4(9), Article e440–e441. https://doi.org/10.1016/S2468-2667(19)30153-7

39.

Woodyatt

C. R.

Finneran

C. A.

Stephenson

(2016). In-person versus online focus group discussions: A comparative analysis of data quality. Qualitative Health Research, 26(6), 741–749. https://doi.org/10.1177/1049732316631510

40.

World Health Organization (2022). Sexually transmitted infections (STIs). Retrieved May 13 from. https://www.who.int/news-room/fact-sheets/detail/sexually-transmitted-infections-(stis)

41.

K. T.

Tang

Ong

J. J.

Huang

Lee

Wei

Tucker

J. D.

(2019). Low chlamydia and gonorrhea testing rates among men who have sex with men in Guangdong and Shandong Provinces, China. Sexually Transmitted Diseases, 46(4), 260–265. https://doi.org/10.1097/OLQ.0000000000000963

42.

Yang

Zhang

T. P.

Tang

Ong

J. J.

Alexander

Forastiere

Kumar

K. T.

Zou

Yang

Wang

Huang

Lee

Zhu

Luo

Vickerman

Yang

Christakis

N. A.

Tucker

J. D.

(2020). Pay-it-forward Gonorrhoea and chlamydia testing among men who have sex with men in China: A randomised controlled trial. The Lancet Infectious Diseases, 20(8), 976–982. https://doi.org/10.1016/S1473-3099(20)30172-9

43.

Yang

L. G.

Zhang

X. H.

Zhao

P. Z.

Chen

Z. Y.

W. J.

Ren

X. Q.

Wang

L. Y.

Chen

W. Y.

Tucker

J. D.

(2018). Gonorrhea and chlamydia prevalence in different anatomical sites among men who have sex with men: A cross-sectional study in Guangzhou, China. BMC Infectious Diseases, 18(1), 675. https://doi.org/10.1186/s12879-018-3579-6

44.

Zhao

Chen

Qin

(2016). Non-disclosure of sexual orientation to parents associated with sexual risk behaviors among gay and bisexual MSM in China. AIDS and Behavior, 20(1), 193–203. https://doi.org/10.1007/s10461-015-1135-6

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.44 MB