Accessibility,Retention and Interactivity of Online Co-Creation Workshops: A Qualitative Post-Hoc Analysis

Abstract

Introduction: Co-creation is becoming increasingly popular to develop interventions that can achieve results beyond scientific findings. Workshops are one of the main ways to collect data and generate ideas in co-creation, which traditionally have been conducted at a fixed, physical location. However, organising face-to-face workshops is considered challenging, due to the transportation time, low flexibility and high costs. This study aims to investigate the online format of co-creation workshops and to discuss methodological considerations. Methods: Co-creation workshops were organised with 78 medical students, general practitioner (GP) trainees and specialist trainees from five Belgian universities. The study included four different cohorts, namely first-year bachelors (n = 12), first-year masters (n = 13), first-year GP trainees (n = 14) and first-year specialist trainees (n = 39). Three consequential online workshops were organised for each cohort, resulting in 12 workshops in total. The collected data included qualitative data (video-and audio recordings, notes from interdisciplinary team discussions, Miro boards) and quantitative data (registrations and actual attendance rates). All workshops were transcribed verbatim and thematically analysed using NVivo. Results: The participants reported increased flexibility and reduced costs as main benefits of the online format. In addition, the online platforms Miro and Microsoft Teams were considered dynamic and inspiring, facilitating high levels of engagement and interactivity. However, the online format also showed some challenges, such as the need for a digitally educated population and a stable internet connection. Conclusion: This study has shown that online workshops, also in the highly interactive method of co-creation, provide a viable alternative to collect data and generate ideas. This is particularly the case, when the target population is geographically dispersed, has a high workload and is digitally educated. Online workshops, however, also face limitations and challenges that need to be considered when choosing this format.

Keywords

co-creation workshops qualitative research focus groups online data collection evaluation

Introduction

Co-creation, also referred to as ‘co-design’, ‘co-production’, ‘user-centred design thinking’ or ‘human-centred design thinking’, is a bottom-up approach aiming to tackle the issue of ‘ivory tower’ research, where researchers and research end users fail to understand and/or fully engage with each other (Altman et al., 2018; Berwick, 2016; Dobe, Gustafssona, & Waldera, 2022; Greenhalgh et al., 2016; Voorberg et al., 2014). Although the above-mentioned terms each relate to different processes (Chathoth, Altinay, Harrington, Robert James Okumus, & Chan, 2013; Halvorsrud et al., 2019), this study uses the term co-creation as an umbrella term defined as the collaborative generation of knowledge and ideas characterised by an iterative, creative and innovative research process, where non-academic stakeholders and research end users are involved in every phase of the process (Benson et al., 2021; Forlizzi & Battarbee, 2004; Halvorsrud et al., 2019; Osborne et al., 2016; Sanders & Stappers, 2014; Tschimmel, 2012).

Co-creation is increasingly being used to develop interventions that have the potential to achieve impact beyond scientific findings (Altman et al., 2018; Berwick, 2016; Dobe et al., 2022; Greenhalgh et al., 2016; Voorberg et al., 2014). For example, a recent review by Fusco et al. (2020) found that the number of health research papers using co-creation increased with 25% each year between 1994 and 2019 (Fusco et al., 2020). The scientific literature is consistent in claiming that co-creation paves the way for more efficient interventions, enhanced end user satisfaction, better service innovation and cost savings (Palumbo, 2016). When participants are involved in the whole process of development, they feel more accountable and responsible, which tends to improve the sustainability of the research outcomes (Renedo et al., 2015).

The practice of co-creation includes a wide range of participatory practices for design-and decision-making, engaging different types of stakeholders, with workshops being the main way of data collection (Jones, 2018; Lee et al., 2018). During these workshops, the researcher adopts the role of a moderator and applies highly interactive and dynamic methods (Lee et al., 2018), such as customer journey, personas, or prototyping (Appendix A). In addition, participants are encouraged to express their opinions and reflect on potential interventions (Lee et al., 2018; Palumbo, 2016). In this context, they should feel safe to express their opinions (Lee et al., 2018; Palumbo, 2016), believe in their ability to influence the outcomes (Parrado et al., 2013) and have the possibility to dedicate sufficient time (Voorberg et al., 2014). In order to provide a safe environment and ensure active engagement from each participant, former studies have recruited between 6 and 15 participants for this type of workshops (IDEO, 2015; Schnall et al., 2016; Stein et al., 2017; Vechakul et al., 2015).

Traditionally, co-creation workshops have been conducted face-to-face at a fixed, physical location instead of online (Richard et al., 2021; Woodyatt et al., 2016). Key limitations of real life workshops are that they are relatively expensive to conduct (e.g., travel or venue costs) (Deakin & Wakefield, 2014; Menary et al., 2021; Richard et al., 2021; Weller, 2015) and that they are bound in time and space (Stewart & Shamdasani, 2017; Woodyatt et al., 2016). In the last decade, technological advances have significantly improved the possibilities to organise online workshops (Keemink et al., 2022; Stewart & Shamdasani, 2017). Moreover, the COVID-19 pandemic has resulted in an increased need for effective online communication and collaboration, giving rise to accessible sophisticated online platforms which facilitate co-creation workshops (Howlett, 2022; Keemink et al., 2022).

While little is known about co-creation workshops in an online format, previous studies have reported positive results of online interviews and focus groups (Deakin & Wakefield, 2014; Keemink et al., 2022; Krouwel et al., 2019; Leemann et al., 2020; Richard et al., 2021; Stewart & Shamdasani, 2017; Tuttas, 2015). The difference between focus groups and co-creation workshops is that the former is used to merely collect information and identify certain information, while the latter is deployed to iteratively create interventions with multiple stakeholders (Huijnen et al., 2017).

The reported positive results in former research included that online interviews and focus groups can be more time (e.g., no transportation time) and cost effective (e.g., no venue costs), and reach a broader geographical spread (e.g., a target group which is geographically distributed) (Archibald et al., 2019; Deakin & Wakefield, 2014; Keemink et al., 2022; Krouwel et al., 2019; Leemann et al., 2020; Stewart & Shamdasani, 2017). In addition, the online format adds scheduling flexibility for both the researcher and participants, such as busy professionals (e.g., physicians) or young people (e.g., medical students), who might otherwise be unable to attend face-to-face meetings (Deakin & Wakefield, 2014; Kite & Phongsavan, 2017; Stewart & Shamdasani, 2017; Weller, 2015). Moreover, a recent study by Richard et al. (2021) found that online focus groups generated an equivalent number of unique ideas (i.e. quantity of ideas) and that there was a high degree of overlap in themes from both groups (i.e., quality of ideas) (Richard et al., 2021).

Nevertheless, there may also be some limitations to the online format. For instance, online focus groups might reduce the spontaneity, the role of nonverbal communication and the intimacy of a group, resulting in participants being less open and interactions being less vivid (Stewart & Shamdasani, 2017). The larger (physical) distance between the researcher and participant might also induce drop-out as participants may feel less committed to the appointment (Deakin & Wakefield, 2014). In addition, participants need to have a certain level of skills using the technology presented, which could result in selection bias, and there needs to be an uninterrupted access to the Internet (Deakin & Wakefield, 2014; Janghorban et al., 2014; Kite & Phongsavan, 2017; Stewart & Shamdasani, 2017; Tuttas, 2015).

To our knowledge, no studies exist that investigated the advantages and disadvantages of the online format of interactive co-creation workshops. In particular, this study aims to gain more insight into how online co-creation workshops are experienced by the participants, and to discuss methodological considerations that need to be taken into account when using this format. Based on the literature, we expect that the online co-creation workshops would be perceived as time and cost efficient, but not as very spontaneous and dynamic. The study ‘WeMeds: Healthy & Work-Engaged Medical Doctors’, which focuses on burnout and work-engagement in the Belgian medical education system, will serve as a framework to collect data on the online format of co-creation workshops (Boone & Godderis, 2021).

Methods

Participants and Design

The population consisted of 3147 medical students, general practitioner (GP) trainees and (hospital) specialist trainees: 1166 bachelor students, 1084 master students, 320 GP trainees and 577 specialist trainees. In total, 78 medical students, GP trainees and specialist trainees from five Belgian universities (i.e., University of Leuven, Free University of Brussels, University of Hasselt, University of Ghent and University of Antwerp) attended the co-creation workshops. The study targeted four different cohorts, namely first-year bachelors (n = 12), first-year masters (n = 13), first-year GP trainees (n = 14) and first-year specialist trainees (n = 39).

Recruitment and Procedure

Participants were recruited through direct (i.e., personalised e-mail) and indirect online communication (i.e., project website, platforms, social media) channels. Potential participants could register online immediately. Once registered, they received an electronic Microsoft Teams (MS) invite link and an e-mail with the necessary information (e.g., objectives of the workshop). On the day of the workshop, they received a final reminder by e-mail to attend the workshop.

From March until May 2022, three online co-creation workshops were organised for each cohort, resulting in 12 workshops in total. The workshops took place on Monday, Tuesday or Wednesday evenings from 6.30 p.m. to 9 p.m., with a 15 min coffee break. All three workshops were piloted among a group of colleague-researchers, to test for issues related to technical problems, time duration and flow of the exercises. Technically, for instance, we tested how to work with breakout rooms in the online platform and how to do the recording. The decision on the number and duration of the workshops was based on content (e.g., what is needed to answer the research question) and practical issues (e.g., agenda of the target group).

The three workshops had a consequential flow and elaborated on each other. The main outline for each workshop, the selected exercises and progression throughout the workshops were decided from the start. Workshop one focused on the problem analysis, aiming to gain insights into the problem and generating some first quick ideas for interventions. Exercises included stakeholder mapping and customer journey (Martin & Hanington, 2012; Miro, 2022b; VMware Tanzu Labs, 2022). Workshop two focused on the ideation of interventions through multiple brainstorm exercises, such as the development of personas, brainwriting and a matrix analysis (IDEO, 2015; Martin & Hanington, 2012; Miro, 2022a). Workshop three elaborated further on the ideas generated in the former workshops, including co-creation exercises, such as dot voting and prototyping (Martin & Hanington, 2012; Miro, 2022c; Unleash, 2022). A brief description of these exercises can be found in Appendix A (Table 2)

Each participant was invited for all three workshops and could choose whether to attend one, two or three workshops. The majority of participants attended only one workshop. Two bachelor students, two master students, two GP trainees and two specialist trainees attended two workshops; and one GP trainee attended all three workshops. To ensure continuity in the co-creation process, participants of workshop two and three were provided with a three page report on the outcomes of the previous workshop. These reports were developed by two researchers. A first researcher developed the first draft of the report, while a second researcher checked for errors and provided feedback. In addition, these reports were sent to the participants of that workshop for validation.

Participants did not receive any remuneration. However, as recruitment and registration for the workshops progressed slowly, especially among the first-year specialist trainees, the researchers decided—in cooperation with the medical faculties—to introduce an incentive to stimulate participation among this cohort. Through communication from their faculty, this cohort was given the opportunity to write a report on the workshop and use this for an assignment within their medical education.

All 12 workshops were moderated by the first author to ensure consistency. This moderator was assisted by a second researcher, who was trained on the content and technical logistics. This researcher also served as a backup moderator if the main moderator would have unresolvable technical issues. For this reason, they were located in the same room to be able to discuss such issues quickly.

In addition, an interdisciplinary research team (i.e. the co-authors and four thesis students) was set-up to discuss methodological considerations of the co-creation workshops and to validate results. From this research team, one to three observer-researchers would join each workshop and contribute by making notes on, amongst others, the atmosphere, co-creation exercises and the levels of interactivity. These observers did not actively participate in the workshops and they turned off their microphones and cameras.

Online Platforms and Tools

We selected MS Teams as the online platform, because this platform is able to support meetings of around 20 people, provides video-and audio recordings, is GDPR compliant, provides the possibility of breakout rooms and is user-friendly. Besides MS Teams, we needed an online tool to manage the interactive and creative component of the co-creation workshops. This tool had to combine the functionality of an online whiteboard, and the possibility to incorporate sticky notes and other interactive tools to increase engagement and incite creativity among participants. For these reasons, we chose Miro (www.miro.com). Appendix B (Figure 1, 2 and 3) shows the Miro boards for the three workshops.

Data Collection

The workshops were conducted in Dutch, and the collected data were translated to English for the purpose of this paper. The data included both qualitative (video-and audio recordings, Miro board exports, notes from the interdisciplinary research team) and quantitative data (registrations and actual attendance rates). At the beginning of each workshop, participants were welcomed in MS Teams. They were asked to switch on their video and microphone, and they were provided with the Miro link. At the end of each workshop, 15 minutes were allocated to collect feedback about the workshops. During these feedback sessions, participants had 5 minutes to write down their feedback in free-text fields on the Miro board (Appendix C, Figure 4) to discuss what they liked about the workshop, what they hope for and what their recommendations for future workshops would be. After this, 10 minutes were allocated to discuss the written feedback amongst all participants. Participants could then elaborate on their comments and the moderator could ask questions. These feedback tables on the Miro board were exported and included in the qualitative data collection. This paper focuses on this methodological evaluation of the online co-creation workshops. The content-related results from the workshops, namely the interventions aiming to reduce burnout risk, will be discussed in a different paper.

Data Analysis

The main author and thesis students transcribed all 12 workshops verbatim, originally for the purpose of the overall WeMeds study. Researchers used Express Scribe Transcription Software (NCH Software, 2022) to aid in the initial transcription of each recording. First, one researcher would conduct the first transcription, while another researcher would listen again to the recordings and check the transcriptions. Second, the main author read carefully through all the transcriptions of each workshop and highlighted the parts that contained feedback about the online format of the workshops, in order to keep focus on the methodological evaluation. As a next step, a thematic analysis was conducted using NVivo (QSR International Pty Ltd., 2020) and following the Braun and Clarke (2006) guidelines (Braun & Clarke, 2006). The researchers decided to focus their analysis on three predetermined themes, namely accessibility, retention and interactivity, for which they based themselves on Keemink et al. (2022). Data saturation was assessed and considered adequate, both thematically (i.e., new information threshold was reached) and retrospectively (i.e., after data collection and analysis was completed) (Guest et al., 2020).

Ethics

The WeMeds study was approved by the Ethics Committee Research xxxx in April 2021 (S64150). All participants provided written informed consent before participation. Due to the online nature of the workshops, the researchers had to email the consent form to the participant, who then returned their signed consent through email. In addition, all participants were made aware beforehand and at the beginning of the workshop that the workshop would be recorded for research purposes only.

Results

This section assesses the feedback and reflections of participants on how the online workshops were perceived and experienced.

Accessibility

The participants reported that the online format of the workshops increased flexibility, and reduced travel time and costs. With regard to the Miro board, participants used words such as ‘very clear and good medium’, ‘accessible, ‘fun’ and ‘easy-to-use’. One bachelor student mentioned: ‘The Miro board was really perfect for this type of workshop!’ and at least 15 other participants mentioned Miro as a very fun part of the workshops. In addition, participants agreed that the visualisations on the Miro board supported a clear and logical flow of the workshop.

However, the online format also showed some challenges. For instance, five participants mentioned their struggle to add text or sticky notes to the Miro board, because they were not familiar with this platform. As a solution, another student or the moderator added the sticky note and requested text to the platform in their place. In another situation during a break-out session, one participant found the platform not very ‘practical’, to which another one responded ‘Yes, I can’t work very well with it yet’. Nevertheless, they were able to finish their exercise as expected without any assistance.

Another disadvantage was that the online format required a stable internet connection, which was a problem for six participants, and resulted in frozen screens or communication difficulties (i.e., delays). To address this issue, these participants turned off their video and managed to continue their participation with audio only. No participant had to leave the workshop due to technical issues. In addition, this technical challenge might have created some frustration, but this was not expressed specifically by any of these participants.

Furthermore, in MS Teams problems were encountered with the breakout rooms, as some participants were consistently removed from the breakout room by the system. In total, this happened to 18 participants. One specialist trainee expressed his frustration by saying: ‘I believe multiple people got kicked out of the breakout rooms, which is a pity as some exercises only last for 10 minutes and then you already miss a substantial part’. To decrease the consequences of this issue, the researchers decided to reorganise the groups when this happened. For instance, when someone was removed from his or her breakout room, this person was allocated to the ‘main room’ of MS Teams, forming a new team with other participants who had also been removed from their breakout rooms. In addition, MS Teams crashed twice at the side of the moderator. As the moderator and the second researcher were in the same room, they could solve this issue by switching computers, so the pace of the workshop was not affected.

Interactivity

Participants considered the workshops highly interactive, partly due to the use of the Miro board and MS Teams. One first-year bachelor student specifically mentioned that it was very interesting to ‘elaborate on each other’s ideas on the Miro board through the brainwriting exercise’, and one specialist trainee stated that he was positively surprised that it was a highly interactive online workshop compared to other webinars that did not use the Miro board. In addition, one GP trainee added that although online, ‘it was easy to stay focused the whole time due to the platform and the interactive exercises’.

Multiple participants liked how Miro facilitated the inclusion of everyone’s opinion through built-in tools, such as sticky notes and the dot voting exercises. One specialist trainee described Miro with the words ‘flashy’, ‘dynamic’ and ‘inspiring’, while one GP trainee considered it a ‘playful way of brainstorming’. Another participant mentioned that Miro created the possibility to work together simultaneously with many people on one whiteboard.

With regard to MS Teams, participants mentioned how the breakout rooms resulted in an increased interactivity and overall active engagement. The breakout rooms had a minimum of two participants and a maximum of five. In particular, participants enjoyed how the breakout rooms contributed to getting to know each other better, increased out-of-the-box thinking and facilitated more in-depth discussions.

Furthermore, multiple participants acknowledged the benefits of meeting medical students, GP trainees and specialist trainees from other Belgian universities to exchange ideas and share struggles. One participant mentioned that this possibility was facilitated by the online format, as it would have been more difficult to gather participants from different regions in face-to-face workshops.

Altogether, participants described that the online format of the co-creation workshops succeeded in creating a safe, informal and neutral environment, where everyone could express their opinions without any judgements or taboos. One first-year master student added: ‘It was very nice that we could express our opinions and that our ideas were recorded for research purposes’, and additionally one bachelor student appreciated ‘the safe spot where every opinion mattered.’

Retention

Originally, the researchers intended to recruit between 6 and 15 medical students, GP trainees and specialist trainees for each workshop. The number of participants who registered for (R) and who actually participated (P) in each workshop is presented in Table 1. This table shows that only two workshops achieved the desired number of participants (i.e., workshop one and two for specialist trainees). Nine workshops had fewer than six participants, with a minimum of three participants (i.e., workshop two for masters and workshop three for bachelors); and one workshop had more participants than 15 (i.e., workshop three for specialist trainees). In addition, Table 1 shows a high attrition rate, with almost half of the registered participants not attending, mostly without informing the researchers beforehand.

Table 1.

Co-Creation Workshop Registrations Versus Actual Attendance.

	Workshop 1		Workshop 2		Workshop 3		Total
	Ra	P^b	Ra	P^b	Ra	P^b	Ra	P^b
Bachelors	9	4	10	5	7	3	26	12
Masters	10	5	9	3	9	5	28	13
GP trainees	8	4	8	5	10	5	26	14
Specialist trainees	12	7	22	13	24	19	58	39
Total	39	20	49	26	50	32	138	78

^aR = Number of registrations for each workshop.

^bP = Number of actual participants in the workshop.

Furthermore, two first-year bachelor’s students specifically mentioned that the reminder that was sent the day of the workshop was very helpful. Without the reminder, they might have forgotten about the workshop.

Some first-year bachelors, masters and GP trainees suggested to encourage more people to join the workshops. However, others appreciated the small groups (even below six participants) as this increased interactivity and engagement and created a safe environment. One GP trainee reported to be ‘initially surprised that there were only four participants in the workshop, however, this created a nice and safe environment’.

Discussion

The results show that participants of this study reported similar benefits and limitations about online co-creation workshops compared to the more traditional online interviews and focus groups. The online format of co-creation workshops has the potential to facilitate the recruitment of medical students, GP trainees and specialist trainees compared to face-to-face workshops, due to increased flexibility, and reduced travel time and costs. The experienced limitations of online workshops were the requirement of a stable internet connection, a wide range of technological problems and a learning curve for the Miro board. Finally, the findings show that the Miro board and MS Teams succeeded in offering a safe, dynamic and interactive environment that incites creative and out-of-the-box thinking.

Online is a Viable Alternative

The main added value of this study is that it provides preliminary evidence showing that online co-creation workshops are a viable alternative compared to their face-to-face counterparts in a co-creation process. Interactivity is considered a crucial aspect in the co-creation process, where researchers intend to minimize social distance and create trust to increase participant disclosure (Weller, 2015).

The Miro board and MS Teams have shown to be useful to ensure a dynamic, interactive and accessible environment to organise co-creation workshops. With regard to creativity, the Miro boards in particular offered tools, including the online whiteboard with in-built functionalities (i.e., sticky notes), that incited out-of-the-box thinking and group discussions. In this online environment, participants were able to work together on future solutions which increased ownership, creativity and empowerment (Palmer et al., 2019).

These findings are in contrast with some former research suggesting that social connections, engagement and interactivity are more challenging in an online context due to the (physical) distance (Abidin & De Seta, 2020; Stewart & Shamdasani, 2017). However, other studies (Kite & Phongsavan, 2017; Matthews et al., 2018) found that, even in the online format, participants are able to feel a strong connection with each other and can be genuinely engaged in the group discussions.

Furthermore, former studies indicated that the level of interactivity and social connection among participants is not mainly determined by the online format or tools (Deakin & Wakefield, 2014; Parker & Tritter, 2006). For instance, Deakin & Wakefield (2016) found that insufficient interactivity in an online format only occurred when participants were more reserved and less responsive overall, also in the face-to-face counterpart.

The use of audio and video might have facilitated this positive perception on engagement, as participants were able to see each other’s facial expressions and non-verbal communication. Former studies confirm that the awareness of being visible via video might positively influence participants attentiveness to what is being discussed (Tuttas, 2015).

Finally, some additional benefits of the online format are that it facilitated the presence of observer-researchers (i.e., possibility to turn of the video and remain in the background) and that recordings of the sessions were made easier due to built-in tools in MS Teams. In this regard, former studies have shown that anonymity might be easier to guarantee in an online format, as participants could use avatars and switch off their video (Woodyatt et al., 2016).

Methodological Considerations

Although the online format seems to be a viable alternative for their face-to-face counterparts, some methodological considerations should be assessed before choosing this format.

The first methodological consideration is the availability of a stable internet connection (for both researchers and participants), and the anticipation of potential technological issues (i.e., frozen screens, communication delays). The researchers implemented certain strategies to proactively counter these issues, such as arranging a backup moderator and organising pilot sessions. These strategies were very similar to the ones reported by former studies (Kite & Phongsavan, 2017; Santhosh et al., 2021; Tuttas, 2015). However, it is likely that technological limitations of online workshops will decrease further in the future, due to advances in technology, internet, and sophisticated online software.

Second, even with digitally-educated target populations, researchers must be aware of a potential ‘digital divide’ that is created with online workshops, as not all participants might have the skills or possibility to join an online workshop (Benson et al., 2021). As shown in the results, even among digitally educated participants, some needed a certain amount of time to get used to the Miro boards and the tools (e.g., sticky notes). This could result in a selection bias beforehand (i.e., non-registration) or less engagement of these participants during the workshop. Consequently, researchers should investigate their target group and be aware that for some populations a face-to-face format or simpler online tools might be needed (Labib et al., 2021). This is to avoid an incompatibility between the format and the target population, which might exclude otherwise eligible participants and/or opinions (Tuttas, 2015). In this regard, hybrid co-creation approaches, that combine online and offline tools, should also be considered and further investigated, as they allow broad engagement of different target groups and might offer a solution for the ‘digital divide’ (Rodriguez Müller et al., 2021).

Third, former research has shown that online interviews and focus groups might be more vulnerable to high drop-out rates compared to their face-to-face counterparts (Curasi, 2001; Deakin & Wakefield, 2014; Kite & Phongsavan, 2017; Matthews et al., 2018; Tuttas, 2015). The drop-out rate in our study was similar to the ones reported in former studies on online focus groups, namely 50% or higher (Kite & Phongsavan, 2017; Matthews et al., 2018; Menary et al., 2021; Tuttas, 2015). Although short-notice changes will always happen, the online format might have made participants feel less committed to attend the workshop (Matthews et al., 2018; Tuttas, 2015). Nevertheless, it is unclear from the results of our study whether the low retention rate can be actually attributed to the online format.

To decrease attrition rates, this study implemented various strategies. For instance, a reminder was sent to participants the day of the workshop. Second, the researchers decided to offer an incentive to the specialist trainees, because their registrations were very low. This strategy was also mentioned by former studies (Benson et al., 2021; Santhosh et al., 2021), which found this equally successful. Finally, in accordance with previous research, we recommend over-recruitment to counter the expected drop-out of 50% or more (Menary et al., 2021).

Interestingly, Deakin and Wakefield (2014) found that the aspect of familiarity is important to increase retention. In their study, they analysed whether a difference existed between potential participants known and unknown to the researchers (Deakin & Wakefield, 2014). They discovered that when participants were known to the researchers, 100% would attend the meeting; while when they were unknown there was a dropout of 15% (Deakin & Wakefield, 2014). This suggests that the familiarity of the researcher to the participant might help in securing the attendance at a workshop. This study did not have the advantage of familiarity, as participants were unknown to the researchers.

Strengths and Limitations

An important strength of this study is that we were able to conduct an explorative analysis about the advantages and challenges of the online format of co-creation workshops, in addition to the overall WeMeds study objectives which focused on developing interventions that aim to reduce burnout risk. Consequently, our research provides valuable explorative information on the use of online platforms, such as MS Teams and Miro boards. Future research may use our insights to elaborate further on the limitations and benefits of online co-creation workshops, and they can also use our findings to organise dynamic and interactive co-creation workshops themselves to address their research questions.

Nevertheless, we should also note several limitations to this study and how the researchers addressed them. First, with regard to the sample size, the researchers tried to achieve groups with between 6 and 15 participants. However, due to difficulties in recruitment and retention, the majority of the workshops had an attendance rate below 6. Nonetheless, researchers were able to collect interesting feedback on the online format of the workshops. Second, researcher bias may be a potential limitation in this type of research. As a strategy to counter this limitation, this study set up an interdisciplinary research team that exchanged ideas via brief discussions after the workshops. Third, due to time and financial constraints, the researchers were restricted to a post-hoc analysis. A randomized controlled trial including one face-to-face workshop (control group) and one online workshop (intervention group) would offer the potential to actually compare the two formats and to include more quantitative data (e.g., word count or number of interruptions). Nonetheless, we feel the first perceptions and experiences of participants are still valuable due to the rather unexplored nature of this topic. Fourth, respondents participated voluntary in the study, implying the possibility of selection bias. It is plausible that those who entered the study share some characteristics that distinguished them from non-participants (e.g., interested in participating in online workshops and digitally-educated).

Conclusions

Organising face-to-face workshops is considered challenging for qualitative research, due to the fixed location, low flexibility and high organisation costs. Our study has shown that online workshops—even in the highly interactive method of co-creation—provide a valuable alternative that might be cheaper, faster and more efficient to collect data and generate ideas, as a safe and interactive environment can be created. This is in particular true when working with a target population that is geographically dispersed, busy and digitally educated. Hence, we argue that online co-creation workshops may be treated as a viable option for qualitative researchers, rather than merely being an alternative or secondary choice when face-to-face workshops are not possible.

Footnotes

Appendix

Acknowledgments

First, we would like to thank the other WeMeds consortium members, who were not involved as co-authors but who contributed significantly to the overall project, namely Prof. Dr. Dirk Devroey, Prof. Dr. Peter Pype, Prof. Dr. Roy Remmen, Prof. Dr. Wim Pinxten and Dr. Sofie vandenbroeck. In addition, we would like to express our gratitude to the students that contributed to the organisation and qualitative analyses of the workshops: Annabel Bijnens, Sofie Boghe, Sofie Van den Acker and Charlotte Vanneck. Third, we would like to thank Elke Smeers, who provided support with the logistical organisation of the online workshops.

Author Contributions

AB, LB, NM, HK, EVH, KVDB and LG all meet the ICMJE criteria. AB en LG conceptualised the design and implementation of the data collection and developed the content of the workshops. LB, NM, HK, EVH and KVDB were observers during the workshops, and were part of the interdisciplinary research team that discussed all findings. LG supervised the overall WeMeds study. All authors contributed to the article and approved the submitted version.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is part of the WeMeds study, which received internal funds from KU Leuven (Category 3) under C3/20/040.

Ethical Approval

The WeMeds study was approved by the Ethics Committee Research UZ/KU Leuven in April 2021 (S64150).

Data Availability

Further inquiries with regard to collected data can be directed to the corresponding author.

ORCID iDs

Anke Boone

Nele Michels

Kris Van den Broeck

References

Abidin

De Seta

(2020). Private messages from the field. Journal of Digital Social Research, 2(1), 1–19. https://doi.org/10.33621/jdsr.v2i1.35

Altman

Huang

T. T. K.

Breland

(2018). Design thinking in health care: Systematic review. Preventing Chronic Disease, 15(E117), 1–13. https://doi.org/10.5888/pcd15.180128

Archibald

M. M.

Ambagtsheer

R. C.

Casey

M. G.

Lawless

(2019). Using zoom videoconferencing for qualitative data collection: Perceptions and experiences of researchers and participants. International Journal of Qualitative Methods, 18, 1–8. https://doi.org/10.1177/1609406919874596

Benson

Pedersen

Tsalis

Futtrup

Dean

Aschemann-Witzel

(2021). Virtual co-creation: A guide to conducting online co-creation workshops. International Journal of Qualitative Methods, 20, 1–15. https://doi.org/10.1177/16094069211053097

Berwick

D. M.

(2016). Era 3 for medicine and health care. JAMA, 315(13), 1329–1330. https://doi.org/10.1001/jama.2016.1509

Boone

Godderis

(2021). WeMeds: Healthy & work-engaged medical doctors. Retrieved August 9, 2022, from www.wemeds.be

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Chathoth

Altinay

Harrington

R. J.

Okumus

Chan

E. S.

(2013). Co-production versus co-creation: A process based continuum in the hotel service context. International Journal of Hospitality Management, 32(1), 11–20. https://doi.org/10.1016/j.ijhm.2012.03.009

Curasi

C. F.

(2001). A critical exploration of face-to face interviewing vs. computer-mediated interviewing. International Journal of Market Research, 43(4), 361–375. https://doi.org/10.1177/147078530104300402

10.

Deakin

Wakefield

(2014). Skype interviewing: Reflections of two PhD researchers. Qualitative Research, 14(5), 603–616. https://doi.org/10.1177/1468794113488126

11.

Dobe

Gustafsson

Walder

(2023). Co-creation and stroke rehabilitation: A scoping review. Disability and Rehabilitation, 45(3), 562–574. https://doi.org/10.1080/09638288.2022.2032411

12.

Forlizzi

Battarbee

(2004) Understanding experience in interactive systems. DIS2004—designing interactive systems: Across the spectrum (261–268). Carnegie Mellon University. https://doi.org/10.1145/1013115.1013152

13.

Fusco

Marsilio

Guglielmetti

(2020). Co-production in health policy and management: A comprehensive bibliometric review. BMC Health Services Research, 20(1), 1–18. https://doi.org/10.1186/s12913-020-05241-2

14.

Greenhalgh

Jackson

Shaw

Janamian

(2016). Achieving research impact through co-creation in community-based health services: Literature review and case study. The Milbank Quarterly, 94(2), 392–429. https://doi.org/10.1111/1468-0009.12197

15.

Guest

Namey

Chen

(2020). A simple method to assess and report thematic saturation in qualitative research. PLoS One, 15(5), 1–17. https://doi.org/10.1371/journal.pone.0232076

16.

Halvorsrud

Kucharska

Adlington

Rüdell

Brown Hajdukova

Nazroo

Haarmans

Rhodes

Bhui

(2021). Identifying evidence of effectiveness in the co-creation of research: A systematic review and meta-analysis of the international healthcare literature. Journal of Public Health, 43(1), 197–208. https://doi.org/10.1093/pubmed/fdz126

17.

Howlett

(2022). Looking at the ‘field’ through a zoom lens: Methodological reflections on conducting online research during a global pandemic. Qualitative Research: QR, 22(3), 387–402. https://doi.org/10.1177/1468794120985691

18.

Huijnen

C. A. G. J.

Lexis

M. A. S.

Jansens

De Witte

L. P.

(2017). How to implement robots in interventions for children with autism? A co-creation study involving people with autism, parents and professionals. Journal of Autism and Developmental Disorders, 47(10), 3079–3096. https://doi.org/10.1007/s10803-017-3235-9

19.

IDEOIDEO . (2015). The field guide to human-centered design. IDEOIDEO Retrieved from www.ideo.org

20.

Janghorban

Roudsari

R. L.

Taghipour

(2014). Skype interviewing: The new generation of online synchronous interview in qualitative research. International Journal of Qualitative Studies on Health and Well-Being, 9(1), 1–4. https://doi.org/10.3402/qhw.v9.24152

21.

Jones

(2018). Contexts of Co-creation: Designing with system stakeholders. In Jones

Kijima

(Eds.), Systemic design: Theory, methods, and practice (pp. 3–52). Springer. https://doi.org/10.1007/978-4-431-55639-8_1

22.

Keemink

J. R.

Sharp

R. J.

Dargan

A. K.

Forder

J. E.

(2022). Reflections on the use of synchronous online focus groups in social care research. International Journal of Qualitative Methods, 21, 1–13. https://doi.org/10.1177/16094069221095314

23.

Kite

Phongsavan

(2017). Insights for conducting real-time focus groups online using a web conferencing service. F1000Research, 6(122), 1–14. https://doi.org/10.12688/f1000research.10427.1

24.

Krouwel

Jolly

Greenfield

(2019). Comparing skype (video calling) and in-person qualitative interview modes in a study of people with irritable bowel syndrome-an exploratory comparative analysis. BMC Medical Research Methodology, 19(1), 219. https://doi.org/10.1186/s12874-019-0867-9

25.

Labib

Pizzolato

Stappers

P. J.

Evans

Lechner

Widdershoven

Tijdink

(2021). Using co-creation for guideline development—how why and when? OSF Preprint (pp. 0–1). Retrieved from https://osf.io/cg3rw/

26.

Lee

J.-J.

Jaatinen

Salmi

Mattelmäki

Smeds

Holopainen

(2018). Design choices framework for co-creation projects. International Journal of Design, 12(2), 15–31.

27.

Leemann

Jeszenszky

Steiner

Studerus

Messerli

(2020). Linguistic fieldwork in a pandemic: Supervised data collection combining smartphone recordings and videoconferencing. Linguistics Vanguard, 6(s3), 1–16. https://doi.org/10.1515/lingvan-2020-0061

28.

Martin

Hanington

(2012). Universal methods of design. Rockport Publishers.

29.

Matthews

K. L.

Baird

Duchesne

(2018). Using online meeting software to facilitate geographically dispersed focus groups for health workforce research. Qualitative Health Research, 28(10), 1621–1628. https://doi.org/10.1177/1049732318782167

30.

Menary

Stetkiewics

Nair

Jorasch

Nanda

A. K.

Guichaoua

Davies

J. A. C.

(2021). Going virtual: Adapting in-person interactive focus groups to the online environment. Emerald Open Research, 3(6), 1–14. https://doi.org/10.35241/emeraldopenres.14163.2

31.

Miro (2022a). Brainwriting template. Miro. Retrieved November 7, 2022, from https://miro.com/templates/brainwriting/

32.

Miro (2022b). Customer journey map template. Miro. Retrieved November 7, 2022, from https://miro.com/templates/customer-journey-map/

33.

Miro (2022c). Dot voting template. Miro. Retrieved November 7, 2022, from https://miro.com/templates/dot-voting/

34.

NCH Software (2022). Express scribe transcription. NCH.

35.

Osborne

S. P.

Radnor

Strokosch

(2016). Co-production and the co-creation of value in public services: A suitable case for treatment? Public Management Review, 18(5), 639–653. https://doi.org/10.1080/14719037.2015.1111927

36.

Palmer

V. J.

Weavell

Callander

Piper

Richard

Maher

Boyd

Herrman

Furler

Gunn

Iedema

Robert

(2019). The participatory zeitgeist: An explanatory theoretical model of change in an era of coproduction and codesign in healthcare improvement. Medical Humanities, 45(3), 247–257. https://doi.org/10.1136/medhum-2017-011398

37.

Palumbo

(2016). Contextualizing co-production of health care: A systematic literature review. International Journal of Public Sector Management, 29(1), 72–90. https://doi.org/10.1108/IJPSM-07-2015-0125

38.

Parker

Tritter

(2006). Focus group method and methodology: current practice and recent debate. International Journal of Research & Method in Education, 29(1), 23–37. https://doi.org/10.1080/01406720500537304

39.

Parrado

Van Ryzin

G. G.

Bovaird

Löffler

(2013). Correlates of co-production: Evidence from a five-nation survey of citizens. International Public Management Journal, 16(1), 85–112. https://doi.org/10.1080/10967494.2013.796260

40.

QSR International Pty Ltd . (2020). NVivo (version 12). QSR International Pty Ltd. Retrieved from https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home

41.

Renedo

Marston

C. A.

Spyridonidis

Barlow

(2015). Patient and public involvement in healthcare quality improvement: How organizations can help patients and professionals to collaborate. Public Management Review, 17(1), 17–34. https://doi.org/10.1080/14719037.2014.881535

42.

Richard

Sivo

S. A.

Orlowski

Ford

R. C.

Murphy

Boote

D. N.

Witta

E. L.

(2021). Qualitative research via focus groups: Will going online affect the diversity of your findings? Cornell Hospitality Quarterly, 62(1), 32–45. https://doi.org/10.1177/1938965520967769

43.

Rodriguez Müller

A. P.

Casiano Flores

Albrecht

Steen

Crompvoets

(2021). A scoping review of empirical evidence on (digital) public services co-creation. Administrative Sciences, 11(4), 130. https://doi.org/10.3390/admsci11040130

44.

Sanders

E. B.-N.

Stappers

P. J.

(2014). Probes, toolkits and prototypes: Three approaches to making in codesigning. CoDesign, 10(1), 5–14. https://doi.org/10.1080/15710882.2014.888183

45.

Santhosh

Rojas

J. C.

Lyons

P. G.

(2021). Zooming into focus groups: Strategies for qualitative research in the era of social distancing. ATS Scholar, 2(2), 176–184. https://doi.org/10.34197/ats-scholar.2020-0127PS

46.

Schnall

Rojas

Bakken

Brown

Carballo-Dieguez

Carry

Gelaude

Mosley

J. P.

Travers

(2016). A user-centered model for designing consumer mobile health (mHealth) applications (apps). Journal of Biomedical Informatics, 60, 243–251. https://doi.org/10.1016/j.jbi.2016.02.002

47.

Stein

Meurer

Boden

Wulf

(2017). Mobility in later life—appropriation of an integrated transportation platform. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA, 5716–5729. https://doi.org/10.1145/3025453.3025672

48.

Stewart

D. W.

Shamdasani

(2017). Online focus groups. Journal of Advertising, 46(1), 48–60. https://doi.org/10.1080/00913367.2016.1252288

49.

Tschimmel

(2012). Design Thinking as an effective toolkit for innovation Proceedings of the XXIII ISPIM Conference: Action for Innovation: Innovating from Experience (1–18). Barcelona, Spain.

50.

Tuttas

C. A.

(2015). Lessons learned using web conference technology for online focus group interviews. Qualitative Health Research, 25(1), 122–133. https://doi.org/10.1177/1049732314549602

51.

Unleash . (2022). Workshop on prototyping. Retrieved November 7, 2022, from https://miro.com/miroverse/workshop-on-prototyping/

52.

Vechakul

Shrimali

B. P.

Sandhu

J. S.

(2015). Human-centered design as an approach for place-based innovation in public health: A case study from Oakland, California. Maternal and Child Health Journal, 19(12), 2552–2559. https://doi.org/10.1007/s10995-015-1787-x

53.

VMware Tanzu Labs . (2022). Stakeholder map. VMware Tanzu Labs. Retrieved November 7, 2022, from https://miro.com/miroverse/stakeholder-map/

54.

Voorberg

W. H.

Bekkers

V. J. J. M.

Tummers

L. G.

(2014). A systematic review of co-creation and co-production: Embarking on the social innovation journey. Public Management Review, 17(9), 1333–1357. https://doi.org/10.1080/14719037.2014.930505

55.

Weller

(2015). The potentials and pitfalls of using skype for qualitative (longitudinal) interviews (p. 50). National Centre for Research Methods Working Paper. Retrieved from https://eprints.ncrm.ac.uk/3757/1/Susie.Weller.pdf.

56.

Woodyatt

C. R.

Finneran

C. A.

Stephenson

(2016). In-person versus online focus group discussions: A comparative analysis of data quality. Qualitative Health Research, 26(6), 741–749. https://doi.org/10.1177/1049732316631510