Sage Journals: Discover world-class research

Abstract

Objectives

We aimed to develop a video animation knowledge translation (KT) resource to explain the purpose, use and importance of evidence synthesis to the public regarding healthcare decision-making.

Methods

We drew on a user-centred design approach to develop a spoken animated video (SAV) by conducting two cycles of idea generation, prototyping, user testing, analysis, and refinement. Six researchers identified the initial key messages of the SAV and informed the first draft of the storyboard and script. Seven members of the public provided input on this draft and the key messages through think-aloud interviews, which we used to develop an SAV prototype. Seven additional members of the public participated in think-aloud interviews while watching the video prototype. All members of the public also completed a questionnaire on perceived usefulness, desirability, clarity and credibility. We subsequently synthesised all data to develop the final SAV.

Results

Researchers identified the initial key messages as 1) the importance of evidence synthesis, 2) what an evidence synthesis is and 3) how evidence synthesis can impact healthcare decision-making. Members of the public rated the initial video prototype as 9/10 for usefulness, 8/10 for desirability, 8/10 for clarity and 9/10 for credibility. Using their guidance and feedback, we produced a three-and-a-half-minute video animation. The video was uploaded on YouTube, has since been translated into two languages, and viewed over 12,000 times to date.

Conclusions

Drawing on user-centred design methods provided a structured and transparent approach to the development of our SAV. Involving members of the public enhanced the credibility and usefulness of the resource. Future work could explore involving the public from the outset to identify key messages in developing KT resources explaining methodological topics. This study describes the systematic development of a KT resource with limited resources and provides transferrable learnings for others wishing to do similar.

Keywords

public involvement user testing user experience knowledge translation evidence synthesis education video animation resource

Introduction

Members of the general population are exposed to a huge variety of health claims from a vast array of sources including the internet, social media, television, radio, and newspapers and friends and family.^1,2 For example, research has previously highlighted that the most common information sources for primary care patients were the internet and patients’ physicians, with the former preferred for ease of access.³ Information found online is often the information that the public bases its healthcare decisions on,⁴ yet many of these claims are not reliable and people can find it challenging to assess the reliability of these claims.^5,6 In addition to this, people's beliefs about and use of treatments with limited evidence may be harmful. For example, in 2018, a large-scale study by Johnson et al. found that patients who received complementary medicine were more likely to refuse conventional cancer treatment and also had a two-fold greater mortality risk compared with no complementary medicine use.⁷

Ensuring that healthcare decisions made at all levels (e.g., by policymakers and healthcare managers, practitioners, patients and the public) are based on evidence has never been so important. Recent technological advances have led to the availability of vast amounts of health information; however, the sheer quantity of information available can be overwhelming and information overload has been identified as an issue for both clinicians⁸ and the public alike.⁶ Evidence synthesis plays a crucial role in ensuring that healthcare is based on relevant, high quality and up-to-date evidence to optimise health outcomes.⁹ Evidence synthesis is an essential way of providing a balanced and comprehensive critical appraisal of all of the available evidence on a topic and is important for all stakeholders, including healthcare professionals, policy-makers and the public, including patients and their caregivers.

Knowledge translation, or “KT”, is the ‘synthesis, dissemination, exchange and ethically sound application of knowledge to improve health, provide more effective health services and products and strengthen the health care system’.¹⁰ KT plays a key role in ensuring that research is available and accessible to the public in a clear and concise manner. Previous research has shown that KT interventions can have beneficial effects on the provision of evidence-based care and patient knowledge and function.^11,12 Videos as explanatory resources have been previously shown to effectively communicate complex health information to audiences with different health literacy levels.¹³ Spoken animated videos or video animations, in particular, are one of the most impactful ways of conveying information and show high levels of knowledge gain and retention over long periods.¹⁴

KT resources that explain evidence synthesis to a general audience, i.e., not aimed at researchers, are somewhat limited. KT efforts have predominantly focused on increasing awareness and understanding for patients, policymakers and the public regarding specific clinical topics, disease areas and potential treatments,^15–18 rather than explaining methodological aspects of healthcare research, such as the importance of evidence synthesis. In addition, there is a distinct lack of evidence regarding the development of existing KT resources and tools and how to incorporate an evidence-based approach, with a limited number of examples exist describing this process.^19–21

Some examples of studies that do exist in this area include the development of ‘My Asthma Diary’, an arts-based KT tool for parents of children with asthma.^20,22 Archibald et al. (2018) describe a multi-stage process involving conducting a literature review and qualitative interviews with parents to develop four eBook prototypes.²⁰ The prototypes were subsequently tested for usability with another sample of parents to determine the final resource.²² The authors concluded that using an iterative user-centred design approach with multiple stakeholders (e.g., researchers, practitioners, artists and parents of children with asthma) was particularly valuable in developing a useful and usable KT resource.^20,22 Archibald et al. (2020) also describe the process of co-designing four video-based KT resources with and for older people and their carers.²¹ The authors highlight the power of video-based KT resources including animations for communicating about health, frailty and self-management. The study also showcases an excellent example of conducting iterative feedback cycles to develop fit-for-purpose video resources and again highlights the value of including multiple stakeholder perspectives. However, both of these resources focus on clinical, rather than methodological topics.

As evidenced by existing literature detailing KT resource development, meaningful patient and public involvement (PPI) can play a vital role in ensuring the relevance and potential impact of health research.^23,24 However, PPI has predominantly been utilised in clinical-based research to date, with less attention to its application in implementation or methodological research.²⁵ In response to this, a Ugandan research team recently developed KT resources through PPI approaches. These resources were not only found to be highly effective in a large trial,²⁶ but their success was in part traced back to the user-centred development approach.²⁷

This study aimed to develop a spoken video animation to explain evidence synthesis and its importance to the public by drawing on a user-centred approach. This involved conducting iterative cycles of prototyping, testing and analysis with members of the public to create a useful, usable, understandable, credible and desirable resource.

Methods

Ethical approval

There was no national public ethics committee in existence at the time this study was conducted, therefore guidance was sought from the local Galway University Hospital Research Ethics Committee. Given the aim of the study (to produce the KT resource in collaboration with researchers and members of the public) the Chair indicated that this project did not require ethical approval. All members of the public provided written or verbal informed consent prior to their involvement.

Study design

We drew on user-centred approaches to determine the format, content and structure of the video animation (Figure 1). User-centred design entails an iterative design cycle with multiple steps: idea generation, prototyping, user testing and analysis and refinement in collaboration with end users, and has been previously employed to develop research resources with input from members of the public.^26,28 We defined the public according to the National Institute for Health Research definition as ‘patients, potential patients, carers and people who use health and social care services as well as people from specific communities and from organisations that represent people who use services’.²⁹

Figure 1.

Iterative steps of the user-centred approach. SAV: Spoken animated video, KT: Knowledge translation.

Feedback was obtained during the user testing through concurrent think-aloud interviews and a questionnaire. Concurrent think-aloud interviews involve observing people as they engage with a prototype of the product and listening to them speak aloud any words in their mind as they engage, with participants acting as quasi-researchers.³⁰ The think-aloud modality of carrying out interviews aims to avoid the fallibility that comes with retrospective analysis of an experience.^30,31 This allows for feedback to be honest, clear and immediate.

After the interviews, members of the public completed a questionnaire previously developed by Nsangi et al.,²⁶ which was based on Morville's honeycomb model of user experience.³² Specifically, people rated the prototype on a scale of 1–10 for specific facets of the honeycomb model about usefulness, desirability, valuableness and/or clarity, and credibility (1 being less useful, 10 being more useful). The rating scale was verbally explained to all participants by CD during interviews and any queries regarding scoring or data collection were clarified before or during the interviews. We chose to use Morville's honeycomb model of user experience as it is an holistic model encompassing multiple facets beyond than just ‘usability’ constructs,³² and was previously shown to be useful in similar studies targeting varied public audiences.^28,33 This model helped structure the interviews questions, and to identify key areas to target in developing and improving the resource. Interviews and questionnaires were conducted by CD in-person or via telephone, depending on the person's individual preference. We chose to provide both modes as options for people, as although both in-person and telephone interviews have advantages and disadvantages (e.g., ability to read body language versus ease and reduced cost), recent research has shown that both options are valid approaches to collecting qualitative data.³⁴ For interviews carried-out via telephone, participants were either previously sent copies of the material or it was screen-shared to them through online video-calling software.

Idea generation

The main aim of the Idea Generation stage was to determine the key messages and purpose of the video animation. We used the KT Planning Primer template from the Public Agency of Canada³⁵ to structure this process. Briefly, the template helps researchers to document their intended audience, objectives, main messages, resource format, development and delivery process, dissemination, available resources, main barriers and desired impact for the project. First, ET and CD completed the template sections referring to the intended audience, objectives, format, development and delivery process and resources. Next, using purposive sampling, authors identified six researchers from Ireland, the UK, Norway, and Australia who had combined research and practice expertise in evidence synthesis and KT. The basis of purposive sampling was to have an international pool of experienced researchers in the particular area of research that we were carrying out to help shape the foundation of the messages to be relayed in our KT resource. Researchers received the partially completed KT template to review before participating in an unstructured one-to-one telephone discussion with CD to provide feedback. We particularly sought input regarding the main messages of the animation, ideas for dissemination, potential barriers and desired impact, and any general thoughts on the project or other components of the KT template. CD recorded researchers’ feedback directly into the KT template during the telephone discussions. Subsequently, CD collated input from all researchers and entered it into the relevant section of the KT template. ET reviewed this synthesised input before moving to the next stage of prototype development. Any uncertainties or conflicting opinions between researchers were discussed between CD, ET, and DD.

Prototyping – round one

Using the researchers’ feedback in the Idea Generation stage, we created a first version of the narration script and rough storyboard outline. The storyboard consisted of sketches of each scene (seven scenes in total) and verbal descriptions of approximate visuals scene-by-scene.

User testing – round one

We recruited members of the public according to the previously detailed NIHR definition of ‘the public’.²⁹ We used a combination of convenience and snowball sampling via word of mouth (e.g., emailing colleagues and asking friends and acquaintances if they would know anyone willing to participate in any given demographic bracket) to recruit people with limited previous experience of evidence synthesis or health research, across a variety of age and gender. Specifically, we sought to include at least one male and female in each of the following age groups; <18 years, 19–30, 31–60 and >60. The timeframe of the recruitment for participants was one week. People were not offered any incentives for their involvement, and we did not collect any personal characteristics data aside from age and gender. They were informed from the outset that the purpose of the project was to develop a video animation resource to explain evidence synthesis to a public audience. The user testing sought to obtain feedback on the storyboard prototype and determine necessary changes to the narration script, the storyboard, and the proposed animation's visual and aesthetic style. People were also shown four types of animation styles - felt stop motion, 2D vector, paper cut-out look and whiteboard drawing. During the think-aloud interviews, members of the public were asked for their first impressions of the script, the visuals accompanying the script, and their impressions of the ‘main messages’ of the proposed video. All were given identical instructions to “think their thoughts aloud and say anything on their mind at any point while interacting with the resource”. Voiced thoughts and comments were documented by CD as facilitator notes.

After the interviews, people were asked to rate the script and storyboard on a scale of 1–10 for specific facets of the honeycomb model about usefulness, desirability, valuableness and credibility. They were also asked to identify any potential missing aspects, content that should be removed, or general suggestions for improving the script and storyboard. The complete questionnaire is provided in Appendix 1. We also obtained feedback on the initial script and storyboard from the six researchers and feedback from one specialist PPI advocate (DS) with substantial experience in KT for the public. We sent a copy of the script and storyboard in Word format to each person for review. All commented on the document using track changes and returned feedback to CD.

Analysis and refinement – round one

We synthesised and analysed the information and feedback gathered from the user testing with members of the public, researchers and our experienced PPI advocate. Specifically, we extracted all comments suggesting potential edits or changes to the prototype script and visuals from the think-aloud facilitator notes and questionnaires. We grouped these into similar categories/concepts to identify a list of main suggestions, spanning from important edits to be made (such as a participant misunderstanding a message or script element) to more minor potential changes (such as the texture or size of aspects of the visuals to be modified). These suggestions were subsequently discussed for feasibility of changes by ET and CD, and where suggestions differed across contributors or consensus was not reached, a third member of the study team (DD) was consulted.

Prototyping – round two

Based on the feedback from the first round of user testing, CD created a second version of the script and recorded an accompanying audio narration. With input from EP, CD developed an initial video prototype from the storyboard sketches.

User testing – round two

We again used snowball sampling via word of mouth to recruit an additional seven members of the public of varying ages and gender for the second round of user testing using the same criteria as in the round one user testing. A new group of members of the public were recruited so the feedback that was gathered in round two could not be influenced by the editing suggestions made in user testing round one. As with the first round, CD conducted think-aloud interviews with all seven members. This time, people were asked to watch the initial video prototype and talk through their first impressions of the video narration, animation and visuals and main messages, with feedback documented by CD as facilitator notes. Stakeholders then completed another questionnaire (Appendix 2) which asked users to rate the video on a similar scale of 1–10 according to aspects of usefulness, desirability, clarity and credibility with questions slightly modified to be specific to the video animation. This prototype was also sent to the researchers and our PPI advocate for feedback before the final stage of refinement.

Analysis and refinement – round two

We synthesised and analysed information, feedback and answers gathered from the second cycle of user testing and used this to inform the final version of the video, including final changes to visuals, script and the speed of narration and animation. In this final stage, we also added further audio aspects to the video, such as the backing music track and sound effects for the animations’ movement to represent the animation's final look and feel of the animation accurately. Similarly to round one, any tensions between participant perceptions and researchers’ priorities were resolved by seeking input from a third member of the primary research team (DD). Multiple iterations of the video were rendered to finalise aesthetic details such as frame-rate or colour palette. Finally, we equalised the audio levels and completed any final adjustments before rendering the final entire video.

Results

Idea generation

CD and ET completed the sections of the KT Planning Primer Template³⁵ as outlined in Table 1. The determined audience for our resource was to be as large as possible and not limited by age (within reasonable margins) or geographical location. In the absence of clear and definitive guidance on desired video animation length for similar resources, ET and CD determined that the video should be as brief as possible, while incorporating the key messages, with an absolute cut-off of 4 min. In addition to this, the research stakeholders overall agreed that the following main messages should be prioritized: 1) Explaining the importance of evidence synthesis for not making healthcare decisions based on a single piece of research due to potential for skewed or inaccurate views from one study alone, 2) Explaining what evidence synthesis is as the process of combining separate individual studies on a singular topic in a structured and trustworthy way, and 3) Explaining the usefulness of evidence synthesis for the audience, with an example of how it can inform everyday healthcare decision-making. The researchers also identified the importance of including funding information in the video animation and using Cochrane resources to disseminate it.

Table 1.

Kt planning primer template.

KT Template Section	Completed Sections
WHO do we want to reach?	Members of the public with little to no research or professional healthcare background
WHY – What are our objectives?	To create a video-based animation resource for our audience, to increase awareness and understanding of what evidence synthesis is (beyond systematic reviews alone) and its importance and role in informing health decision-making amongst the public
HOW – What format will we use?	We will use a video-based Spoken Animation resource. Animation length will likely be ∼3 min + /-1 min (in order to include all determined-to-be necessary key messages).
HOW – How will we develop the resource?	The resource will be co-produced with groups of stakeholders, using a user-centred approach. The first group will comprise researchers with experience of evidence synthesis and knowledge translation to determine the key messages. The second group will comprise members of the public to user test the video development in think-aloud interviews. An adapted version of Morville's Honeycomb Framework for collecting and analysing data will structure the user testing.
HOW – Choose the delivery	YouTube, social media e.g., Twitter
HOW – Assess your resources	One summer student (CD) and two supervisors – evidence synthesis (ET) and technology (EP)

Prototyping - round one

The initial prototype storyboard consisted of sketches of seven scenes in total, with a draft narration script and verbal descriptions of approximate accompanying visuals for each scene.

User testing - round one

Seven members of the public were involved in the first round of user testing (details in Table 2). We were successful in recruiting one male and female participant from each age category except in the <18 years category where one male was recruited, due to the fact that minors are more difficult to recruit given the necessity of parent/guardian consent. Interviews were completed in 40–60 min with each participant. People rated the storyboard prototype as median 9 (range 8–10) for usefulness, 7 (range 5–10) for desirability, 9 for value (range 6–10) and 9 for credibility (range 8–10) (full dataset provided in Appendix 3). Overall, they commented on the script and visuals favourably. Predominantly, suggestions for improvement pertained to making specific sentences more concise and using simpler language and having less imagery and activity in the visuals. One person mentioned the need to further clarify the impact of evidence syntheses on everyday life. Four people preferred the felt animation style, with one preferring the 2D vector style, and another preferring the whiteboard style.

Table 2.

Members of the public stakeholder demographics round 1 and round 2.

	Round 1	Round 2
Variable	N (%)	N (%)
Gender	_
Female	3 (43)	3 (43)
Male	4 (57)	4 (57)
Age group, years
<18	1 (14)	1 (14)
19–30	2 (29)	2 (29)
31–60	2 (29)	2 (29)
>60	2 (29)	2 (29)
Interview format
In-person	4 (57)	3 (43)
Telephone	3 (43)	4 (57)

Similarly, the research and PPI advocate stakeholders commented favourably overall on the script and visuals. They provided suggestions in relation to wording (e.g., ‘as opposed to a single research study, say “a single piece of evidence”’) and ensuring the visuals of the video were inclusive and reflective of diversity (e.g., ‘a number of people emerge - including women, different races and disabilities, a sense of people working together…Each bring different papers and begin to compare and contrast’).

Analysis and refinement - round one

After discussion with ET and DD, CD assimilated the synthesised comments and critiques from the first round of user testing and used them to edit the script, audio narrative and storyboard in order to transition to Round 2 of prototype creation.

Prototyping - round two

A basic video animation was created from the storyboard sketches. The video was kept simple with basic vector animations (simple linear movement) to allow easy changes further in the production process (Figure 2). The audio narration was added to the animation, and in-sync sound effects were added to the animation to develop a second prototype of the video animation resource for user testing in Round 2.

Figure 2.

Animation style for round 2 spoken animated video prototype.

User testing - round two

Seven additional members of the public were involved in the second round of user testing (details in Table 2). Again interviews lasted between 40–60 min. They rated the prototype video animation as median 9 (range 7–9) for usefulness, 8 (range 5–9) for desirability, 8 (range 4–10) for clarity, and 9 (9–10) for credibility (full dataset provided in Appendix 3). Overall, people liked the visuals and script. Suggestions for editing are mostly related to slowing the speed of the visuals and audio.

Similarly, researchers liked the narration, visuals and script. Minor changes were suggested, such as removing a mnemonic that CD and ET had initially developed to summarise steps for checking the quality of a systematic review to avoid confusion with existing systematic review quality checklists and rephrasing part of the script to be more supportive rather than dictatorial (e.g., ‘hopefully you should feel equipped in using research to make more informed decisions about healthcare’ rather than ‘evidence synthesis can and should directly impact how you make decisions about health care’).

Analysis and refinement - round two

The final version of the video animation was developed using the feedback from the second round of user testing. CD and EP created representative sound effects, a backing audio track, and intro and outro scenes. CD applied a felt stop-motion animation style in response to user feedback from the first round of user testing (Figure 3). After numerous pre-renderings CD rendered and uploaded the final spoken animated video resource onto the Evidence Synthesis Ireland and Cochrane YouTube channels.^36,37

Figure 3.

Animation style for final video animation.

The video can be viewed at the following link: https://www.youtube.com/watch?v = nZR0xQmZVQg. At the time of writing (29/03/2023), the video has over 3300 views on the Evidence Synthesis Ireland YouTube channel and over 12,000 views on the Cochrane YouTube channel. The average view duration on the Cochrane YouTube channel is 2:07, with 60.6% being the average percentage of the video viewed. 31.7% watch the whole video, with 74% still watching at the 0:30 mark. Resource-use for the project was minimal, as this was conducted over 8 weeks as an undergraduate summer student project supported by Evidence Synthesis Ireland for €2000.

Discussion

Drawing on a user-centred approach, we created a video-based KT resource to explain evidence synthesis to the public. Involving multiple stakeholders in the development of the resource, including members of the public, enabled us to target and enhance key facets of the Morville honeycomb of user experience,³² namely the usefulness, desirability, valuableness, clarity and credibility of the resource. This study describes the structured and systematic development of a KT resource with limited resources and provides an important template and transferrable learnings for others wishing to develop similar products in an evidence-based manner.

Social media & health information

Advances in online technology and social media have seen substantial increases in the amount of health information available to and accessed by the public. While there are obvious benefits to using social media and online resources for mass communication of health information, such tools are typically unregulated and the information shared can be of varying quality and consistency.⁵ The recent COVID-19 pandemic has shone a spotlight on the twin issues of misinformation and disinformation, i.e., the inadvertent or deliberate and coordinated spread of misleading and false information, respectively.³⁸ The severity of this situation has led to the coining of the new term ‘infodemic’ by the World Health Organization as the ‘overabundance of information and the rapid spread of misleading or fabricated news, images, and videos’.³⁹ Enabling the public to think critically about health claims is a crucial component in the fight to combat the effects of misinformation and disinformation. Moreover, understanding the value and relative importance of a single study on a topic compared to the synthesised body of evidence for a topic is a key part of this.⁴⁰ Developing an online KT resource that explains evidence synthesis to the public can leverage the advantages afforded by social media and be used to spread accurate and valid health information and help people make more informed decisions.

Comparison with prior work

It has been previously estimated that approximately 85% of biomedical research is wasted.⁴¹ Therefore, to minimise research waste and optimise the investment in KT resources and dissemination strategies, it is important to ensure that these are well developed and evidence-based. However, previous research has shown that such resources are often poorly explored and evaluated.^42,43 Recent work by Cross et al. involved participation between families and researchers in the area of childhood disability to develop, implement and evaluate a web-based KT resource to promote adoption of the ‘F-words’ concepts (function, family, fun, friends and future).¹⁹ The project used the Knowledge-To-Action Framework⁴⁴ to guide this process and culminated in developing a theory-informed resource that was deemed relevant and meaningful to its target audiences. In a similar way, we drew on Morville's honeycomb model of user experience to guide our process and ensure the development of a resource that is useful, desirable, valuable, clear and credible.

In our project, we drew on a user-centred design approach to obtain feedback which had been heavily guided by the methodology used by Semakula et al..²⁸ Semakula et al. applied a user-centred design approach to develop an educational podcast for the public comprising eight episodes focusing on the Informed Health Choices key concepts. The Informed Health Choices key concepts are an evidence-based list of 32 concepts that aim to enable people to understand and assess the trustworthiness of health claims.⁴⁰ Semakula et al. applied five cycles of user testing and feedback with several different groups including journalists, researchers, parents and other members of the public. This contrasts to our approach where we applied two cycles of user testing. However, this project was much larger in scope, took approximately three years to complete and required substantially more resources than our work.²⁸ Applying similar stages of a user-centred approach (e.g., idea generation, prototyping, user testing, analysis) enabled us to collect feedback in a structured and iterative manner. This enabled us to build the KT resource gradually with the input of stakeholders embedded throughout the process, using minimal resources. As such, our work adds to the extensive work of Semakula et al. by providing additional examples of what is possible to achieve in this area with a scaled-back approach.

Areas for future research

Evaluating the effectiveness of our video and the impact of certain video characteristics (e.g., duration) was outside the scope of this project. There is limited evidence-based definitive guidance on determining the optimal duration of videos such as ours, with much variability reported in the literature. For example, Zaila et al. (2020) examined YouTube educational videos on men's health and found that video length ranged from 29 to 51 min, with a mean video duration of 39 min 41 sec.⁴⁵ Actual mean watch time by viewers ranged from 3:45 to 8:30 min. Similarly, Guo et al. (2014) examined student engagement in online educational videos and found that videos ranged up to 40 min in length, and that the median engagement time was at most 6 min, regardless of total video length.⁴⁶ They also found that shorter videos (0–3 min) were most engaging, and recommended making videos no longer than 6 min. We aimed to keep our video as brief as possible with an absolute cut-off of 4 min, and frequently revisited the issue of length with all stakeholders involved. However, given the complexity of evidence synthesis, and that it was a methodologically-focused topic where it was felt that a public audience would have less existing familiarity than with specific clinical topics (e.g., cancer), and based on input from our study stakeholders, this was deemed to be the shortest possible duration that would allow us to convey the key messages in an understandable manner. This issue was similarly highlighted by Archibald et al. in the development of their ebook KT resource, who noted the issue of length versus information comprehensiveness as one of the fundamental tensions facing arts-based KT work.²⁰ Our future work will aim to evaluate further the effectiveness and impact of the video resource and to explore which characteristics may influence engagement with or understanding of video content. For example, although the YouTube analytics data showed relatively good engagement with the video in comparison to ‘typical’ videos (as flagged in YouTube analytics), further research could explore whether video content could be condensed to reduce video length, and whether this would impact on viewing duration and/or overall understanding of key concepts.

Strengths and limitations

A strength of our approach is the involvement of multiple individuals, including researchers with expertise in KT and evidence synthesis, a PPI advocate with substantial experience in KT for the public, and members of the public with less experience in health research and KT. Although efforts were made to recruit members of the public of varying ages and genders, a limitation of our approach is the use of convenience sampling. This meant that some of the members of the public involved were known to the lead authors (CD and ET), and therefore may have felt some degree of pressure not to be too critical of the resource for fear of offending. Attempts were made to mitigate this by providing clear instructions at the outset before user testing about the importance of honest and critical feedback, with the overall purpose of creating the most user-friendly resource possible.

In addition, owing to the iterative nature of the user-centred design approach, further user testing cycles could have been applied with additional users who may have yielded extra insights. This echoes concerns highlighted by Archibald et al. regarding determining adequate sample sizes for user testing.²² However, the costs (in time and resources) of conducting additional feedback cycles need to be weighed against that effort's likelihood resulting in significant improvements. We deemed that two rounds of testing were sufficient as no new insights were identified during the second round of testing. Archibald et al. similarly concluded that the generation of no new insights towards the end of the testing process meant that sufficient numbers had more than likely been reached.²²

By drawing on user-centred design methods, we were able to provide a structured and transparent approach to something that is not often described or structured, i.e., the development of KT resources. However, our use of researchers to determine the initial key messages of the resource meant that our approach was not fully ‘user-centred’. Given that this was a KT resource explaining a methodological topic (as opposed to a clinical topic where we felt public users would be more likely to have first-hand experience), and given our time restraints, we felt it a necessary pragmatic starting point for those identifying the initial key messages to have a solid understanding of evidence synthesis. However, the use of PPI in methodological research is growing rapidly.^47–49 Therefore, we feel the involvement of members of the public from the outset in identifying key messages regarding methodological topics is an important endeavour that could be further explored in future similar studies.

Finally, a key strength of this project was the quality of the resource produced given the limited resources and time afforded. As identified by Archibald et al. and others, cost can be a significant barrier to the development of KT resources, including digital and video-based formats.^21,50,51 This project was carried out as summer research for a final-year medical student (CD) over 8 weeks, with a budget of €2000. Similarly to Archibald et al.,²¹ we found that careful collaborative brainstorming at the outset (e.g., to identify initial key messages and clarify the purpose of the resource), and use of iterative cycles of feedback throughout the process (e.g., exploring storyboard and script prototypes first, then video prototypes) helped ensure that we did not have to go back and redo or fix elements of the final product which would have added to the time and cost Additionally, as our limited time and resources hindered our ability to outsource expertise (e.g., the hiring of a digital media company as used by Archibald et al.²⁰), it was important to ensure that the research team included as much relevant expertise as possible. For example, our interdisciplinary team included researchers with expertise in evidence synthesis, (ET, DD, SR, PH, AH), knowledge translation for public audiences (DD, SR, PH, AH, ET), qualitative interviewing techniques (ET, PH, AH, SR) and digital media and video development (EP). Crucial to our project's success was the identification of a tech-savvy healthcare student who was able to work competently with video-editing software (CD) and was mentored in evidence synthesis, research and interviewing skills by ET and DD and in video-editing by EP. A substantial amount of time at the start was also spent identifying relevant frameworks and methods to structure our process (e.g., the KT Primer, Morville's honeycomb model, thinkaloud interviews, user-centred design principles). As such we believe our work will provide a valuable template and minimise this time for other researchers wishing to develop KT resources such as this with limited investment.

Conclusion

In an age of increasing exposure to health information, misinformation and disinformation, a good understanding of evidence synthesis is crucial for enabling people to think critically about treatment claims and choices. Drawing on user-centred design methods was extremely beneficial in providing a structured and transparent approach to the development of a video animation KT resource to explain evidence synthesis to the public. The input of stakeholders from multiple backgrounds and iterative revisions based on their feedback was critical. Specifically, involving members of the public played a key role in enhancing the credibility and usefulness of the resource by guiding how the animation looked and sounded, as well as ensuring clarity of its key messages and wording. However, future work could explore applying a user-centred approach from the outset by involving the public to identify key messages in developing KT resources explaining methodological topics. As health information becomes increasingly abundant and available, facilitating the development of a properly informed public is of paramount importance. This study describes the structured and systematic development of a KT resource with limited resources, involving key stakeholders and end-users, and provides transferrable learnings for others wishing to do similar.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076231170696 - Supplemental material for Development of a video-based evidence synthesis knowledge translation resource: Drawing on a user-centred design approach

Supplemental material, sj-docx-1-dhj-10.1177_20552076231170696 for Development of a video-based evidence synthesis knowledge translation resource: Drawing on a user-centred design approach by Cristian Deliv, Declan Devane, El Putnam, Patricia Healy, Amanda Hall, Sarah Rosenbaum and Elaine Toomey in DIGITAL HEALTH

Supplemental Material

sj-docx-2-dhj-10.1177_20552076231170696 - Supplemental material for Development of a video-based evidence synthesis knowledge translation resource: Drawing on a user-centred design approach

Supplemental material, sj-docx-2-dhj-10.1177_20552076231170696 for Development of a video-based evidence synthesis knowledge translation resource: Drawing on a user-centred design approach by Cristian Deliv, Declan Devane, El Putnam, Patricia Healy, Amanda Hall, Sarah Rosenbaum and Elaine Toomey in DIGITAL HEALTH

Supplemental Material

sj-docx-3-dhj-10.1177_20552076231170696 - Supplemental material for Development of a video-based evidence synthesis knowledge translation resource: Drawing on a user-centred design approach

Supplemental material, sj-docx-3-dhj-10.1177_20552076231170696 for Development of a video-based evidence synthesis knowledge translation resource: Drawing on a user-centred design approach by Cristian Deliv, Declan Devane, El Putnam, Patricia Healy, Amanda Hall, Sarah Rosenbaum and Elaine Toomey in DIGITAL HEALTH

Supplemental Material

sj-docx-4-dhj-10.1177_20552076231170696 - Supplemental material for Development of a video-based evidence synthesis knowledge translation resource: Drawing on a user-centred design approach

Supplemental material, sj-docx-4-dhj-10.1177_20552076231170696 for Development of a video-based evidence synthesis knowledge translation resource: Drawing on a user-centred design approach by Cristian Deliv, Declan Devane, El Putnam, Patricia Healy, Amanda Hall, Sarah Rosenbaum and Elaine Toomey in DIGITAL HEALTH

Footnotes

Acknowledgements

The authors would like to thank sincerely our public, PPI and researcher knowledge users who contributed to our resource development. They are (alphabetical order): Allen Conor, Booth Andrew, Carey Ryan, Chapman Sarah, Cunningham Niamh, Deliv Daniel, Deliv Eric, Fahey Sarah, Hall Maura, McCool Grainne, McEntee Joe, Mosenet Ion, Nazarewicz Wiktoria, O’Brien John, O’Brien Mary, Ryan Rebecca, Ryan-Vig Selena, Stewart Derek, Zerkaljis Deniss. We would also like to thank Michelle Ryan (EpikMedia) for her voiceover. Cristian Deliv was supported to work on this project by a Summer Studentship from Evidence Synthesis Ireland.

Contributorship

The CRediT Taxonomy was used to describe contributions as follows. Cris Deliv: Investigation, Resources, Data Curation, Visualisation, Writing - Original Draft, Writing - Review & Editing El Putnam: Software, Methodology, Resources, Writing - Review & Editing. Declan Devane: Conceptualization, Methodology, Resources, Writing - Review & Editing, Funding acquisition. Sarah Rosenbaum: Conceptualization, Methodology, Writing - Review & Editing. Amanda Hall: Methodology, Writing - Review & Editing. Patricia Healy: Conceptualization, Resources, Writing - Review & Editing, Funding acquisition. Elaine Toomey: Conceptualization, Methodology, Investigation, Resources, Writing - Original Draft, Writing - Review & Editing, Supervision, Project administration, Funding acquisition.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

There was no national public ethics committee in existence at the time this study was conducted, therefore guidance was sought from the local Galway University Hospital Research Ethics Committee. Given the aim of the study (to co-produce the KT resource in collaboration with researchers and members of the public) the Chair indicated that this project did not require ethical approval. All members of the public provided written or verbal informed consent prior to their involvement.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article

Guarantor

Elaine Toomey elaine.toomey@universityofgalway.ie (corresponding author)

ORCID iDs

Cristian Deliv

Sarah Rosenbaum

Elaine Toomey

Supplemental material

Supplemental material for this article is available online.

References

Criss

Woo Baidal

Goldman

, et al. The role of health information sources in decision-making among hispanic mothers during their children’s first 1000 days of life. Matern Child Health J 2015; 19: 2536–2543.

Couper

Singer

Levin

, et al. Use of the internet and ratings of information sources for medical decisions: results from the DECISIONS survey. Med Decis Making 2010; 30: 106–114.

Clarke

Moore

Steege

, et al. Health information needs, sources, and barriers of primary care patients to achieve patient-centered care: a literature review. Health Informatics J 2015; 22: 992–1016.

De Jesus

. The impact of mass media health communication on health decision-making and medical advice-seeking behavior of u.s. Hispanic population. Health Commun 2013; 28: 525–529.

Moorhead

Hazlett

Harrison

, et al. A new dimension of health care: systematic review of the uses, benefits, and limitations of social media for health communication. J Med Internet Res 2013; 15: 85.

Arora

Hesse

Rimer

, et al. Frustrated and confused: the American public rates its cancer-related information-seeking experiences. J Gen Intern Med 2008; 23: 223–228.

Johnson

Park

Gross

, et al. Complementary medicine, refusal of conventional cancer therapy, and survival among patients with curable cancers. JAMA Oncol 2018; 4: 1375–1381.

Hall

Walton

. Information overload within the health care system: a literature review. Health Info Libr J 2004; 21: 102–108.

Marquez

Johnson

Jassemi

, et al. Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study. Implement Sci 2018; 13: 84.

10.

Straus

Tetroe

Graham

. Defining knowledge translation. Cmaj 2009; 181: 165–168.

11.

Tworek

Ickert

Bakal

, et al. Examining the impact of knowledge translation interventions on uptake of evidence-based practices by care aides in continuing care. Worldviews Evid Based Nurs 2019; 16: 21–28.

12.

Ospina

Taenzer

Rashiq

, et al. A systematic review of the effectiveness of knowledge translation interventions for chronic noncancer pain management. Pain Res Manag 2013; 18: e129–e141.

13.

Meppelink

van Weert

Haven

, et al. The effectiveness of health animations in audiences with different health literacy levels: an experimental study. J Med Internet Res 2015; 17: 11.

14.

Houts

Doak

, et al. The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence. Patient Educ Couns 2006; 61: 173–190.

15.

Atkinson

Massett

Mylks

, et al. Assessing the impact of user-centered research on a clinical trial eHealth tool via counterbalanced research design. J Am Med Inform Assoc 2011; 18: 24–31.

16.

Witteman

Dansokho

Colquhoun

, et al. User-centered design and the development of patient decision aids: protocol for a systematic review. Syst Rev 2015; 4: 11.

17.

Mairs

McNeil

McLeod

, et al. Online strategies to facilitate health-related knowledge transfer: a systematic search and review. Health Inf Libr J 2013; 30: 261–277.

18.

Sarkies

Bowles

K-A

Skinner

, et al. The effectiveness of research implementation strategies for promoting evidence-informed policy and management decisions in healthcare: a systematic review. Implement Sci 2017; 12: 32.

19.

Cross

Rosenbaum

Grahovac

, et al. A web-based knowledge translation resource for families and service providers (The “F-Words” in childhood disability knowledge hub): developmental and pilot evaluation study. JMIR Rehabil Assist Technol 2018; 5: e10439–e10439.

20.

Archibald

Hartling

Ali

, et al. Developing “My Asthma Diary”: a process exemplar of a patient-driven arts-based knowledge translation tool. BMC Pediatr 2018; 18: 86.

21.

Archibald

Ambagtsheer

Lawless

, et al. Co-Designing evidence-based videos in health care: a case exemplar of developing creative knowledge translation “evidence-experience” resources. Int J Qual Methods 2021; 20: 16094069211019623.

22.

Archibald

Scott

. Learning from usability testing of an arts-based knowledge translation tool for parents of a child with asthma. Nursing Open 2019; 6: 1615–1625.

23.

Price

Albarqouni

Kirkpatrick

, et al. Patient and public involvement in the design of clinical trials: an overview of systematic reviews. J Eval Clin Pract 2018; 24: 240–253.

24.

Montague

. Patient-provider partnerships in healthcare: enhancing knowledge translation and improving outcomes. Healthc Pap 2006; 7: 53–61.

25.

Gray-Burrows

Willis

Foy

, et al. Role of patient and public involvement in implementation research: a consensus study. BMJ Qual Saf 2018; 27: 58.

26.

Nsangi

Semakula

Oxman

, et al. Effects of the informed health choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial. Lancet 2017; 390: 374–388.

27.

Nsangi

Semakula

Glenton

, et al. Informed health choices intervention to teach primary school children in low-income countries to assess claims about treatment effects: process evaluation. BMJ Open 2019; 9: e030787.

28.

Semakula

Nsangi

Oxman

, et al. Development of mass media resources to improve the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about the effects of treatments: a human-centred design approach. Pilot Feasibility Stud 2019; 5: 55.

29.

National Institute for Health Research. Briefing notes for researchers - public involvement in NHS, health and social care research, https://www.nihr.ac.uk/documents/briefing-notes-for-researchers-public-involvement-in-nhs-health-and-social-care-research/27371?pr = 2021.

30.

Charters

. The use of think-aloud methods in qualitative research an introduction to think-aloud methods. Brock Educ J 2003; 12: 68–80.

31.

van Someren

Barnard

Sandberg

JAC

. THE THINK ALOUD METHOD: A practical guide to modelling cognitive processes. In: UoA (ed.) Department of Social Science Informatics. London: Academic Press, 1994, pp.21–27.

32.

Morville

. User Experience Design.

33.

Mansson

Wiklund

Öhberg

, et al. Co-creation with older adults to improve user-experience of a smartphone self-test application to assess balance function. Int J Environ Res Public Health 2020; 17: 2–7. 10.3390/ijerph17113768

34.

Thunberg

Arnell

. Pioneering the use of technologies in qualitative research – A research review of the use of digital interviews. Int J Soc Res Methodol 2022; 25: 757–768.

35.

Public Health Agency of Canada. Knowledge Translation (KT) Planning Primer. Ottawa (ON): Public Health Agency of Canada, 2012.

36.

Deliv

Putnam

Devane

, et al. Evidence Synthesis - What is it and why do we need it?, https://www.youtube.com/watch?v = nZR0xQmZVQg 2019.

37.

Deliv

Putnam

Devane

, et al. Evidence Synthesis, What is it and Why do we Need it?, https://www.youtube.com/watch?v = xGdZQNHC8-k&t = 1 s 2019.

38.

Rodgers

Massac

. Misinformation: a threat to the public's health and the public health system. J Public Health Manag Pract 2020; 26: 294–296.

39.

World Health Organization. Immunizing the public against misinformation 2020.

40.

Chalmers

Oxman

Austvoll-Dahlgren

, et al. Key concepts for informed health choices: a framework for helping people learn how to assess treatment claims and make informed choices. BMJ Evidence-Based Med 2018; 23: 29.

41.

Chalmers

Glasziou

. Avoidable waste in the production and reporting of research evidence. Lancet 2009; 374: 86–89. DOI: 10.1016/S0140-6736(09)60329-9

42.

De Angelis

Davies

King

, et al. Information and communication technologies for the dissemination of clinical practice guidelines to health professionals: a systematic review. JMIR Med Educ 2016; 2: e16–e16.

43.

Curran

Fleet

. A review of evaluation outcomes of web-based continuing medical education. Med Educ 2005; 39: 561–567.

44.

Graham

Logan

Harrison

, et al.

Lost in knowledge translation: time for a map?

J Contin Educ Health Prof 2006; 26: 13–24.

45.

Zaila

Osadchiy

Anderson

, et al. Popularity and worldwide reach of targeted, evidence-based internet streaming video interventions focused on men's health topics. Transl Androl Urol 2020; 9: 1374–1381.

46.

Guo

Kim

Rubin

. How video production affects student engagement: an empirical study of MOOC videos. In: Proceedings of the first ACM conference on learning @ scale conference. Atlanta, Georgia, USA: Association for Computing Machinery, 2014, pp.41–50.

47.

Healy

Galvin

Williamson

, et al. Identifying trial recruitment uncertainties using a James Lind alliance priority setting partnership – the PRioRiTy (prioritising recruitment in randomised trials) study. Trials 2018; 19: 47.

48.

Brunsdon

Biesty

Brocklehurst

, et al. What are the most important unanswered research questions in trial retention? A James Lind alliance priority setting partnership: the PRioRiTy II (prioritising retention in randomised trials) study. Trials 2019; 20: 93.

49.

Beecher

Toomey

Maeso

, et al. Priority III: top 10 rapid review methodology research priorities identified using a James Lind alliance priority setting partnership. J Clin Epidemiol 2022; 151: 151–160.

50.

Glegg

SMN

Levac

. Barriers, facilitators and interventions to support virtual reality implementation in rehabilitation: a scoping review. PM R 2018; 10: 1237–1251.e1231.

51.

Kalbarczyk

Rodriguez

Mahendradhata

, et al. Barriers and facilitators to knowledge translation activities within academic institutions in low- and middle-income countries. Health Policy Plan 2021; 36: 728–739.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

0.02 MB