Sage Journals: Discover world-class research

Abstract

Objective:

To evaluate the quality of information presented on YouTube regarding functional endoscopic sinus surgery (FESS) for patients.

Methods:

YouTube was searched using FESS-specific keywords under the setting of “relevance.” The first 50 videos from each keyword were reviewed and analyzed by 2 independent physician reviewers. Videos not related to FESS and duplicates were excluded. Outcome measures included the modified DISCERN score (range 0-5), the Journal of the American Medical Association (JAMA) benchmark criteria (range: 0-4), a novel scoring checklist for FESS assessing usefulness (range: 0-16), and the Video Power Index (VPI). Intraclass correlation coefficient (ICC) was calculated.

Results:

Of the 200 videos identified, 95 videos were analyzed after exclusions. Videos had an average VPI of 40.8 and SD 133.2. Average scores from the 3 objective checklists among all videos were low: modified DISCERN: 1.91, SD: 1.15; JAMA benchmark: 1.91, SD: 0.76; and FESS score: 3.54, SD: 1.77. The ICC between the 2 independent reviewers was excellent for all 3 checklists. We noted significant positive Pearson correlation between all 3 checklist scores (P < .001). In between-group comparisons of mean scores, there was significantly higher DISCERN and JAMA scores for videos from university/professional organizations, as compared to videos from medical advertising/for-profit companies and independent users. There were no significant differences in FESS scores noted between the 3 groups.

Conclusion:

There were overall low scores across the modified DISCERN, JAMA benchmark criteria, and FESS scoring checklists, reflecting the poor quality of YouTube videos as a source of patient information for FESS.

Keywords

endoscopic sinus surgery FESS patient information YouTube

Background

With the advancement of the Internet and the availability of online resources, patients are increasingly referring to online materials for information regarding disease and surgical procedures. In fact, it is estimated that upward of 70% of adults use the Internet to obtain health and health-care-related information.¹ The 2018 Health Information National Trends Survey in the United States found that over a third of patients watched health-related videos on YouTube.^2,3 YouTube is the most popular online open-access video-sharing platform, with over 2 billion videos viewed daily, coming from more than 30 million unique visitors.^4,5

YouTube videos as a source for patient information and education have been evaluated in the domain of bariatric surgery, vascular surgery, and gynecology.^3,6-8 In the field of otolaryngology–head and neck surgery (OHNS), functional endoscopic sinus surgery (FESS) is one of the most commonly performed operations, utilized for the treatment of sinonasal disease including chronic rhinosinusitis. It is estimated that in Canada, there are 10 000 to 15 000 FESS cases performed each year, with that number estimated to be 250 000 in the United States.^9,10 A study on patients’ knowledge surrounding FESS found that upward of 30% looked to YouTube as a source of information.¹¹

Despite this, there has been no formal evaluation of the accuracy and completeness of information presented within popular YouTube videos for FESS. Given the powerful and popular nature of YouTube as an information source, it is imperative that we understand the accuracy and reliability of these sources. Herein, we aimed to quantitatively evaluate the quality of information presented on YouTube regarding preoperative, procedural, and postoperative aspects of FESS, with the goal to better inform the patients about YouTube’s role as a potential source of information.

Methods

Ethics

The research study received exemption from research ethics board review from the St Michael’s Hospital Research Ethics Board as YouTube videos are publicly available.

Search Strategy

The video search strategy employed herein was adapted from previous studies evaluating YouTube content.^3,6-8,12 An Internet browser with its cache cleared was used to conduct a one-time search on www.youtube.com on January 21, 2020, using FESS-related keywords, including “FESS,” “functional endoscopic sinus surgery,” “endoscopic sinus surgery,” and “sinus surgery.” The default search setting of “relevance” was used to replicate a typical search attempt performed by the public on YouTube. As such, the first 50 results from each keyword search were reviewed and analyzed. Videos that were non-English, not related to FESS, or duplicates were excluded from the study.

Video Assessment

Two independent physician reviewers (V.W. and D.J.L.) assessed each video for eligibility and content. Discrepancies with regard to eligibility were addressed and discussed with the senior author (J.M.L.).

Outcome Measures

The primary outcome measure was the reliability of the videos. This was determined using 3 checklists: (1) modified DISCERN criteria, (2) the Journal of the American Medical Association (JAMA) benchmark criteria score, and (3) a novel FESS scoring system. These checklists are included as part of Tables 1 to 3, respectively.

Table 1.

Modified DISCERN Criteria.^a

1. Are the aims clear and achieved?

2. Are reliable sources of information used? (ie, publication cited, speaker is otolaryngologist)

3. Is the information presented balanced and unbiased?

4. Are additional sources of information listed for patient reference?

5. Are areas of uncertainty mentioned?

^a Adapted from Singh et al¹³ and Radonjic et al.³

Table 2.

Modified Journal of the American Medical Association (JAMA) Benchmark Criteria.^a

Authorship	Authors and contributors, their affiliations, and relevant credentials should be provided
Attribution	References and sources for all content should be listed clearly, and all relevant copyright information should be noted
Disclosure	“Ownership” should be prominently and fully disclosed, as should any sponsorship, advertising, underwriting, commercial funding arrangements or support, or potential conflicts of interest
Currency	Dates when content was posted and updated should be indicated

^a Adapted from Radonjic et al.³

Table 3.

Checklist for the Functional Endoscopic Sinus Surgery (FESS) Score.

Preoperative phase	Procedure	Postoperative phase
1. Normal anatomy and physiology	8. Surgical setup (ie, patient position, equipment, image guidance)	13. Hospital course—outpatient vs inpatient
2. History of the disease (ie, CRS and diagnostic criteria)	9. Endoscopic polypectomy	14. Discussion of recovery time
3. Physical examination—nasal endoscopy	10. Opening sinus ostium (1 or more of maxillary antrostomy, anterior/posterior ethmoidectomy, sphenoid sinusotomy, frontal sinusotomy)	15. Quantification of revision surgery rate
4. Radiographic findings—review of CT scan	11. Extent of surgery—mini-FESS vs full-house FESS	16. Postoperative care (sinus rinse and/or antibiotics and/or oral steroids)
5. Indications for surgery	12. Discussion of specimens sent for pathology and/or microbiology
6. Discussion of appropriate medical therapy or alternatives to surgery
7. Discussion of complications (minimum 4 of bleeding, infection, globe damage, brain damage, CSF leak, and septal perforation)

Abbreviations: CRS, chronic rhinosinusitis; CSF, cerebrospinal fluid; CT, computed tomography.

The modified DISCERN score was used to evaluate clarity of videos, reliability, bias, reference supplementation, and uncertainty of content.¹³ The total score was 5 points, 1 assigned for each criteria, with higher points indicating more reliability of the videos. This is presented in Table 1. The DISCERN criterion is a brief questionnaire that has been used to assess the reliability and quality of written information. The JAMA benchmark criteria score evaluated for authorship, attribution, disclosure, and currency, with a maximum of 4 points, as given in Table 2.¹⁴ Within the literature, both the modified DISCERN and JAMA benchmark criteria have been used in the evaluation of YouTube videos.^3,15,16

As there was no available checklist in evaluating the quality of online information on FESS, a novel FESS scoring checklist was developed specifically for this study. We constructed this checklist for what a reasonable physician would need to counsel patients about the procedure. We utilized information and adapted parts of the novel FESS scoring system from the previous studies to assess YouTube videos on surgical procedures, along with a review of the current literature.^3,12,17,18 Face and content validity was confirmed by 2 expert rhinologists (A.V. and J.M.L.). The novel FESS scoring checklist was marked, with a maximum of 16 points. The FESS scoring system was used to reflect the usefulness of the videos, with ratings of exceptional (13-16), useful (9-12), moderately useful (5-8), and poor (0-4). This is found in Table 3.

Secondary outcome measures included YouTube video metrics on each video, including website link, video title, duration, upload date, number of views, likes, dislikes, number of comments, and references to sources. Based on these metrics, the Video Power Index (VPI), which assesses video popularity, was also calculated: VPI = ([view ratio × like ratio]/100), where view ratio = views per day, and like ratio = (likes × 100)/(likes + dislikes).

Videos were also classified into 4 categories based on their source: university/professional organization, medical advertising/for-profit companies, independent user, and other (ie, news/media, governmental organization).

Statistical Analysis

All statistical analyses were performed using SPSS version 20, with statistical significance set at P < .05. Evaluation of inter-rater agreement was done using intraclass correlation coefficient (ICC). Descriptive statistics (mean, SD, median, frequency, percentage) was used for displaying the data. Categorical variables were reported as frequencies and relative frequencies. Continuous variables were reported as means and SD. Pearson correlation was calculated between checklists and VPI and expressed with 95% CI. Videos were categorized based on the sources of the videos and were also stratified based on usefulness via the FESS scoring checklist. Comparison of means was done using 1-way analysis of variance with Bonferroni correction.

Results

Of the 200 videos initially identified, 88 were duplicates and 17 were excluded based on the predefined exclusion criteria (Figure 1). The remaining 95 videos were available for analysis. The shortest video was 20 seconds, and the longest videos were 1 hour, 18 minutes, and 14 seconds. The videos combined for a total duration of 17 hours and 14 minutes. Video views ranged from 40 to 824 686, with a total of 6 417 472 views. Uploaded dates were from September 28, 2007, to October 14, 2019. Based on the categories, there were 23 (24.2%) videos from university/professional organizations, 13 (13.7%) from medical advertising/for-profit companies, 59 (62.1%) from independent users, and no videos from other sources (ie, news/media, governmental organization). Videos had an average VPI of 40.8, SD: 133.2.

Figure 1.

PRISMA flowchart for YouTube search.

Average scores from the 3 objective checklists among all videos were low: modified DISCERN score (1.91, SD: 1.15), JAMA benchmark score (1.91, SD: 0.76), and FESS score (3.54, SD: 1.77). Based on the predefined FESS score usefulness categories, 71 (74.7%) videos were considered poor, 21 (22.1%) moderately useful, and 3 (3.2%) useful. The ICC between the 2 independent reviewers was excellent for all 3 checklists: DISCERN (0.875, 95% CI, 0.812-0.916), JAMA benchmark criteria (0.847, 95% CI, 0.770-0.98), and FESS score (0.921, 95% CI, 0.881-0.947).

We noted statistically significant correlation between all 3 checklist scores (P < .001): DISCERN and JAMA: r = 0.745, DISCERN and FESS score: r = 0.434, and FESS and JAMA: r = 0.476. The VPI was not significantly correlated (P = 1.000), with DISCERN, JAMA, and FESS scores having Pearson correlations of r = −0.043, r = −0.045, and r = −0.088, respectively.

Comparisons of checklist scores based on the source of video categories are presented as part of Table 4. There were significant differences between groups with DISCERN and JAMA scores (P = .002 and P < .001, respectively). We did not see significant differences in the FESS score between all 3 categories (P = .675). In between-group comparisons, we only noted significantly higher DISCERN score from university/professional organizations, compared to videos from medical advertising/for-profit companies (0.91, 95% CI: 0.25-1.57, P = .003) and independent users (0.54, 95% CI: 0.07-1.01, P = .018). Higher JAMA scores were only noted for university/professional organizations videos, compared to those from medical advertising/for-profit companies (0.56, 95% CI: 0.13-0.98, P = .006) and independent users (0.55, 95% CI: 0.25-0.86, P < .001). There were no statistically significant differences seen in the mean scores from either 3 checklists between videos categorized as being from medical advertising/for-profit companies and independent users.

Table 4.

Between-Group Video Category Comparison of Checklist Scores and VPI.

Video source	University/professional organization	Medical advertise/profit companies	Independent users	P value
Modified DISCERN	2.37, SD: 1.25	1.46, SD: 1.17	1.83, SD: 1.05	.002
JAMA benchmark	2.33, SD: 0.76	1.77, SD: 0.86	1.77, SD: 0.67	<.001
FESS score	3.74, SD: 2.06	3.46, SD: 2.34	3.47, SD: 1.50	.675
VPI	49.0, SD: 66.7	120.7, SD: 334.2	20.0, SD: 32.8	.002

Abbreviations: FESS, functional endoscopic sinus surgery; JAMA, Journal of the American Medical Association; VPI, Video Power Index.

When evaluating VPI scores, there were significant differences noted between groups (P = .002; Table 4). Comparing between groups, there was a significantly higher VPI for videos categorized as being from medical advertising/for-profit companies than independent users (mean difference: 100.7, 95% CI: 33.1-168.3, P = .001). No significant difference in VPI was noted for other comparisons between video sources.

Discussion

This study evaluated the quality and usefulness of information presented within popular YouTube videos on FESS for patients, with the use of independent physician evaluators and objective measurement tools. We found that of the 95 videos included for analysis, there was excellent inter-rater reliability across all 3 of the checklists. There were overall low scores across the modified DISCERN, JAMA benchmark criteria, and FESS scoring checklists, reflecting poor quality and low usefulness of YouTube as a source of patient information for FESS with currently available videos.

Within the surgical literature, studies evaluating YouTube as a source of patient information have in general found that the information presented was often varied and unreliable.^3,6-8 Radonjic et al reported on the quality and usefulness of YouTube videos on patients with abdominal aortic aneurysm (AAA) and found low DISCERN (2.37, SD: 0.97), JAMA (1.74, SD: 0.84), and specific AAA (6.63, SD: 3.23) scores.³ These findings are similar to the results from our study, and both point to the low quality of YouTube videos on surgical procedures/diseases as sources of useful information for patients. Specifically, within the field of OHNS, Haymes and Harries evaluated YouTube videos on the management of epistaxis and found that inappropriate and dangerous advice was offered within videos. The authors concluded that YouTube videos were not recommended as an information source for patients in managing epistaxis due to the varying degree of quality.¹⁶ Another study examining YouTube videos as a source of patient information for rhinoplasty found that videos were of average quality and reliability.¹⁵ Taken together, there is evidence suggesting overall low quality of YouTube videos as an information source for patients regarding surgical procedures and diseases.

We did not find any association between VPI, which is a reflection of the popularity of YouTube videos, and the 3 objective checklist scores. This would suggest that higher ratio of videos and activity/engagement (ie, likes, comments) on a video does not necessarily correlate with its objective quality or usefulness. This is important as patients may look toward the popularity of a video as pertaining to its accuracy, which is not the case based on findings in our study. We did note that VPI was higher from medical advertising/for-profit companies video sources, which may introduce certain commercial biases into the information presented. Additionally, we noted that videos from a university or professional organization had significantly higher scores on both DISCERN and JAMA criteria as compared to other video sources. This may reflect that the information used by universities or professional organizations were from more reputable sources and that information was presented in a more unbiased manner. However, we noted low FESS scores across video sources, which were not significantly different. This may imply that video content may not provide adequate amount of information to patients, regardless of the source. This finding highlights the need for otolaryngologists to provide credible high-quality sources of information regarding FESS to patients in the era of information explosion.

In evaluating YouTube videos, one of the themes that arose, but was not explicitly captured within this study, was on FESS educational videos geared toward medical trainees. Rapp et al identified through a cross-sectional study at a tertiary institution that YouTube was the most frequently used educational resource for surgical preparation and was used by 95% of medical students and residents.¹⁹ Another study found that online videos were frequently used by medical trainees to prepare for surgical observerships.²⁰ Specific evaluation of YouTube videos as it pertained to laparoscopic surgery has revealed that videos were useful and appropriate educational tools.²¹ However, other studies within the general surgery literature have shown low quality of the videos for surgical trainees, with poor or suboptimal techniques demonstrated.⁸ In our study, we are unable to determine whether the videos can be used to teach FESS as this was not the aim of the study. With this uncertainty, studies in the future can aim to evaluate the accuracy and usefulness of YouTube videos on FESS in surgical training for medical trainees.

This study had potential limitations. As there was a lack of validated objective measurement tools for assessing FESS, a scoring checklist was developed for the study. This was our group’s attempt at objectively evaluating the quality of YouTube videos as it pertained to information surgeons would like to have presented to patients undergoing FESS. The checklist was developed with guidance from 2 staff rhinologists, for face and content validity, as it pertained to patient counseling. However, many of the videos on YouTube were made with regard to surgical education or promotional purposes and not purely from the perspective of patients. Moreover, although the checklist may reflect information that physicians would like to tell patients, it does not necessarily reflect the information that patients would like from YouTube videos. Noting that most patients supplement the discussions they had with their surgeons/physicians with additional sources of information such as YouTube videos, one way of identifying the subjective usefulness of the information from the patient’s perspective would be a survey of the patients who watch these videos as a future study, identifying gaps in knowledge from videos/information sources. Moreover, our cross-sectional study only evaluated videos on YouTube at one point in time. Given the enormous amount of content that is uploaded daily onto the website, there may exist more updated and useful videos in the near future. Moreover, we did not include search terms related to “balloon sinuplasty/dilation” as the procedure is currently not publicly funded within Canada. Additionally, we did not evaluate other websites aside from YouTube, as YouTube is considered the second most visited website in the world, after Google, containing 60% of all videos stored on the Internet.^22,23 The majority of Google-based searches on FESS also linked to YouTube videos. Many academic institutions, including our own at the University of Toronto, have specific websites dedicated to providing patient information, often in video and interactive formats. These resources may be evaluated in the future as to determine its utility in providing patients with greater knowledge and understanding.

Conclusion

Although YouTube videos are frequently used as a source of information by patients, the overall quality and usefulness is poor as shown in the modified DISCERN, JAMA benchmark criteria, and FESS scoring checklists. Given YouTube is a frequently used resource for information by patients, otolaryngologists should aim to create and provide patients with more accurate, reliable, and useful sources of information.

Footnotes

Authors’ Note

Vincent Wu and Daniel Lee contributed to study design, data acquisition, data analysis, manuscript preparation, and revision. Allan Vescan and John Lee contributed to study design, data analysis, manuscript preparation, and revision.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Vincent Wu

References

Fox

Purcell

. Chronic disease and the internet. Pew Research Center. Published 2010. Accessed January 17, 2020. www.pewinternet.org/2010/03/24/chronic-disease-and-the-internet/

Langford

Loeb

. Perceived patient–provider communication quality and sociodemographic factors associated with watching health-related videos on YouTube: a cross-sectional analysis. J Med Internet Res. 2019;21(5):e13512.

Radonjic

Hing

Harlock

Naji

. YouTube as a source of patient information for abdominal aortic aneurysms. J vascular Surg. 2020;71(2):637–644.

Madathil

Rivera-Rodriguez

Greenstein

Gramopadhye

. Healthcare information on YouTube: a systematic review. Health Informat J. 2015;21(3):173–194.

Gokcen

Gumussuyu

. A quality analysis of disc herniation videos on YouTube. World Neurosurg. 2019;19:S1878-S8750(19).

Adhikari

Sharma

Arjyal

Uprety

. YouTube as a source of information on cervical cancer. North Am J Med Sci. 2016;8(4):183–186.

Erdem

Sisik

. The reliability of bariatric surgery videos in YouTube platform. Obes Surg. 2018;28(3):712–716.

Rodriguez

Young

Jackson

Oelschlager

Wright

. Viewer discretion advised: is YouTube a friend or foe in surgical education? Surg Endosc. 2018;32(4):1724–1728.

Rudmik

Bird

Dean

, et al. Geographic variation of endoscopic sinus surgery in Canada: an Alberta-based small area variation analysis. Otolaryngol Head Neck Surg. 2015;153(5):865–874.

10.

Bhattacharyya

. Ambulatory sinus and nasal surgery in the United States: demographics and perioperative outcomes. Laryngoscope. 2010;120(3):635–638.

11.

Neubauer

Tabaee

Schwam

Francis

Manes

. Patient knowledge and expectations in endoscopic sinus surgery. Int forum Allergy Rhinol. 2016;6(9):921–925.

12.

MacLeod

Hoppe

Simunovic

, et al. YouTube as an information source for femoroacetabular impingement: a systematic review of video content. Arthroscopy. 2015;31(1):136–142.

13.

Singh

. YouTube for information on rheumatoid arthritis—a wakeup call? J Rheumatol. 2012;39(5):899–903.

14.

Silberg

Lundberg

Musacchio

. Assessing, controlling, and assuring the quality of medical information on the Internet: let the reader and viewer beware. JAMA. 1997;277(15):1244–1245.

15.

Oremule

Patel

Orekoya

Advani

Bondin

. Quality and reliability of YouTube Videos as a source of patient information on rhinoplasty. JAMA Otolaryngol Head Neck Surg. 2019;145(3):282–283.

16.

Haymes

Harries

. ‘How to stop a nosebleed’: an assessment of the quality of epistaxis treatment advice on YouTube. J Laryngol Otol. 2016;130(8):749–754.

17.

Bowden

Church

Chiu

Vaughan

. Informed consent in functional endoscopic sinus surgery: the patient’s perspective. Otolaryngol Head Neck Surg. 2004;131(1):126–132.

18.

Wolf

Chiu

Palmer

, et al. Informed consent in endoscopic sinus surgery: the patient perspective. Laryngoscope. 2005;115(3):492–494.

19.

Rapp

Healy

Charlton

, et al. YouTube is the most frequently used educational video source for surgical preparation. J Surg Educ. 2016;73(6):1072–1076.

20.

Brickman

Zevin

. Preparing for surgery observerships during medical school: a cross-sectional study. J Surg Educ. 2020;77(5):1097–1105.

21.

de’Angelis

Gavriilidis

Martínez-Pérez

, et al. Educational value of surgical videos on YouTube: quality assessment of laparoscopic appendectomy videos by senior surgeons vs. novice trainees. World J Emerg Surg. 2019;14(1):22.

22.

Bae

Baxter

. YouTube videos in the English language as a patient education resource for cataract surgery. Int Ophthalmol. 2018;38(5):1941–1945.

23.

Sahin

Schwenter

Sebajang

. YouTube videos as a source of information on colorectal cancer: what do our patients learn? J Cancer Educ. 2019;34(6):1160–1166.

Evaluating YouTube as a Source of Patient Information for Functional Endoscopic Sinus Surgery