Readability of the American,Canadian,and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials

Abstract

Objective

Patient education materials across 3 national English otolaryngology–head and neck surgery (OHNS) societies: the American Academy of Otolaryngology–Head and Neck Surgery (AAO-HNS), the Canadian Society of Otolaryngology–Head and Neck Surgery (CSOHNS), and Ear, Nose, and Throat United Kingdom (ENT UK) were examined to determine whether they are written at a level suitable for patient comprehension.

Study Design

Cross-sectional study.

Setting

Online patient materials presented through OHNS national societies.

Methods

Readability was calculated using the Flesch-Kincaid Grade Level, Flesch-Kincaid Reading Ease Score, and Simple Measure of Gobbledygook Index. All public patient education materials available through the CSOHNS, AAO-HNS, and ENT UK websites were assessed. Patient education materials were grouped into categories by subspecialty.

Results

In total, 128 patient materials from the 3 societies were included in the study. All 3 societies required a minimum grade 9 reading comprehension level to understand their online materials. According to Flesch-Kincaid Grade Level, the CSOHNS required a significantly higher reading grade level to comprehend the materials presented when compared to AAO-HNS (11.3 vs 9.9; 95% CI, 0.5-2.4; P < .01) and ENT UK (11.3 vs 9.4; 95% CI, 0.9-2.9; P < .01). Patient education materials related to rhinology were the least readable among all 3 societies.

Conclusion

This study suggests that the reading level of the current patient materials presented through 3 national OHNS societies are written at a level that exceeds current recommendations. Promisingly, it highlights an improvement for the readability of patient materials presented through the AAO-HNS.

Keywords

readability patient education health literacy otolaryngology

One in 5 patients obtain medical information about their disease or surgical procedure from the Internet prior to their appointment with an otolaryngologist–head and neck surgeon.^1-3 It is estimated that approximately 54% of patients with head and neck cancer use the Internet to find information about their treatments.^4-6 A single-institutional study exploring Internet use among otolaryngology–head and neck surgery (OHNS) patients concluded that approximately 61% of patients conducted an Internet search with regards to their condition before seeing a general otolaryngologist, 60% before a rhinologist, 44% before seeing a laryngologist, and 52% before seeing a neurotologist.² With patients becoming increasingly dependent on web-based health information,^7,8 having online educational materials that can be understood by the patient population is of the utmost importance.

Health literacy, the degree to which an individual has the ability to understand and apply basic health information, has been established as one of the strongest predictors of overall health.^9-12 Low health literacy is associated with poor health outcomes, including increased mortality rates and hospitalizations.^13-15 This is of great concern since as high as 49% of Canadians,^16,17 36% of Americans,^18,19 and 59% of Europeans^20,21 have low literacy and limited ability to understand the health information presented to them. Within OHNS, many studies have demonstrated that available online information has been deficient in readability and is not written at a level easily understood by the public.^22-24 However, some of these materials were not necessarily intended for patient education specifically, and thus commonly used readability targets may have been unnecessary.

The American Medical Association (AMA) has recommended that medical information written for patients should aim for a reading level less than or equal to sixth grade as the average American reads at approximately an eighth-grade reading level.²⁵ Similarly, in Canada and the United Kingdom, it is estimated that 48% of Canadians have literacy levels that fall below a high school level, and 16% of adults in Northern Ireland and England read below the lowest level of reading proficiency, which equates to the estimated reading level of an individual 5 to 7 years of age.^26,27 Previous studies have shown that OHNS web-based health information does not fulfill AMA recommendations.^28-30 The Canadian Society of Otolaryngology–Head and Neck Surgery (CSOHNS), the American Academy of Otolaryngology–Head and Neck Surgery (AAO-HNS), and Ear, Nose, and Throat United Kingdom (ENT UK) have materials specifically targeted for patient education. However, the readability of CSOHNS and ENT UK materials has yet to be established, and the readability of the AAO-HNS was last assessed in 2016.³¹ Herein, we aimed to analyze the readability of the patient resources made available through 3 of the largest English-speaking national societies, as physicians often turn their patients to these resources for further information.

Methods

All public patient education materials available through the CSOHNS, AAO-HNS, and ENT UK websites were assessed. Patient education materials were grouped into categories by subspecialty: otology, rhinology, head and neck oncologic surgery, and other. The other category included topics related to general OHNS, facial plastics, laryngology, and pediatric OHNS given the low number of patient education material in each individual category (see Supplementary Material 1 in the online version of the article). This study was exempted from the University of Toronto review board review because the educational material on the website is publicly accessible and does not involve patient records.

Readability Evaluation

The text from each webpage was pasted to Microsoft Word (Microsoft Corp) and removed of any formatting elements that might interfere with readability assessment (headings, symbols, author information, references, etc). The plain text from each webpage was then assessed using an online readability calculator (https://readable.com/), as demonstrated in other readability studies.^29,31,32 This resource evaluates the text’s readability based on criteria such as the average number of syllables and sentence length and yields the following readability scores: Flesch-Kincaid Grade Level (FKGL), the Flesch-Kincaid Reading Ease (FRE) score, and Simple Measure of Gobbledygook Index (SMOG) reading grade. The formulas can be found in Supplementary Material 2 (in the online version of the article). The FKGL is the most validated readability formula used for analyzing medical texts, and both the FRE and FKGL have been widely used to assess readability in the OHNS literature.^22,23,33-36 FKGL approximates the reading grade level necessary to comprehend the text (eg, a score of 6 corresponds to a sixth-grade reading level). The FRE determines the readability using a score from 0 to 100, with higher scores indicating a higher level of readability, as shown in Table 1 . The SMOG reading grade is a measure of readability that estimates the years of education needed to understand a piece of writing.³⁷ Materials with low FKGL, high FRE, and low SMOG scores are considered more readable.

Table 1.

Flesch-Kincaid Reading Ease (FRE) Readability Score and Interpretation.

FRE score	Interpretation
90-100	Very easy
80-89	Easy
70-79	Fairly easy
60-69	Standard
50-59	Fairly difficult
30-49	Difficult
<30	Very confusing

Statistical Analyses

Categorical variables were reported using frequencies and proportions. Continuous variables were presented as mean and SD. Normalcy was assessed through the Shapiro-Wilk test and visual inspection of histogram and quantile-quantile plots. Comparison of continuous variables was performed using 1-way analysis of variance (ANOVA) with post hoc Tukey test. Comparisons of readability scores were made between CSOHNS, AAO-HNS, and ENT UK, along with different OHNS subspecialties (otology, rhinology, head and neck oncology, and other). A threshold of P < .05 was considered statistically significant. The effect sizes were reported using standardized mean differences (Cohen’s d).³⁸ The absolute value of the standardized mean differences was categorized as small (0.2-0.5), medium (0.5-0.8), or large (≥0.8).³⁹ Statistical analysis was performed using SPSS (v.26.0; SPSS, Inc).

Results

In total, 128 online patient materials from 3 different national societies were included in the study: 63 from the AAO-HNS, 20 from the CSOHNS, and 45 from ENT UK.

The mean FRE, FKGL, and SMOG scores across the 3 societies are outlined in Table 2 . The FRE scores corresponded to a reading interpretation of “fairly difficult” for ENT UK and AAO-HNS, whereas the FRE score for CSOHNS demonstrated a “difficult” interpretation. Furthermore, ENT UK had a significantly higher FRE score, compared to AAO-HNS (58.1 vs 50.6; d = 0.8, P < .01) and CSOHNS (58.1 vs 45.2; d = 1.3, P < .01). With regards to the SMOG analysis, CSOHNS was the least readable with a mean SMOG score corresponding to an undergraduate reading comprehension level (mean [SD], 13.5 [1.5]). ENT UK was the most readable according to the SMOG score and yet required a high school junior literacy on average (mean [SD], 11.8 [1.5]).

Table 2.

FRE, FKGL, and SMOG scores for American Academy of Otolaryngology–Head and Neck Surgery, Canadian Society of Otolaryngology–Head and Neck Surgery, and Ear, Nose, and Throat United Kingdom.

Society	FRE, mean (SD)	FKGL, mean (SD)	SMOG, mean (SD)
American Academy of Otolaryngology–Head and Neck Surgery (n = 63)	50.6 (8.0)	9.9 (1.3)	12.2 (1.1)
Canadian Society of Otolaryngology–Head and Neck Surgery (n = 20)	45.2 (10.2)	11.3 (1.8)	13.5 (1.5)
Ear, Nose, and Throat United Kingdom (n = 45)	58.1 (9.9)	9.4 (1.8)	11.8 (1.5)

Abbreviations: FKGL, Flesch-Kincaid Grade Level; FRE, Flesch-Kincaid Reading Ease; SMOG, Simple Measure of Gobbledygook Index.

The FKGL grade-reading levels are outlined in Figure 1 . According to FKGL scores, the CSOHNS required a significantly higher reading grade level to comprehend the materials presented when compared to AAO-HNS (11.3 vs 9.9; d = 1.00, P < .01) and ENT UK (11.3 vs 9.4; d = 1.3, P < .01). All 3 societies required a minimum grade 9 reading comprehension level to understand their online materials, as evidenced by their FKGL scores.

Figure 1.

Mean FKGL scores for American Academy of Otolaryngology–Head and Neck Surgery (AAO-HNS), Canadian Society of Otolaryngology–Head and Neck Surgery (CSOHNS), and Ear, Nose, and Throat United Kingdom (ENT UK). The dotted line signifies the recommended sixth-grade reading level.

The mean FRE, FKGL, and SMOG scores for the OHNS subspecialties presented through the 3 societies is shown in Table 3 . The patient materials on topics related to rhinology were the least readable between all 3 societies, as demonstrated in Figure 2 . The rhinology patient materials from the CSOHNS had higher FKGL scores and lower FRE scores when compared with all other subspecialties and societies. The materials that were the most readable from the CSOHNS and AAO-HNS were in head and neck oncology with mean (SD) FRE scores of 57.5 (10.4) and 52.6 (7.2), respectively. Similarly, across the 3 societies (AAO-HNS, CSOHNS, and ENTUK, respectively), patient materials for head and neck oncology were the most readable according to SMOG analysis—mean (SD) of 11.8 (0.9), 12.1 (1.3), and 11.6 (1.1). The least readable subspecialty material corresponded to the rhinology materials presented through CSOHNS with a mean (SD) SMOG score of 14.3 (0.9).

Table 3.

FRE, FKGL, and SMOG Scores per Subspecialty.

Society	FRE, mean (SD)	FKGL, mean (SD)	SMOG, mean (SD)
American Academy of Otolaryngology–Head and Neck Surgery
Otology (n = 16)	50.1 (8.1)	10.1 (1.2)	12.3 (1.1)
Rhinology (n = 10)	43.9 (6.5)	11.0 (0.9)	13.2 (0.8)
Head and neck oncology (n = 15)	52.6 (7.2)	9.4 (1.1)	11.8 (0.9)
Other (n = 22)	52.5 (7.9)	9.6 (1.3)	12.0 (1.1)
Canadian Society of Otolaryngology–Head and Neck Surgery
Otology (n = 3)	41.8 (7.0)	11.7 (1.3)	13.4 (0.9)
Rhinology (n = 4)	41.5 (11.8)	12.4 (2.2)	14.3 (1.8)
Head and neck oncology (n = 3)	57.5 (10.4)	9.2 (1.4)	12.1 (1.3)
Other (n = 10)	44.0 (8.8)	11.4 (1.6)	13.7 (1.4)
Ear, Nose, and Throat United Kingdom
Otology (n = 16)	55.2 (9.6)	10.0 (2.0)	12.3 (1.6)
Rhinology (n = 11)	55.2 (10.1)	10.0 (1.8)	12.0 (1.6)
Head and neck oncology (n = 9)	59.6 (7.5)	8.9 (1.3)	11.6 (1.1)
Other (n = 9)	65.1 (9.7)	8.1 (1.4)	10.9 (1.2)

Abbreviations: FKGL, Flesch-Kincaid Grade Level; FRE, Flesch-Kincaid Reading Ease; SMOG, Simple Measure of Gobbledygook Index.

Figure 2.

Mean Flesch-Kincaid Grade Level (FKGL) score per subspecialty across the societies. The solid line signifies the recommended sixth-grade reading level.

Discussion

In this study, it was demonstrated that the reading level of the current online patient materials presented through the 3 national OHNS societies was written at a level that exceeded current readability recommendations. The patient materials on topics related to rhinology were the least readable between all 3 societies.

Patient education is an integral component of patient-centered care and informed consent. Studies have shown that the use of online health information by patients can have a significant impact on the health care decision-making process.^5,13,40,41 A study that set out to estimate the prevalence of poor reading skills in the OHNS outpatient population in the United Kingdom found that 28% of their patients had poor reading skills and were unable to comprehend medical leaflets.⁴² Moreover, studies on the utilization of Internet videos and websites for OHNS procedures have concluded they are of poor quality and difficult for patients to comprehend, making national society websites the ideal source for trusted patient education material.^43-46 This requires OHNS societies to provide accurate information, written at a comprehensible level, as these are viewed as trusted sources of information for patient education.²⁹

Our study concluded that neither AAO-HNS, CSOHNS, nor ENT UK had online patient materials written at a sixth-grade reading level as recommended by the AMA. Our current study has demonstrated that for each of these societies, across every subspecialty, the required reading grade level required is at a minimum of grade 11. Previous literature examining the materials presented through the AAO-HNS concluded that the online materials were written at a FKGL of 10.5 to 11.4.³¹ This ignited change as it is well established in the literature that patient comprehension of medical information is integral to their involvement in treatment decisions and improves patient outcomes.^47,48 Our study demonstrated improvement in the FKGL reading levels of the material of the AAO-HNS, as the FKGL reading level was 9.8—this is encouraging as it demonstrates a decrease in reading difficulty of 1 to 2 academic grades. Furthermore, the online materials presented through CSOHNS and ENT UK have not been previously analyzed for their degree of readability. This study found that both patient materials from ENT UK and CSOHNS were found to have significantly higher FRE, FKGL, and SMOG scores, respectively, when compared to AAO-HNS. These findings highlight that there still remains room for improvement to make educational material on national society websites more comprehensible for patients.

With regards to our analysis of OHNS subspecialties, rhinology was unanimously the least readable topic across the 3 societies with reading grade levels ranging from the 10th grade to the 12th grade. These findings are in keeping with previous studies analyzing the readability of patient materials presented through the American Rhinology Society.⁴⁹ One possible explanation for this is the inherent complexity embedded in rhinology, as concepts surrounding the location and conceptualization of the sinuses may be difficult for the average reader. Kasabwala et al²⁹ analyzed patient materials from the AAO-HNS in 2012 and found that materials related to conditions of the nose and mouth had the highest percentage of complex words when compared to other categories. Wong and Levi³¹ in 2016 displayed methods of mitigating the medical jargon intrinsic to rhinology, including explaining the location and purposes of the sinuses before describing the symptoms of sinusitis.

It is estimated that 54% of patients with head and neck cancer use the Internet as a source of health information.⁴ This study also found that patient materials related to head and neck oncology from the AAO-HNS and CSOHNS were the most readable at approximately a ninth-grade reading level. This demonstrates an improvement from the study done by Kasabwala et al²⁹ in 2012 as patient materials related to cancer from the AAO-HNS had an average FKGL score of 11.5. Although this is reassuring, studies have demonstrated that 12% to 47% of head and neck cancer survivors have inadequate health literacy^41,48,50; as such, there remains room for improvement as many of these resources are written above the recommended sixth-grade reading level. A study analyzing the readability of online medical information looked at the online patient materials presented through the national societies for the numerous surgical subspecialties, and they compared the readability results to those presented through the AAO-HNS. They concluded that, similar to the AAO-HNS, the readability scores for the patient education materials from the various other surgical subspecialty websites were above the recommended sixth-grade level.³² Many patients access online patient education materials preconsultation without having prior discussions with health care professionals to provide a foundational understanding of their condition⁵¹; thus, patients should be encouraged and be given opportunities to bring in the materials that they have read independently for the physician to address any questions that have arisen from these resources.

As evidenced by the reading difficulty of the analyzed texts, writing production for conditions within OHNS is a difficult task. The results of this study highlight the need for improving the readability and the overall comprehension of patient education materials available on the websites of OHNS national societies. First, individuals with expertise in health literacy and patient education, such as specialized librarians, could be incorporated into the writing process to ensure readability.^52,53 Patient engagement in the development of this material is critical, but many of the patients chosen for this task may have a higher level of education and may not reflect the target population.⁵⁴ As such, patients with a wide range of educational backgrounds should be engaged in the production process. In addition, materials should be put through a readability software prior to publication to objectively evaluate their reading difficulty. Some other suggestions for improving the readability of patient education materials include limiting the number of words in sentences to a maximum of 8 to 10 words,⁵⁵ using 1- to 2-syllable words,⁵⁶ and replacing medical jargon with simple terms where possible.^57,58 Supplemental Material 3 (in the online version of the article) provides examples of these additional suggestions implemented to further improve the readability of current text found on the 3 OHNS societies analyzed. In addition, some ways authors can improve readability that are not necessarily captured with readability scores include (1) writing in an active voice^59,60; (2) the strategic use of formatting, including highlighting or bolding important points⁶¹; and (3) incorporating mixed media such as videos and images to improve the understanding of medical concepts.⁶² These findings further necessitate the need to employ strategies that ensure that the patient information available online is easily understood by the patient population at large.

There are a few limitations to our study. It is difficult to determine whether the patient education materials included in this study were intended for patients to read in the absence of discussions with their health care provider. FRE and FKGL are readability formulas that calculate the number of syllables and words in a sentence, and these may not adequately account for nuanced complexity of the text, which could potentially bias our results toward a conservative or lower readability score. Furthermore, readability is only 1 aspect of the appropriateness of written information. Availability, format, style, and content are all important factors that, if not addressed, can act as barriers to patient access. In addition, these assessment tools do not take into account videos, diagrams, and other presentation elements that contribute to the quality and understandability of the online patient materials.

Conclusion

The findings of this study suggest that the reading level of the current online patient materials presented through 3 national OHNS societies (AAO-HNS, CSOHNS, and ENT UK) does not meet readability standards. Encouragingly, it highlights an improvement for the readability of patient materials presented through the AAO-HNS. Societies developing patient material should ensure that a professional writer and editor are included as part of the team to ensure readability as a key requirement. Ensuring resources are easily comprehensible has the potential to improve shared decision-making, patient satisfaction, and postoperative outcomes, all of which are important facets of patient-centered health care delivery. Thus, we hope the findings of this current study ignite revisions to the current patient materials presented through the AAO-HNS, CSOHNS, and ENT UK.

Supplemental Material

sj-docx-1-oto-10.1177_01945998211033254 – Supplemental material for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials

Supplemental material, sj-docx-1-oto-10.1177_01945998211033254 for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials by Joo Hyun Kim, Elysia Grose, Justine Philteos, David Forner, Christopher W. Noel, Vincent Wu and Antoine Eskander in Otolaryngology–Head and Neck Surgery

Supplemental Material

sj-docx-2-oto-10.1177_01945998211033254 – Supplemental material for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials

Supplemental material, sj-docx-2-oto-10.1177_01945998211033254 for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials by Joo Hyun Kim, Elysia Grose, Justine Philteos, David Forner, Christopher W. Noel, Vincent Wu and Antoine Eskander in Otolaryngology–Head and Neck Surgery

Supplemental Material

sj-docx-3-oto-10.1177_01945998211033254 – Supplemental material for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials

Supplemental material, sj-docx-3-oto-10.1177_01945998211033254 for Readability of the American, Canadian, and British Otolaryngology–Head and Neck Surgery Societies’ Patient Materials by Joo Hyun Kim, Elysia Grose, Justine Philteos, David Forner, Christopher W. Noel, Vincent Wu and Antoine Eskander in Otolaryngology–Head and Neck Surgery

Footnotes

Author Contributions

Joo Hyun Kim, carried out the study, acquired the data set, drafted the manuscript, and read and approved the final manuscript and agree to be accountable for the author’s own contributions; Elysia Grose, carried out the study, analyzed the data set, drafted the manuscript, and read and approved the final manuscript and agree to be accountable for the author’s own contributions; Justine Philteos, participated in the design of the study, performed data analysis, drafted the manuscript, read and approved the final manuscript and agree to be accountable for the author’s own contributions; David Forner, contributed to study design, revised the manuscript, assisted with statistics, and read and approved the final manuscript and agree to be accountable for the author’s own contributions; Christopher W. Noel, contributed to study design, revised the manuscript, assisted with statistics, and read and approved the final manuscript and agree to be accountable for the author’s own contributions; Vincent Wu, contributed to study design, formulated study design, revised the manuscript, and read and approved the final manuscript and agree to be accountable for the author’s own contributions; Antoine Eskander, revised the manuscript and read and approved the final manuscript and agree to be accountable for the author’s own contributions.

Disclosures

Competing interests: None.

Sponsorships: None.

Funding source: None.

Supplemental Material

Additional supporting information is available in the online version of the article.

References

Tassone

Georgalas

Patel

Appleby

Kotecha

Do otolaryngology out-patients use the internet prior to attending their appointment?

J Laryngol Otol. 2004;118(1):34-38.

Pagedar

Schularick

Lee

Karnell

LH.

Health-related Internet use among otolaryngology patients. Ann Otol Rhinol Laryngol. 2018;127(8):551-557.

Ren

Deng

Hong

Zhang

Health information in the digital age: an empirical study of the perceived benefits and costs of seeking and using health information from online sources. Health Info Libr J. 2019;36(2):153-167.

Rogers

Rozek

Aleyaasin

Promod

Lowe

Internet use among head and neck cancer survivors in the North West of England. Br J Oral Maxillofac Surg. 2012;50(3):208-214.

Ziebland

Chapple

Dumelow

Evans

Prinjha

Rozmovits

How the Internet affects patients’ experience of cancer: a qualitative study. BMJ. 2004;328(7439):564.

Ni Riordain

McCreary

. Head and neck cancer information on the Internet: type, accuracy and content. Oral Oncol. 2009;45(8):675-677.

Cline

Haynes

KM.

Consumer health information seeking on the Internet: the state of the art. Health Educ Res. 2001;16(6):671-692.

Diaz

Griffith

Reinert

Friedmann

Moulton

AW.

Patients’ use of the Internet for medical information. J Gen Intern Med. 2002;17(3):180-185.

Megwalu

UC.

Health literacy in patients with head and neck cancer: an understudied issue. JAMA Otolaryngol Head Neck Surg. 2017;143(7):645-646.

10.

Baur

Harris

Squire

The U.S. national action plan to improve health literacy: a model for positive organizational change. Stud Health Technol Inform. 2017;240:186-202.

11.

Bujnowska-Fedak

Wegierek

The impact of online health information on patient health behaviours and making decisions concerning health. Int J Environ Res Public Health. 2020;17(3):880.

12.

Bussey

Sillence

The role of internet resources in health decision-making: a qualitative study. Digit Health. 2019;5:2055207619888073.

13.

Wallace

Rogers

Weiss

BD.

Relationship between health literacy and health-related quality of life among Tennesseans. Tenn Med. 2008;101(5):35-39.

14.

Safeer

Keenan

Health literacy: the gap between physicians and patients. Am Fam Physician. 2005;72(3):463-468.

15.

Bostock

Steptoe

Association between low functional health literacy and mortality in older adults: longitudinal cohort study. BMJ. 2012;344:e1602.

16.

Rootman

Ronson

Literacy and health research in Canada: where have we been and where should we go?

Can J Public Health. 2005;96(suppl 2):S62-S77.

17.

O’Donnell

Health literacy in Canada; a primer for students: Laurie Hoffman-Goetz, Lorie Donelle, and Rukhsana Ahmed : Canadian Scholars Press, Toronto, Ontario, Canada, 2014. J Cancer Educ. 2017;32(2):211-212.

18.

Gazmararian

Curran

Parker

Bernhardt

DeBuono

BA.

Public health literacy in America: an ethical imperative. Am J Prev Med. 2005;28(3):317-322.

19.

Cutilli

Bennett

IM.

Understanding the health literacy of America: results of the National Assessment of Adult Literacy. Orthop Nurs. 2009;28(1):27-32.

20.

Sorensen

Brand

Health literacy—a strategic asset for corporate social responsibility in Europe. J Health Commun. 2011;16(suppl 3):322-327.

21.

van der Heide

Uiters

Sorensen

, et al. Health literacy in Europe: the development and validation of health literacy prediction models. Eur J Public Health. 2016;26(6):906-911.

22.

Chi

Jabbour

Aaronson

NL.

Quality and readability of websites for patient information on tonsillectomy and sleep apnea. Int J Pediatr Otorhinolaryngol. 2017;98:1-3.

23.

Quality and readability of online information on in-office vocal fold injections. Ann Otol Rhinol Laryngol. 2020;129(3):294-300.

24.

Seymour

Lakhani

Hartley

Cochrane

Jephson

Cochlear implantation: an assessment of quality and readability of web-based information aimed at patients. Cochlear Implants Int. 2015;16(6):321-325.

25.

Weiss

BD.

Health Literacy and Patient Safety: Help Patients Understand. American Medical Association; 2007.

26.

Harding

C RE

Williams

Peters

The 2011 Skills for Life Survey: A Survey of Literacy, Numeracy, and ICT Levels in England. Department for Business Innovation and Skills (BIS); 2012.

27.

Statistics Canada. Skills in Canada: First Results from the Programme for the International Assessment of Adult Competencies (PIAAC). Statistics Canada; 2013.

28.

Svider

Agarwal

Choudhry

, et al. Readability assessment of online patient education materials from academic otolaryngology–head and neck surgery departments. Am J Otolaryngol. 2013;34(1):31-35.

29.

Kasabwala

Agarwal

Hansberry

Baredes

Eloy

JA.

Readability assessment of patient education materials from the American Academy of Otolaryngology–Head and Neck Surgery Foundation. Otolaryngol Head Neck Surg. 2012;147(3):466-471.

30.

Best

Muzaffar

Mitchell-Innes

Quality of information available via the Internet for patients with head and neck cancer: are we improving?

Eur Arch Otorhinolaryngol. 2015;272(11):3499-3505.

31.

Wong

Levi

JR.

Readability trends of online information by the American Academy of Otolaryngology–Head and Neck Surgery Foundation. Otolaryngol Head Neck Surg. 2017;156(1):96-102.

32.

Hansberry

Agarwal

Shah

, et al. Analysis of the readability of patient education materials from surgical subspecialties. Laryngoscope. 2014;124(2):405-412.

33.

Kandula

Zeng-Treitler

Creating a gold standard for the readability measurement of health texts. AMIA Annu Symp Proc. 2008;2008:353-357.

34.

Williamson

Martin

AG.

Analysis of patient information leaflets provided by a district general hospital by the Flesch and Flesch-Kincaid method. Int J Clin Pract. 2010;64(13):1824-1831.

35.

Grabeel

Russomanno

Oelschlegel

Tester

Heidel

RE.

Computerized versus hand-scored health literacy tools: a comparison of Simple Measure of Gobbledygook (SMOG) and Flesch-Kincaid in printed patient education materials. J Med Libr Assoc. 2018;106(1):38-45.

36.

O’Connell Ferster

Evaluating the quality and readability of Internet information sources regarding the treatment of swallowing disorders. Ear Nose Throat J. 2017;96(3):128-138.

37.

Brangan

Development of SMOG-Cro readability formula for healthcare communication and patient education. Coll Antropol. 2015;39(1):11-20.

38.

Cohen

Statistical Power Analysis for the Behavioral Science. 2nd ed. Lawrence Erlbaum; 1988.

39.

Sullivan

Feinn

Using effect size—or why the P value is not enough. J Grad Med Educ. 2012;4(3):279-282.

40.

Wilson

Mitchelson

Tzeng

, et al. Caring for the surgically anxious patient: a review of the interventions and a guide to optimizing surgical outcomes. Am J Surg. 2016;212(1):151-159.

41.

Koay

Schofield

Gough

, et al. Suboptimal health literacy in patients with lung cancer or head and neck cancer. Support Care Cancer. 2013;21(8):2237-2245.

42.

Kubba

Reading skills of otolaryngology outpatients: implications for information provision. J Laryngol Otol. 2000;114(9):694-696.

43.

Kong

Readability assessment of online tracheostomy care resources. Otolaryngol Head Neck Surg. 2015;152(2):272-278.

44.

Lee

Vescan

Lee

JM.

Evaluating YouTube as a source of patient information for functional endoscopic sinus surgery [published online October 6, 2020]. Ear Nose Throat J.

45.

Balakrishnan

Chandy

Hseih

Bui

Verma

SP.

Readability and understandability of online vocal cord paralysis materials. Otolaryngol Head Neck Surg. 2016;154(3):460-464.

46.

Levin

M WV

Lee

Cusimano

Lee

JM.

Validity and usefulness of YouTube videos related to endoscopic transsphenoidal surgery for patient information [published online January 19, 2021]. J Neurol Surg Pt B Skull Base.

47.

Miller

TA.

Health literacy and adherence to medical treatment in chronic and acute illness: a meta-analysis. Patient Educ Couns. 2016;99(7):1079-1086.

48.

Clarke

Dunne

Coffey

, et al. Health literacy impacts self-management, quality of life and fear of recurrence in head and neck cancer survivors [published online January 9, 2021]. J Cancer Surviv.

49.

Kasabwala

Misra

Hansberry

, et al. Readability assessment of the American Rhinologic Society patient education materials. Int Forum Allergy Rhinol. 2013;3(4):325-333.

50.

Nilsen

Moskovitz

Lyu

, et al. Health literacy: impact on quality of life in head and neck cancer survivors. Laryngoscope. 2020;130(10):2354-2359.

51.

Clarke

Moore

Steege

, et al. Health information needs, sources, and barriers of primary care patients to achieve patient-centered care: a literature review. Health Inform J. 2016;22(4):992-1016.

52.

Blanck

Marshall

Patient education materials from the layperson’s perspective: the importance of readability. J Nurses Staff Dev. 2011;27(2):E8-10; quiz E11-12.

53.

Mayo

HG, Pestonjee SF

. Creating understandable health information. J Hosp Librarianship. 2004;4(4):113-118.

54.

Coulter

Entwistle

Gilbert

Sharing decisions with patients: is the information good enough?

BMJ. 1999;318(7179):318-322.

55.

Mayeaux

Jr Murphy

Arnold

Davis

Jackson

Sentell

Improving patient education for patients with low literacy skills. Am Fam Physician. 1996;53(1):205-211.

56.

Bernier

MJ.

Developing and evaluating printed education materials: a prescriptive model for quality. Orthop Nurs. 1993;12(6):39-46.

57.

Jackson

Davis

Bairnsfather

George

Crouch

Gault

Patient reading ability: an overlooked problem in health care. South Med J. 1991;84(10):1172-1175.

58.

Cherla

Sanghvi

Choudhry

Liu

Eloy

JA.

Readability assessment of Internet-based patient education materials related to endoscopic sinus surgery. Laryngoscope. 2012;122(8):1649-1654.

59.

Albert

Chadwick

How readable are practice leaflets?

BMJ. 1992;305(6864):1266-1268.

60.

Manning

Writing readable health messages. Public Health Rep. 1981;96(5):464-465.

61.

Brooks

DA.

Techniques for teaching ED patients with low literacy skills. J Emerg Nurs. 1998;24(6):601-603.

62.

Pentz

Lohani

Hayban

, et al. Videos improve patient understanding of misunderstood chemotherapy terminology. Cancer. 2019;125(22):4011-4018.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB

0.01 MB