Abstract
Background:
The internet has become an increasingly popular resource among sports medicine patients seeking injury-related information. Numerous organizations recommend that patient educational materials (PEMs) should not exceed sixth-grade reading level. Despite this, studies have consistently shown the reading grade level (RGL) of PEMs to be too demanding across a range of surgical specialties.
Purpose:
To determine the readability of online sports medicine PEMs.
Study Design:
Cross-sectional study.
Methods:
The readability of 363 articles pertaining to sports medicine from 5 leading North American websites was assessed using 8 readability formulas: Flesch-Kincaid Reading Grade Level, Flesch Reading Ease Score, Raygor Estimate, Fry Readability Formula, Simple Measure of Gobbledygook, Coleman-Liau Index, FORCAST Readability Formula, and Gunning Fog Index. The mean RGL of each article was compared with the sixth- and eighth-grade reading level in the United States. The cumulative mean website RGL was also compared among individual websites.
Results:
The overall cumulative mean RGL was 12.2 (range, 7.0-17.7). No article (0%) was written at a sixth-grade reading level, and only 3 articles (0.8%) were written at or below the eighth-grade reading level. The overall cumulative mean RGL was significantly higher than the sixth-grade [95% CI for the difference, 6.0-6.5; P < .001] and eighth-grade (95% CI, 4.0-4.5; P < .001) reading levels. There was a significant difference among the cumulative mean RGLs of the 5 websites assessed.
Conclusion:
Sports medicine PEMs produced by leading North American specialty websites have readability scores that are above the recommended levels. Given the increasing preference of patients for online health care materials, the imperative role of health literacy in patient outcomes, and the growing body of online resources, significant work needs to be undertaken to improve the readability of these materials.
Worldwide internet use has increased more than 11-fold over the past 20 years. 29 The widespread accessibility of the internet has led to a marked change in the physician-patient relationship and each day in the United States, >8 million individuals seek online health care information. 15 Mirroring this trend, the internet has quickly become the preferred source for orthopaedic patient education. Sports medicine patients represent a high-demand, high-functioning population who frequently go online to research their injury. 34,59 Recognition of the importance of health care information has led to the development of a number of websites with dedicated sports medicine patient educational materials (PEMs).
Central to the success of PEMs is the ability of consumers to comprehend them. Health literacy is defined as the “capacity to obtain, interpret, and understand basic health information and services and the competence to use such information and services to enhance health.” 51 Poor health literacy has been referred to as an “under-recognized silent epidemic,” 49 and in the sports medicine setting can be particularly detrimental, as it is associated with treatment noncompliance, 53 hindering return-to-play, and worse functional outcomes after injury. 26 Key to the improvement of health literacy is the ability of patients to comprehend the materials available to them. As a result, it is essential that the readability of available PEMs is at an appropriate level to communicate their intended meaning.
The average adult in the United States reads at an eighth-grade reading level, 28,33 and the average patient reads 5 grade levels below their reported graduation grade. 14,30 A further 1 in 5 American adults have difficulty comprehending materials written at a fourth-grade level. 13 In addition, an estimated 36% of American adults have basic, or below basic, health literacy, 37 with the prevalence of limited musculoskeletal health literacy estimated to be even greater. 55 These patients may struggle to make informed decisions and manage their injuries effectively. Even among patients with strong literacy skills, there is a preference for online content with a lower reading grade level (RGL). 11,66
As a result, several health care organizations including the American Medical Association, 67 the Agency for Healthcare Research and Quality, 5 and the National Work Group in Cancer and Health 10 have advised that PEMs should not exceed the sixth-grade reading level. Similarly, the National Institutes of Health has advised that PEMs should not surpass the seventh- to eighth-grade levels. 45 However, studies have shown repeatedly that the RGL of PEMs across multiple orthopaedic subspecialties fails to meet these recommendations. 17,25,42,46
In separate studies in 2014, Ganta et al 20 and Eltorai et al 15 examined the RGL of PEMs available from the patient education libraries of 2 established American sports medicine websites, using a single readability algorithm [Flesch-Kincaid Grade Level (FKGL)]. 32 They determined that the majority of PEMs were written at a level that was inappropriate for the average American adult, with a mean FKGL score of at least 10 found across both studies; 7 years later, the volume of PEMs available on the websites of these organizations is suspected to have increased significantly.
The aim of this study was to apply multiple algorithms to calculate the readability of PEMs available from 5 North American specialty websites to determine if progress has been made. Despite calls for improved PEM readability made by the authors of the index studies, we hypothesized that PEMs in the field of sports and exercise medicine continue to be produced at a level that exceeds current recommendations.
Methods
In May 2021, we searched the patient education libraries of the American Academy of Orthopaedic Surgeons (AAOS; www.aaos.org), American College of Sports Medicine (ACSM; www.acsm.org), American Medical Society for Sports Medicine (AMSSM; www.amssm.org), American Orthopaedic Society for Sports Medicine (AOSSM; www.sportsmed.org/aossmimis), and Canadian Orthopaedic Association (COA; www.coa-aco.org) websites. Patient-orientated educational materials in the field of sports medicine were identified by locating articles specifically designated as “patient” or “public” resources. Articles were excluded if the content was not provided in English; if there was insufficient text to analyze; or if the material was presented primarily in graphic, table, or list format.
Assessment of RGL
All text from each article was copied and pasted into separate Microsoft Word documents. All hyperlinks, pictures, advertisements, copyright notices, and any other text that was not directly related to PEM were removed. The reformatted educational resources were then analyzed for readability using Readability Studio Professional Edition Version 2019 (Oleander Software Ltd). The software assessed readability using 8 different instruments: FKGL, 32 Flesch Reading Ease Score (FRES), 32 Raygor Estimate, 52 Fry Readability Formula, 18,19 Simple Measure of Gobbledygook, 39 Coleman-Liau Index, 9 FORCAST Readability Formula, 7 and Gunning Fog Index (Supplemental Table S1). 24 All RGLs were reported as a US grade level, which denotes the years of education (based on the US educational system) required to easily read and understand a piece of text. For each text document, the 7 RGL tests generated 7 RGL scores, as well as a mean RGL. Unlike the other readability formulas, the FRES formula expressed results as an index score (range, 0-100; higher score indicated easier readability) based on sentence length and number of syllables.
Data were also provided on key linguistic units to determine which aspects of PEMs contributed to reduced readability. These key linguistic units included the number and percentage of complex words, long words, Dale-Chall unfamiliar words, 8 and Fog hard words, 32 as well as the number of “wordy” items, overly long sentences and longest sentence length. Dale-Chall unfamiliar words were defined as those that did not appear on a list of 3000 common words that were known to most fourth-grade students. 8 Fog hard words were defined as those with at least 3 syllables with the exception of proper nouns, 3-syllable words formed by the suffix “ed” or “es,” and compound words composed of simpler words because of hyphenation. 32 Complex words were defined as words with at least 3 syllables; long words, as those with at least 6 characters. 47 Wordy items included complex words and phrases that contained too many words. 47 Overly long sentences were defined as those with a word count >22 words. 47
Statistical Analysis
The number of articles with a mean RGL less than or equal to the eighth grade (the average readability level of American adults) and sixth grade (the recommended readability level for PEMs) was determined. The mean RGL of each article was compared with the sixth-grade and eighth-grade reading levels using 1-sample t tests. The mean RGLs for each article by website and the mean percentage of key linguistic units across all of the articles were compared using 1-way analysis of variance. Post hoc analysis was performed using Bonferroni correction. Statistical significance was set at P < .05. All statistical analysis was carried out in IBM SPSS Statistics for Windows, Version 26 (IBM Corp).
Results
Initially, 369 unique patient educational articles were identified from the 5 websites. Five articles from the COA and 1 article from the AOSSM websites were excluded because of an insufficient amount of text, leaving 363 articles available for final analysis. The overall cumulative mean RGL for all 363 articles was 12.2 (range, 7.0-17.7) (Figure 1).
Box plot displaying the mean reading grade level for each test. The upper edge of the box represents the 75th percentile, the lower edge of the box represents the 25th percentile, the box represents the IQR, the horizontal line in each box represents the median, upper and lower whiskers extend within 1.5 times the IQR, and red circles represent outliers. IQR, interquartile range. SMOG, Simple Measure of Gobbledygook.
The breakdown of articles and cumulative mean RGL by website is shown in Table 1. The cumulative mean RGL of the ACSM website (mean ± SD, 14.6 ± 2.0) was significantly higher than that of the AAOS (95% CI for the difference, 2.5 to 4.5; P < .001), AMSSM (95% CI, 1.8 to 6.6; P = .001), and COA (95% CI, 3.2 to 5.2; P < .001) websites. The cumulative mean RGL of the AOSSM website (mean ± SD, 14.2 ± 1.9) was also significantly higher than that of the AAOS (95% CI, 2.5 to 3.7; P < .001), AMSSM (95% CI, 1.4 to 6.2; P = .005), and COA (95% CI, 3.1 to 4.4; P < .001) websites. There was no significant difference (95% CI, -1.5 to 0.6; P = .791) in the cumulative mean RGL of the ACSM and the AOSSM websites (Table 2).
Web-Based Patient Educational Materials by Website a
a AAOS, American Academy of Orthopaedic Surgeons; ACSM, American College of Sports Medicine; AMSSM, American Medical Society for Sports Medicine; AOSSM, American Orthopaedic Society for Sports Medicine; COA, Canadian Orthopaedic Association.
Mean Reading Grade Level of Patient Educational Articles a
a Data are reported as mean (range) unless otherwise indicated. The Flesch Reading Ease Score was excluded because it expresses results as an index score. AAOS, American Academy of Orthopaedic Surgeons; ACSM, American College of Sports Medicine; AMSSM, American Medical Society for Sports Medicine; AOSSM, American Orthopaedic Society for Sports Medicine; COA, Canadian Orthopaedic Association; GL, Grade Level; SMOG, Simple Measure of Gobbledygook.
Examining the mean RGL for each article revealed that no article (0%) was written at or below the recommended sixth-grade reading level, and only 3 articles (0.8%) were written at or below the eighth-grade reading level. The overall cumulative mean RGL of the articles exceeded the sixth-grade level by an average of 6.2 grade levels (95% CI, 6.0-6.5; P < .001) and the eighth-grade level by an average of 4.2 grade levels (95% CI, 4.0-4.5; P < .001). Of the 363 analyzed PEMs, 34 (9.4%) could not be evaluated via the Fry test, and 52 (14.3%) could not be evaluated via the Raygor Estimate due to too many complex words. The overall cumulative mean FRES index was 47, which is classified as “difficult.”
A summary of the key linguistic units across all of the articles is presented in Table 3 and Figure 2. There was a significantly higher mean percentage of long words (38.7% ± 6.1%) compared with complex words (95% CI, 19.4%-21.8%; P < .001), Fog hard words (95% CI, 20.7%-23.0%; P < .001), Dale-Chall unfamiliar words (95% CI, 9.4%-12.0%; P < .001), and overly long sentences (95% CI, 14.4%-19.1%; P < .001). The mean longest sentence across all articles was 41 words (range, 23 to 177 words). As a reference, all wordy items and suggested alternatives are listed alphabetically in Supplemental Table S2.
Mean Percentage of Linguistic Units for Each Website a
a AAOS, American Academy of Orthopaedic Surgeons; ACSM, American College of Sports Medicine; AMSSM, American Medical Society for Sports Medicine; AOSSM, American Orthopaedic Society for Sports Medicine; COA, Canadian Orthopaedic Association.
Linguistic unit summary. *Significantly different compared with all other linguistic units (P < .05, analysis of variance).
Discussion
As medicine progresses toward a shared decision-making model, patients are increasingly utilizing the internet as their primary source of information on their condition, 35 facilitating greater involvement in health care decisions. However, studies have demonstrated that internet searches for sports medicine PEMs can often lead patients to poor-quality websites with potentially misleading information on diagnosis, treatment, and expected outcomes. 23,59 Patients expect physicians to recommended online PEMs and have a preference for physician-recommended websites. 12,54 Accordingly, sports medicine professionals should be able to direct patients to reputable websites with high-quality information. The goal of the PEMs contained on these websites should be to disseminate accurate information that is easily comprehended by as many patients as possible.
To facilitate this, numerous organizations have recommended the sixth-grade reading level as the upper limit of readability. 5,10,67 Unfortunately, our study found that the average RGL of PEMs contained on 5 leading sports medicine websites was >12, significantly exceeding recommendations. Previously, Ganta et al 20 reported a mean FKGL of 10.2 for articles contained on the AAOS and AOSSM websites. Similarly, Eltorai et al 15 found a mean FKGL of 10.0 for PEMs published on the AAOS website. Our study demonstrated mean FKGLs of 9.6 and 13.1 for PEMs on the AAOS and AOSSM websites, respectively, suggesting that the readability of PEMs has not improved in >7 years. Furthermore, we applied 6 more RGL formulas and included 3 additional websites, all reporting RGLs substantially higher than recommended.
Addressing health literacy among sports medicine patients and caregivers is of critical importance. The health literacy of coaches, parents, and athletes has the potential to influence sports injury–related outcomes. Health literacy is the single best predictor of an individual’s health status. 3,68 Reduced health literacy is associated with poorer general health, 3,69 increased hospitalizations, 2,41 missed appointments, 4 and increased postoperative complications. 57 In addition, those with poor health literacy demonstrate a reduced comprehension of their injury, surgical intervention, and discharge instructions. 31 Compounding these issues, patients with reduced health literacy also express feelings of shame, 48 are less likely to ask clarifying questions, 22 and spend less time with their surgeon during clinic visits. 40 Therefore, many patients may seek online PEMs about their diagnosis, management, and expected outcomes to address gaps in their knowledge. 35
Of concern, none of the articles available across the 5 websites assessed in this study were written at or below the recommended sixth-grade reading level, and only 3 articles (0.8%) were written at or below the eighth-grade reading level. These findings are not exclusive to sports medicine, with similar challenges noted across a diverse array of other medical and surgical specialities. 1,6,27,43,58,70
As a result, the Agency for Healthcare Research and Quality has advised adopting a “universal precautions” approach during PEM content production. 5 This approach involves the implementation of plain-language best practices to produce PEMs that can be understood easily by the majority of patients and caregivers the first time they are read. Several organizations have published guidelines to facilitate the implementation of this approach including using a conversational style, limiting the number of messages, adding useful headings, displaying information in a logical order, using short sentences, and avoiding jargon (Table 4). 44,50,63 –65 Consistent with our linguistic unit analysis, multiple organizations have also advised that the use of simple words and phrases can reduce the health literacy demands of PEMs. 44,50,63 –65 To facilitate this strategy, we have listed all wordy items encountered during document analysis (Supplemental Table S2) with more readable alternatives. We have also provided an example of this strategy in practice with an edited excerpt from 1 of the included articles (Table 5). Furthermore, several plain-language dictionaries are in existence and can be used to substitute difficult clinical nomenclature with lay alternatives during future PEM production. 61,62
Top 20 Plain-Language Best-Practice Guidelines
Edited Passage of Text With Improved Readability a
a RGL, reading grade level.
b Excerpt from “Adolescent Anterior Knee Pain” article on OrthoInfo website provided by American Academy of Orthopaedic Surgeons (https://orthoinfo.aaos.org/en/diseases--conditions/adolescent-anterior-knee-pain).
c Calculated from the 7 RGL tests as described in the Methods section.
Multiple plain-language guidelines have also advised that, before publication, PEMs should be tested with their target audience to address any specific health literacy barriers. This can be achieved with the use of field testing and calculating PEM readability using easily accessible software, such as the one used in this study. Finally, measures such as providing multilingual PEMs that address cultural and linguistic barriers, which can significantly affect the health literacy of sports medicine patients, 60 may also enhance patient health literacy. Of the 5 websites assessed in this study, only the AAOS website provided multilingual content.
Limitations
We acknowledge several potential limitations to our study. First, from the countless number of sports medicine PEMs available online, we chose to only analyze those written in English from a limited number of websites. This may have introduced a selection bias, limiting the generalizability of our findings. Given that >25 million people in the United States have limited English proficiency, 71 assessing and optimizing the readability of PEMs in other languages is imperative. However, despite the absence of non–English language PEMs included in our analysis, to the best of our knowledge, this is the largest study of its kind undertaken in the field of sports medicine.
Furthermore, a fundamental flaw of readability analysis is the inability to account for the effect of visual materials such as figures, tables, and other multimedia adjuncts. These nontextual elements contribute significantly to a patient’s capacity to comprehend complex medical information and are considered a fundamental part of overall understandability. 36 Other measures such as the Suitability Assessment of Materials and the Patient Education Materials Assessment Tool, which consider these nontextual elements, have been described previously. However, these measures do not produce an RGL, which acts a quantifiable target with which to align PEM readability with health care organizations’ recommendations. In addition, these instruments remain relatively novel and unvalidated and possess an inherent degree of subjectivity. 21,38,56,66 Furthermore, other essential elements of communication, including validity and accessibility, were not included in our analysis. Another potential limitation of our methodology is that we did not assess the literacy levels of our patient population. We assumed that the literacy level of the average sports medicine patient was similar to that of the general public. This assumption was based on the fact that, with improved internet access, the online population and the general public have become increasingly similar. 16
Finally, although the 8 readability algorithms used in this study are employed commonly in education, their use in health care literature remains unvalidated. Readability formulas estimate, in one way or another, reading difficulty level using letters per word, syllables per word, and/or words per sentence. This calculation method ignores the nuances of medical jargon, where acronyms or complex words with few syllables such as “Sever’s” or “SLAP” tear may produce low readability scores despite a lack of understanding to a naive reader. However, the converse is also true, where complex terminology, despite being familiar to patients as it represents their own diagnosis, may generate unfavorable readability scores. As there is no standard health care literature readability assessment tool, we chose to implement several formulas that emphasize various aspects of readability to improve our validity.
Conclusion
Sports medicine PEMs published by some of the world’s leading orthopaedic organizations have readability scores significantly exceeding recommendations. Despite the growing appreciation of the importance of health literacy, the readability of sports medicine PEMs has not improved over the past 7 years. Given the increasing preference of patients for online health care materials, the imperative role of health literacy in patient outcomes, and the growing body of online resources, significant works needs to be undertaken to improve the readability of these materials. It is crucial, therefore, that sports medicine professionals adopt a leadership role by recognizing the deficiencies of current PEMs and galvanizing a paradigm shift that emphasizes compliance with plain-language best practices. The adoption of this approach has the potential to lower health literacy demands and improve patient outcomes.
Supplemental material for this article is available at http://journals.sagepub.com/doi/suppl/10.1177/23259671221092356.
Supplemental Material
Supplemental Material, sj-pdf-1-ojs-10.1177_23259671221092356 - Readability of Patient Educational Materials in Sports Medicine
Supplemental Material, sj-pdf-1-ojs-10.1177_23259671221092356 for Readability of Patient Educational Materials in Sports Medicine by Tiarnán Ó Doinn, James M. Broderick, Rebecca Clarke and Niall Hogan in Orthopaedic Journal of Sports Medicine
Footnotes
Final revision submitted December 23, 2021; accepted February 3, 2022.
The authors declared that there are no conflicts of interest in the authorship and publication of this contribution. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
Ethical approval was not sought for the present study.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
