Abstract
Background:
Appropriate education on the disease processes associated with orthopaedic pathology can affect patient expectations and functional outcome.
Hypothesis:
Patient education resources from the American Orthopaedic Society for Sports Medicine (AOSSM) are too complex for comprehension by the average orthopaedic patient.
Study Design:
Cross-sectional study.
Methods:
Patient education resources provided by the AOSSM were analyzed with software that provided 10 readability scores as well as opportunities for improving readability. The readability scores were compared with the recommended eighth-grade reading level.
Results:
A total of 39 patient education resources were identified and evaluated. The mean ± SD reading grade-level scores were as follows: Coleman-Liau Index, 12.5 ± 1.11; New Dale-Chall Readability Formula, 10.9 ± 1.37; Flesch-Kincaid Grade Level, 9.9 ± 1.06; FORCAST Readability Formula, 11.4 ± 0.51; Fry Readability Formula, 12.8 ± 2.79; Gunning Fog Index, 11.9 ± 1.37; Raygor Readability Index, 13.1 ± 2.37; Simple Measure of Gobbledygook, 12.3 ± 0.90; Automated Readability Index, 11.2 ± 1.18; and New Automated Readability Index, 10.6 ± 1.27. After averaging the reading grade-level scores, only 1 patient education resource was found to be written at an 8th- to 9th-grade level, and 14 (36%) were written above a 12th-grade level. All scores were significantly different from the eighth-grade level (P < .0065). The percentage of complex words and long words were 19.6% ± 2.67% and 41.4% ± 3.18%, respectively.
Conclusion:
Patient education resources provided by the AOSSM are at a significantly higher reading level than recommended. Simple changes can drastically improve these scores to increase health literacy and possibly outcome.
The search for reliable information on orthopaedic injuries often leads to subspecialty-specific websites. With the proliferation of internet access, gathering disease-specific information is more convenient than ever. 4 Health literacy has been linked to patient expectation and satisfaction and continues to shape outcomes following treatments for orthopaedic conditions. 1,15,19 Many studies have demonstrated that the readability of patient education resources in various medical fields is too complex for the average patient. In fact, the National Institutes of Health recommends a reading level of sixth to eighth grade. 2,12,13 Studies evaluating the reading level of orthopaedic patient education resources have consistently shown the readability score to be above this recommended level. 5 –7,11,14
Readability is commonly assessed with the Flesch-Kincaid Grade Level score. 11,20 This score is easy to interpret and is incorporated into commonly used word processing software. However, there are other readability scales available, including the Coleman-Liau Index, New Dale-Chall Readability Formula, FORCAST Readability Formula, Fry Readability Formula, Gunning Fog Index, Raygor Readability Index, Simple Measure of Gobbledygook (SMOG), Automated Readability Index (ARI), and New Automated Readability Index (New ARI). These readability scales emphasize different metrics compared with the Flesch-Kincaid and are supported by the Centers for Disease Control and Prevention and the National Cancer Institute. 2,12
The purpose of this study was to evaluate, with these scales, the readability of the American Orthopaedic Society for Sports Medicine (AOSSM) patient education resources. Additionally, we examined the scales for opportunities and examples on how to improve readability. Our hypothesis was that AOSSM patient education resources are more complex than the recommended reading level.
Methods
We conducted a search of the injury-preventing resources at STOP Sports Injuries (“Prevent Injuries,” http://www.STOPSportsInjuries.org). Patient education material was downloaded in June 2017. All patient education resources were written in English. The text of the patient education resource was reformatted into Microsoft Word document files; during the reformatting process, we removed all figures, disclaimers, acknowledgments, citations, references, and hyperlinks. The reformatted patient education resources were then analyzed for readability with Readability Studio Professional Edition 2015 (Oleander Software Ltd). The software provided data, including scores from the Coleman-Liau, New Dale-Chall, FORCAST, Fry, Gunning Fog, Raygor, SMOG, ARI, New ARI, and Flesch-Kincaid. Table 1 provides a summary of the assessments. The software provided additional data, including percentage of complex words (defined as ≥3 syllables) and percentage of long words (defined as ≥6 characters). These terms are defined by the software.
Readability Assessments With the Formulas Used to Calculate Them a
a SMOG, Simple Measure of Gobbledygook.
Descriptive statistics were generated using Microsoft Excel. All reading level scores were averaged to produce a reading grade level for each patient education resource. A 1-sample, 2-tailed t test was used to compare the readability scores with the eighth-grade reading level. Significance was set at P = .0065, based on Bonferroni correction with alpha = 0.05.
Results
A total of 39 patient education resources were identified and evaluated by the Readability Studio software (see Appendix Table A1). Readability scores are summarized in Table 2. Averaging the reading grade-level scores demonstrated that only 1 patient education resource was written at an 8th- to 9th-grade level (“Preventing Baseball Injuries”) and 14 (36%) were written above a 12th-grade reading level (Figure 1). Of 39 patient education resources, 28 (72%) could not be evaluated via Fry on the basis of too many complex words. Similarly, 30 (77%) could not be evaluated for Raygor because of the excessive number of long words. Comparison was made between the reading grade levels of the patient education resources and an eighth-grade reading level. All scores were significantly different from the eighth-grade level (P < .0065) (Table 2).
Readability Data Produced by the Readability Studio Software a
a All scores compared with eighth-grade reading level by 2-tailed t test. Significance was set at P = .0065. ARI, Automated Readability Index; SMOG, Simple Measure of Gobbledygook.
The reading grade level of all scales was averaged to provide a composite readability score. Only 1 patient education resource was written at an 8th- to 9th-grade level, and 14 (36%) were written above a 12th-grade reading level.
Total word count per patient education resource was 714 ± 143 words. The percentages of complex words and long words were 19.6% ± 2.67% and 41.4% ± 3.18%, respectively. Complex words were defined as words with ≥3 syllables, and long words were those with ≥6 characters. Each patient education resource contained a mean 6.90 overly long sentences. The longest sentence was 35.6 ± 6.18 words. Table 3 lists words that were deemed problematic by the software along with suggested alternatives to improve readability.
Problem Words and Suggested Alternatives Produced by Readability Studio Pro Software
Discussion
With the proliferation and nearly ubiquitous consumption of internet access, patients are looking for reliable online sources regarding medical care. 4 Internet searches can often lead to specialty sites such as the AOSSM. Multiple studies have shown that the readability score of patient education resources may be too complex for the average reader. 5 –7,11,14 Our study reveals that the material found on the STOP Sports Injuries website is no different.
Previous studies have used the Flesch-Kincaid score to evaluate readability. 11 , 20 In the current study, we used multiple measures of readability in addition to Flesch-Kincaid, including Coleman-Liau, New Dale-Chall, FORCAST, Fry, Gunning Fog, Raygor, SMOG, ARI, and New ARI. All reported reading grade levels were significantly higher than the recommended eighth-grade reading level. Our results add to the validity of the Flesch-Kincaid scores previously reported.
Health literacy is critical in orthopaedic surgery. Studies have linked literacy to patients’ expectations and outcomes and have emphasized the importance of setting realistic expectations. 1,15,19 A significant portion of the patient education process lies in patient education resources, as studies have also shown that patients may not ask questions in the office or may express understanding when they are actually unsure. 8,10 Additionally, the increasing use of patient-reported outcome measures highlights the need for health literacy given the roles that patient expectation and satisfaction play.
Multiple studies have found that orthopaedic patient education resources are too complex, often well above the eighth-grade reading level. 5 –7,11,14 Other fields of medicine have faced similar challenges and have attempted to simplify the language to improve readability. 3,17,18 Colaco et al 3 evaluated online urology patient education resources using 10 commonly used assessment tools and determined that most were written at an 11th-grade level or higher. Svider et al 18 concluded that online otolaryngology patient education resources were too difficult based on the recommended reading level. They used the same 10 assessment tools as in this study and also determined that most resources were at an 11th-grade level or higher. Both author groups concluded that patient education resources should be simplified to facilitate comprehension. Schoof and Wallace 17 reported on the readability of online family medicine patient education resources. Using an online program that focuses on sentence length and word frequency, the authors found that the majority of currently available family medicine patient education resources were at a sixth-grade level. Interestingly, they found that the number of resources at a sixth-grade level in 2012 had increased significantly since 2004 (59% vs 5%). These authors concluded that efforts should continue to be made to simplify and reduce the reading demand of patient education resources to support patient understanding and communication.
Our data indicate that word complexity and word length are driving the increased reading grade level for online patient education resources for orthopaedic sports medicine. Orthopaedic surgery involves the use of complex terms to describe anatomic structures as well as mechanical principles. This jargon can be intimidating, especially for those untrained in medicine. Orthopaedic surgeons undergo years of specialized training to fully understand the anatomy and disease process; attempting to condense this into generalizable and understandable concepts is a challenge. Even in the medical field, musculoskeletal education was shown to be lacking during medical school. 9,16 Nonetheless, we provide an example of simplifying language with the use of more common terms to refer to anatomy, as well as simpler descriptions in shorter sentences. Figure 2 provides a comparison between the original text and the modified version. The modified version produces a reading grade level lower in all indices except Gunning Fog (Table 4). Here we demonstrate that by making a few simple changes, the reading grade level can be brought to an appropriate level.
An excerpt from a section on anterior cruciate ligament (ACL) injury prevention. The text was edited with the use of simpler language as recommended by Readability Studio Pro software to produce more accessible reading scores.
Readability Scores Comparing the Original Text and the Revised Text a
a Nearly every readability assessment revealed an improved score. ARI, Automated Readability Index; SMOG, Simple Measure of Gobbledygook.
b Test could not be completed because of the short text.
There are several limitations to this study. While there is no gold standard measurement for readability, we chose to use multiple scales that emphasize different aspects of readability to improve our validity. Additionally, the analysis evaluated the text of the patient education material and excluded material such as images, tables, and animations, which may prove helpful. Moreover, while metrics such as syllables per word, words per sentence, and word length may influence readability as measured by software, this does not necessarily correlate with comprehension. The strengths of this study include the use of sophisticated software to provide data on complex and long words. Given the use of multiple comparisons, a Bonferroni correction was used for significance.
The overall goal is for better patient understanding of the disease-specific condition or treatment, and readability of patient education resources is only one factor toward that end. Thus far, there is no consensus on the minimum readability of patient education resources to have a meaningful impact on patient understanding. This study shows that with modification, patient education resources can be provided at an appropriate reading grade level to promote understanding and to ultimately inform medical decisions and better define expectations. Other surgical specialties have noted reading levels of online patient education resources to be too high and are simplifying the language to improve understanding and communication. 3,18 Our study demonstrates that the readability of online patient education resources in orthopaedic sports medicine remains higher than recommended; however, efforts to simplify language can yield a lower reading demand. Providers bear the responsibility of ensuring adequate patient education as well as removing barriers toward that goal. Ultimate outcome can be influenced by patient expectations and satisfaction, and the readability of patient education resources is a modifiable factor in improving care.
Conclusion
Patient education resources provided by the AOSSM are written above recommended reading grade levels. Small changes can produce significant improvement in the readability scores to support increased health literacy. Ultimately, this may improve patient understanding and expectations and lead to improved satisfaction and outcomes.
Footnotes
One or more of the authors has declared the following potential conflict of interest or source of funding: B.G. has received educational support from Arthrex and Smith & Nephew. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
Ethical approval was not sought for the present study.
APPENDIX
Patient Education Resources Identified and Evaluated by the Readability Studio Software
a
a
Asterisks indicate that readability score was unable to be calculated due to too many long or high syllable words. ACL, anterior cruciate ligament; ARI, Automated Readability Index; SMOG, Simple Measure of Gobbledygook.
b
Complex words: ≥3 syllables (values in proportions).
c
Long words: ≥6 characters.
Document
Coleman-Liau
New Dale-Chall
Flesch-Kincaid
FORCAST
Fry
Gunning Fog
Raygor
SMOG
ARI
New ARI
Mean Reading Grade
Complex Words,
b
%
Long Words,
c
%
Range
Mean
ACL
13.1
13-15
14.0
11.1
11.7
14
13.4
17
13.4
13.3
12.3
13.5
19.30
43
Baseball
8.9
7-8
7.5
7.4
10.8
*
8.5
*
10
9.7
8.7
8.9
15.20
36.20
Basketball
11.2
11-12
11.5
9.1
11.2
*
11.5
*
11.8
10
9.1
11.1
19.20
39.60
Cheerleading
11.3
9-10
9.5
9.5
10.9
11
11.1
12
12.3
10.7
9.8
11.0
17.60
37.70
Coaches Tips for Parents
10.8
9-10
9.5
9.4
10.4
10
11.4
11
12.1
10.2
8.9
10.6
14.80
34.50
Concussion
13.3
11-12
11.5
10.3
12.1
*
12
*
12.3
13.2
12.5
11.9
21.50
44.50
Conditioning
14.3
11-12
11.5
11.4
12
16
13.7
*
13.8
12.9
12.3
13.2
22.40
45.50
Cycling
11.4
9-10
9.5
8.9
10.6
10
10.4
11
11.6
9.9
8.8
10.4
15.10
37
Dance
12.1
9-10
9.5
9.1
11.4
*
11.9
*
11.8
11.6
10.6
11.0
19.40
41.90
Encourage Success
11.8
9-10
9.5
8.8
10.4
10
11.6
11
12
10.1
9.3
10.6
17
36.20
Female Triad
13.5
11-12
11.5
11.8
11.9
17
14.3
17
13.8
13
12
13.9
22.40
43.10
Field Hockey
13.2
11-12
11.5
10.9
11.7
*
13.6
*
13.1
11.7
11.3
12.3
23.40
43
Figure Skating
13.5
13-15
14.0
9.6
11.9
*
11.4
*
11.8
11.7
11.6
12.0
21.20
46
Football
12.6
11-12
11.5
10.2
11.5
*
12.2
*
12.5
11.5
10.7
11.8
19.90
43.80
Golf
10.9
9-10
9.5
8.7
10.3
*
9.9
*
11.3
9.9
8.9
10.1
14.80
34.70
Gymnastics
13.2
11-12
11.5
10.1
11.7
*
12.2
*
12.2
11.6
11.2
11.8
22.10
42.50
Heat Illness
11.9
11-12
11.5
10.3
11.7
*
12.1
*
12.4
10.4
9.5
11.7
18.90
39.80
Hockey
12.3
11-12
11.5
9.9
11.5
*
12.8
*
12.6
11.2
10.4
11.8
20.20
42.90
Inline Skating
13.4
11-12
11.5
10.7
11.7
*
10.4
*
12.3
10.8
10.8
11.7
19.30
45.10
Instrumentalists
14.1
11-12
11.5
11.2
12.2
*
13.5
*
13.5
12.3
12
12.7
23.60
45.30
Knee Injuries
10.8
11-12
11.5
10
10.5
11
12.9
13
13
10.4
9
11.6
18.90
38.20
Lacrosse
13.1
11-12
11.5
10.3
11.1
13
11.9
13
12.9
12
11.3
12.1
19.60
40.10
Martial Arts
12.5
11-12
11.5
9.8
11.2
*
10.7
*
11.8
10.2
10.2
11.3
18.70
40.90
Overuse
12.8
11-12
11.5
10.8
12
*
12.8
*
13
10.9
10.6
12.2
22.00
44.40
Prevent Overuse Injuries
12.7
9-10
9.5
9.9
11.5
*
11.8
*
12.2
10.5
9.7
11.3
17.90
39.70
Rowing
12.4
11-12
11.5
9.9
10.9
12
11.2
13
11.9
11.3
10.5
11.6
15.70
38.80
Rugby
13.2
11-12
11.5
10.5
11.7
*
13.5
*
12.8
11.3
10.6
12.2
22.30
42.80
Running
11.9
7-8
7.5
8.1
11.3
*
9.6
*
10.5
10.6
10.3
9.8
16.80
41.30
Skiing and Snowboarding
13.6
11-12
11.5
9.6
11.9
*
12.5
*
11.7
11.5
11.6
11.8
24.20
43.60
Soccer
12.1
11-12
11.5
9.9
11.2
*
12
*
12.6
10.7
10.1
11.6
20.20
40
Softball
11.3
9-10
9.5
8.4
11.2
*
10.1
*
11
9.6
9.5
10.3
18.10
38.80
Sports Nutrition
13.3
11-12
11.5
12.1
11.8
*
14.1
*
14
12.9
12.2
12.8
23.50
44.60
Strength Training
15.2
11-12
11.5
11.7
11.5
17
14.4
*
13.8
14.3
13.8
13.6
22.80
45.70
Swimming
13.9
11-12
11.5
10.1
11.9
*
12.2
*
12.5
12.4
11.8
12.0
20
45.60
Tennis
11
9-10
9.5
8.2
11
*
10.2
*
10.9
9.9
9.1
10.1
14.90
41.20
Volleyball
13.3
11-12
11.5
10.4
11.8
8
12.8
*
12.4
11.4
11.1
11.5
22.20
43.70
Water Polo
13.2
11-12
11.5
10.2
11.7
*
11.3
*
12.3
10.9
10.8
11.7
21
41.80
When Play Is Too Much
11.7
9-10
9.5
8.8
11
*
11.3
*
11.8
9.6
8.9
10.7
18.70
37.90
Wrestling
13.4
11-12
11.5
10.5
11.5
*
12.9
*
13
12.5
11.9
12.1
21.10
43.50