Abstract
Background:
Given the increasing accessibility of Internet access, it is critical to ensure that the informational material available online for patient education is both accurate and readable to promote a greater degree of health literacy. This study sought to investigate the quality and readability of the most popular online resources for ankle fractures.
Methods:
After conducting a Google search using 6 terms related to ankle fractures, we collected the first 20 nonsponsored results for each term. Readability was evaluated using the Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), and Gunning Fog Index (GFI) instruments. Quality was evaluated using custom created Ankle Fracture Index (AFI).
Results:
A total of 46 of 120 articles met the inclusion criteria. The mean FKGL, FRE, and GFI scores were 8.4 ± 0.5, 57.5 ± 3.2, and 10.5 ± 0.5, respectively. The average AFI score was 15.4 ± 1.4, corresponding to an “acceptable” quality rating. Almost 70% of articles (n = 32) were written at or below the recommended eighth-grade reading level. Most articles discussed the need for imaging in diagnosis and treatment planning while neglecting to discuss the risks of surgery or potential future operations.
Conclusion:
We found that online patient-facing materials on ankle fractures demonstrated an eighth-grade average reading grade level and an acceptable quality on content analysis. Further work should surround increasing information regarding risk factors, complications for surgery, and long-term recovery while ensuring that readability levels remain below at least the eighth-grade level.
This is a visual representation of the abstract.
Introduction
Ankle injuries constitute roughly one-third of lower extremity injuries, and fractures attribute about one-fourth and one-eighth of ankle and foot injuries, respectively. 19 Approximately 120 000 ankle fractures are seen nationally. 28 As a result of increasing access to the Internet over the past 10 years, it is important that orthopaedic surgeons are aware of the information that is available online. More than 60% of orthopaedic patients refer to the Internet for information regarding their conditions, and up to one-third of them engage their surgeons in conversations about what they’ve read. 11 Although this phenomenon aids in the dissemination of health information and may increase health literacy, the potential for the spread of misinformation remains high and works to negate the benefits that this increased access may afford.9,10,15
Patient health literacy, defined as one’s ability to process and comprehend basic health information that is required to make informed health decisions, 15 has been shown to drastically improve patient outcomes and expedite patient recovery and is largely dependent on the readability of the patient education materials (PEMs).3,24 Taking into account the average reading grade level in America, the American Medical Association (AMA) and the National Institute of Health (NIH) both advocate for presenting PEMs at a sixth- and eighth-grade reading level, respectively, to ensure that the majority of patients can comprehend the provided health materials.8,34
Unfortunately, it has been shown that the majority of online PEMs written about various orthopaedic conditions fail to satisfy the AMA and NIH readability standards.3,7,26 Readability analysis of PEMs regarding ankle arthroplasty,13,30 ankle instability,1,29 ankle fusion, 16 and other various foot and ankle conditions2,14,20 demonstrates that, similar to orthopaedics as a whole, the majority of online foot and ankle PEMs are written at a reading grade level above the national recommendations. This study seeks to evaluate the quality and readability of information presented in the most popular online resources regarding ankle fractures. We hypothesize that articles on ankle fractures will be of inadequate quality and readability for patient education.
Methods and Materials
Data Gathering
A comprehensive search using the Google search engine in an incognito window was used to identify relevant PEMs on ankle fracture on September 3, 2023. Data from the first 20 results under the following search terms were extracted: “ankle fracture,” “broken ankle,” “broken foot,” “foot fracture,” “pilon fracture,” and “fibular fracture.” When the same article was encountered under multiple search terms, it was included once under whichever search term provided greater traffic to the article (ie, the article appeared closer to the top of the search query). Exclusion criteria included video links, scientific papers, non-English publications, articles shorter than 100 words, and subscription-based articles. Any search result listed as “Advertisement” or “Sponsored” was excluded. An overview of the search methodology and screening flow can be found in Figure 1. Data extracted included website title, date of publication if available, publisher type, and word count. Publisher type was categorized as described in Ng et al 22 with the following categories: (1) academic, (2) commercial, (3) nonprofit organizations (NPOs), and (4) physician. Academic sources were defined as websites affiliated with a university, health care system, or health care society such as American Orthopaedic Foot & Ankle Society (AOFAS) and the American College of Foot and Ankle Surgeons. Commercial websites were sources that received funding through advertisements or sold products/services. NPO sources were defined as websites operating through government funding or on a donation-based platform, and physician websites were those representing physicians and physician groups not affiliated with an academic institution. Institutional review board approval was not required because of the open-access availability of the websites analyzed in this study.
Screening flow diagram.
Readability Assessment
All text from the articles was copied and pasted into separate Microsoft Word documents (Microsoft, Redmond, WA). All pictures, videos, advertisements, copyright notices, references, and other text not directly related to ankle fractures were removed. ReadablePro (https://app.readable.com/text/), 25 an online calculator, was then used to evaluate the readability of each reformatted document as described in Sudah et al. 31 The calculator was used to generate a score for the following validated instruments: Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease (FRE), and Gunning Fog Index (GFI). Both FRE and FKGL calculate readability based on the average sentence length and the average word length. The FKGL formula translates the readability of written materials into grade levels as defined by US education standards, with higher FKGL scores correlating to increased difficulty in reading and comprehension. Conversely, a higher FRE score is associated with greater comprehensibility. Similar to FKGL, the GFI score provides another estimation for grade level but is calculated using the average words per sentence and the average number of complex words (those containing 3 or more syllables). 17 Thus, articles with good readability will have low FKGL and GFI scores while having a high FRE score.
Quality Analysis
The content of the articles was evaluated using the ankle fracture index (AFI), a 25-item scoring rubric assessing a PEM’s inclusion of pertinent information surrounding the diagnosis, treatment, and rehabilitation that was created by a fellowship-trained foot and ankle orthopaedic surgeon (D.F.). The inclusion of a checklist item equates to 1 point, for a maximum of 25 points. The authors designated an AFI score of 0 to 10 as “poor” quality, 11 to 20 as having “acceptable” quality, and 21 to 25 as “good” quality. Importantly, the threshold for awarding points was low, such that an article was given 1 point if it satisfied any aspect of the checklist item. The AFI is based on recommendations by the AOFAS as well as additional information thought to be important for shared decision-making between the patient and physician.
Statistical Analysis
Comparative statistics were used to analyze the readability and quality scores. One-way analysis of variance (ANOVA) was used to identify statistical significance between source types, AFI scores, and FK readability scores. Correlation analysis via Pearson correlation coefficient for normally distributed data was conducted to determine associations among website word count, AFI, and readability scores. The threshold for significance was P <.05 in all statistical tests. The statistical analysis was conducted with SPSS, version 16.0 (IBM, Armonk, New York)
Results
Of the initial 120 results, 46 PEMs were included in the analysis from 46 unique sources, with some websites populating under multiple search terms (Appendix 1).
Readability Analysis
The mean FKGL, FRE, and GFI scores were 8.38 ± 0.48, 57.47 ± 3.15, and 10.49 ± 0.53, respectively. The average word count was 1129.45 ± 181. Articles populated under the search query “broken ankle” displayed the highest FRE and lowest FKGL scores at 61.03 ± 4.01 and 7.89 ± 0.69, respectively. Almost half of the PEMs were categorized as being fairly difficult to read (n = 19, 41.3%) (Table 1). Only 1 PEM (2.2%) was written above a standard reading ease level. No statistically significant differences between FK readability scores and search term (P = .3828) or word count (P = .8476) were observed. Almost 70% (n = 32, 69.6%) of PEMs were written at or below an eighth-grade reading level, with 24 articles scoring between a seventh- and eighth-grade level and 8 articles below a sixth-grade level. A summary of the quality and readability scores of all the analyzed websites can be found in Table 2.
FRE Score Interpretation.
Abbreviation: FRE, Flesch Reading Ease.
Summary of Quality and Readability Results of All Websites.
Abbreviations: AFI, Ankle Fracture Index; FKGL, Flesch-Kincaid Grade Level; FRE, Flesch Reading Ease; GFI, Gunning Fog Index; NPO, nonprofit organization.
Quality Analysis
The mean overall AFI score was 15.41 ± 1.4, corresponding to an overall acceptable quality (Table 3). Nearly all articles discussed the potential need for definitive fixation via open reduction and internal fixation (ORIF) and the utility of radiographic films in confirming the diagnosis of ankle fractures and planning treatment (n = 44, 95.7% and n = 43, 93.5%, respectively). Articles generally lacked information regarding the potential for future surgeries (n = 8, 17.4%), risk factors and prevention of ankle fractures (n = 13, 28.3%), or complications and risks of surgical treatment (n = 14, 30.4%). We found AFI to correlate significantly with increasing word count (r = 0.603, P < .001). No association was found between AFI and FKGL, FRE, or GFI. No statistical difference was found between search term and AFI (P = .2118).
Ankle Fracture Index.
Abbreviations: CT, computed tomography; MRI, magnetic resonance imaging; XR, radiography.
Source Analysis
Academic sources comprised the majority of articles (n = 27, 58.7%) and had the lowest reading grade level (8.01 ± 0.49), whereas NPO sources accounted for the lowest number of articles (n = 3, 6.5%) and the highest reading grade level (8.51 ± 9.80). AFI, FK scores, and word count did not vary substantially between source types.
Discussion
Ankle fractures are one of the most common types of fracture, most commonly occurring in the winter time secondary to falls or sports-related injuries. 27 Although the average age of patients with ankle fractures varies depending on patient sex, incidence typically peaks earlier between 10 and 25 years of age for male patients and later between 70 and 80 years of age for female patients.27,28,35 The present study found that the most popular online PEMs regarding ankle fracture are of acceptable quality and meet the NIH recommendation of being written at an eighth-grade reading level. The articles evaluated in this study demonstrated an average quality score of 15.07 with an average FKGL of 8.38. Nearly 70% of the evaluated PEMs scoring below the eighth-grade reading level. To our knowledge, this study demonstrates the closest that online PEMs focusing on orthopaedic conditions come to satisfying the NIH guidelines for readability. Our results were similar to those described by Smith et al, 30 who found that online PEMs regarding total ankle arthroplasty were written with an average FKGL of 8.97.
Given that peak incidence of ankle fractures is during the winter months when orthopaedic clinics and urgent cares are more likely to be closed because of the holiday season, as well as the high volume of ankle fractures in adolescents, patients are more inclined to turn to the Internet for information regarding the source of their ankle pain. In addition, the oftentimes staged approach to the treatment of pilon fractures specifically, offers patients a chance to research their condition after receiving a confirmed diagnosis. 21 One survey found that up to 50% of orthopaedic foot and ankle surgeons point patients to the Internet for information. 1 Although this is an opportunity for patients to increase their awareness and knowledge of their diagnoses, the lack of regulation of online materials can introduce significant potential for misinformation and subsequent decreased health literacy. This can create potentially uncomfortable conversations where physicians may need to correct a patient’s misconceptions, which may negatively impact the physician-patient relationship. Decreased health literacy also presents a substantial economic burden, increasing national health care costs by up to $73 billion. 34 Thus, it is of particular interest to orthopaedic surgeons to ensure that the content patients are consuming on the web is written at an appropriate grade level for optimized comprehension.
After evaluating the over 100 articles listed on the AOFAS website, Hartnett et al 14 found that the average readability has worsened over the past 15 years, with articles written, on average, at a 9th- to 12th-grade reading level. Similarly, Abousayed et al 1 found that 90% of the evaluated ankle instability-related PEMs were also written at a reading grade level above NIH recommendations, although they used a seventh-grade cutoff as opposed to our eighth-grade benchmark. Noback et al 23 also investigated ankle fracture-related PEMs, using the terms “ankle fracture,” “broken ankle,” and “fibular fracture,” and found the average grade level to be 9.6 ± 1.7. Similar to us, Noback et al also found “broken ankle” to populate the most readable articles. The researchers also applied a custom quality grading criteria and found the average score to be 13.1 ± 6.8 out of a maximum of 36.
Contrary to these studies, we found that ankle fracture-related PEMs, when analyzed from numerous sources across the Internet, are written, on average, at an eighth-grade reading level. The number of articles written below a sixth-grade reading level, as recommended by the AMA, however, does not differ substantially between those evaluated by Hartnett et al 14 and those evaluated in our study. This is of great importance because, although the NIH recommends articles be written at least at an eighth-grade reading level, over half of American adults read below the sixth-grade level. 36 With regard to potential trade-offs between quality and readability, Abousayed et al 1 found the average quality of online PEMs to be poor and reported a statistically higher article quality when written above a seventh-grade reading level. On the other hand, Noback et al 23 report that articles with lower FKGL scores had higher quality measures. Our results differ from both of these studies by showing the average quality of PEMs to be higher with no appreciable difference in quality based on reading grade level. Although a majority of the articles discussed the diagnosis and evaluation of ankle fractures in detail, most articles failed to provide sufficient information regarding the risks and complications of ankle fractures, risks of surgical treatment, or postoperative recovery. In addition to improving the readability of ankle fracture–related PEMs, improvements to such articles could include efforts to provide information surrounding options for postoperative instructions such as air boots vs plaster casts, weightbearing limitations, and early mobilization.4,6 This may prove difficult, however, as care decisions such as whether to splint, cast, or boot may vary based on physician preference and specific patient characteristics. Nonetheless, introducing this topic may prime patients for whichever option their surgeon decides in the postoperative period, potentially increasing patient compliance. Increased attention to providing accurate and easily accessible references may also benefit those patients who are interested in learning more about their condition.
Thus, although there have been advancements in the quality and readability of ankle fracture-related PEMs, there is still room for further improvement. Given the variability in quality and readability levels, patients are likely to encounter information that is not entirely accurate. In such instances, physicians may benefit from having investigated popular PEMs so that they may redirect patients to a curated list of surgeon-approved articles. The relative scarcity of physician-created PEMs identified in our study (n = 5, 11%) indicates a potential space for surgeons to take proactive steps in combating misinformation. In particular, this can be accomplished through the use of artificial intelligence and large language models (LLMs), such as ChatGPT, to interpret educational material into a sixth- to eighth-grade reading level and include a comprehensive overview of the diagnosis, treatment, recovery, and risk/complications of ankle fractures.18,33 Vallurupalli et al 33 found that the public-use version of ChatGPT (ChatGPT 3.5) had the capability of simplifying PEMs to an acceptable reading grade level using the prompt “Rewrite this paragraph for an 8th-grader without losing information from the original paragraph.” While these studies investigated ChatGPT’s ability to translate existing PEMs into a lower reading grade level, there were limited analyses on the quality of the simplified material. Tao et al, 32 however, describe ChatGPT’s utility in generating novel PEMs regarding neuro-ophthalmology that may be deemed satisfactory to fellowship-trained ophthalmologists. Thus, it is feasible that ChatGPT may be used to either augment existing PEMs with missing information, such as risks of surgical treatment, or render new PEMs entirely. Indeed, further research is indicated to investigate the application of LLM in generating novel PEMs in orthopaedics. Future applications may even explore the potential for translating English-language PEMs into alternate languages.
Limitations
The dynamic nature of the Internet stands out as a primary limitation in this study. Because information online is constantly changing, our study reflects an analysis of the most popular resources for only the date of data extraction. Furthermore, patients may use alternate avenues and search terms when researching ankle fractures or may visit sites past the top 20 results that were evaluated in this study. Nevertheless, studies have shown that the Google search engine is among the most trusted by patients 12 and that the majority of people emphasize the first 10 results for any given Google search query. 5 Given that the AFI rubric used for quality analysis was based off of both AOFAS recommendations and the subjective considerations of a practicing foot and ankle surgeon, the items to include in the rubric may be up to interpretation and can differ among surgeons. Another limitation was the use of a single evaluator for our quality analysis, which may increase the risk of bias in our results. We, however, believe that by employing a “yes” or “no” rubric and using a low threshold for awarding a point, we limit the amount of bias in our quality analysis.
Conclusion
Ankle fractures are a common injury among both adolescents and the elderly with varying treatment options. With the high degree of shared decision making in treatment planning, patient health literacy is heavily influenced by the quality and readability of online materials that patients may encounter before speaking with their orthopaedic surgeon. The most popular online ankle fracture–related PEMs were found to be of acceptable quality and written at an eighth-grade reading level. Efforts to improve such PEMs should revolve around including information regarding risk factors for ankle fractures and complications of surgery as well as further decreasing the reading grade level.
Supplemental Material
sj-pdf-1-fao-10.1177_24730114241241310 – Supplemental material for Analysis of the Most Popular Online Ankle Fracture–Related Patient Education Materials
Supplemental material, sj-pdf-1-fao-10.1177_24730114241241310 for Analysis of the Most Popular Online Ankle Fracture–Related Patient Education Materials by Haad A. Arif, Gavin LeBrun, Simon T. Moore and David A. Friscia in Foot & Ankle Orthopaedics
Footnotes
Appendix 1
Ethical Approval
Ethical approval was not sought for the present study because no human subjects were included for use.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Disclosure forms for all authors are available online.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
