Republication of “Online Patient Resources for Ankle Instability: An Objective Analysis of Available Materials”

Abstract

Background:

The Internet has drastically changed how patients access health-related information. There are several ways the public can access online health-related information such as search engines, blogs, support groups, and webinars. A recent study found that 45% of orthopedic patients searched for information online regarding their injury. Also, 78% believed they had better understanding of their condition after visiting these websites; furthermore, 41% felt the Internet supplied them with questions and concerns to discuss with their physicians. The aim of our study is to evaluate the accuracy, quality, and readability of online available information using the search terms “ankle sprain” and “ankle instability.”

Methods:

Three search engines (Google, Bing, and Yahoo) were used to search for the terms “ankle sprain” and “ankle instability.” The first 25 websites from each search were collected. Each website was assessed for quality, accuracy, and readability by 3 orthopedic residents blinded to the search term used. Websites were also evaluated for commercial bias and whether written by physicians or not.

Results:

Twenty sites were identified using Google, 14 using Bing, and 3 using Yahoo while the remaining 19 appeared in multiple search engines. Sixty-nine percent of the websites (39/56) were written by physicians whereas only 21% (12/56) were associated with commercial bias. The mean quality and accuracy of the websites written above a seventh-grade level was statistically significantly higher than those at or below a seventh-grade level (P = .01). The mean accuracy of websites written by physicians was not statistically different from those not written by physicians (P = .055).

Conclusion:

The current study highlights the poor quality and accuracy of online information related to ankle sprains, especially those with commercial bias. Furthermore, although websites written by or under supervision of physicians were found to be of superior quality, a majority of sites were found to have an unacceptably high reading level.

Level of Evidence:

Level IV, case series.

Commentary: This study evaluated the accuracy, quality, and readability of online available information for ankle sprain and ankle instability. Rigorous methodology was used. Perhaps not surprisingly, the authors found poor quality and accuracy of online information related to ankle sprains, especially those with commercial bias.

Introduction

The Internet has drastically changed how patients access health-related information. There are several ways the public can access online health-related information such as search engines, blogs, support groups, and webinars.^5,8 Some of the potential benefits of patients’ using the Internet include patients’ utilizing the health care system and specialist visits more effectively, time savings to the patients and their physicians, increased patient satisfaction, and improved outcomes.^4,10,17,28 A recent study found that 45% of orthopedic patients searched for information online regarding their injury. Also, 78% believed they had better understanding of their condition after visiting these websites; furthermore, 41% felt the Internet supplied them with questions and concerns to discuss with their physicians. ³¹

Unfortunately, several studies have reported on the poor quality and inaccuracy of online information pertaining to several orthopedic problems.^{3,7,18,19,22,25} One study reported on the quality and accuracy of 82 sites with respect to shoulder instability using the search terms “shoulder instability,” “loose shoulder,” and “shoulder dislocation.” ¹⁸ Overall, they found that the quality and accuracy of the sites were poor and written at an inappropriately high reading level. The average reading level of the population has been previously cited as equivalent to an eighth-grade level; however, the National Institutes of Health (NIH) recommends patient health information to be written at a seventh-grade level in order for the materials to be accessible to a broader audience. ²⁴

Ankle sprains are one of the most frequently encountered orthopedic problems. It is particularly common in athletes, representing about 40% of all sports-related injuries. ⁹ Although most patients experience favorable outcomes after proper management of an isolated sprain, inadequate treatment can lead to chronic ankle instability. Patients with first-time ankle sprains rarely present to orthopedic surgeons. In the past, these injuries were managed by patients, primary care physicians, or emergency medicine physicians; however, the rise of the Internet has coincided with an increased level of self-care. One study compared the accuracy, quality, and credibility of online resources using the term “ankle sprain.” ²⁰ They used two search strategies—one conducted by orthopedic surgeons and the second by information experts. They reported generally poor quality and accuracy of online content, particularly regarding diagnosis and treatment of ankle sprains. However, this study did not include “ankle instability” in their search. We are not aware of published studies that used different search terms to evaluate accuracy and quality of ankle instability–related online information.

The aim of our study is to evaluate the accuracy, quality, and readability of online available information using the search terms “ankle sprain” and “ankle instability.” We believe the Internet should contain accurate and high-quality content pertaining to ankle sprains and more importantly at an average reading level. This is particularly important given the frequency of this problem and the likelihood of patients turning to the Internet for guidance. We assumed that websites written by or under supervision of orthopedic surgeons would have a higher level of quality. Furthermore, we predicted most websites would have a higher readability grade than the average level recommended by the NIH.

Methods

Three search engines (Google, Bing, and Yahoo) were used to search for the terms “ankle sprain” and “ankle instability.” These search engines were previously found to represent >90% of all searches. The first 25 websites from each search were collected. After removal of duplicates, 65 websites were left for evaluation (Figure 1). Nine websites were excluded because of the following reasons: contained video links only (n=3), lacked information (n=3), presented unrelated information (n=2), or was not written in English (n=1). A total of 56 websites met the inclusion criteria.

Figure 1.

A flowchart representing the selection process of the websites. After initial analysis, 150 searches were obtained. With use of exclusion criteria (duplicate websites, unrelated websites, websites that lacked information, websites meant for peer review, websites not available in English, and websites with video-only content), a total of 56 met our inclusion criteria. These 56 websites were used to score and analyze online ankle instability resources.

Each website was assessed for quality, accuracy, and readability by 3 orthopedic residents blinded to the search term used. We used quality as an indication of the credibility of each website. The quality was evaluated using a previously developed grading rubric containing 25 points pertaining to relevant anatomy, diagnosis, treatment, and complications of ankle sprains (Figure 2).^{11,12,15,18,23,29,30} Each reviewer evaluated the quality of each website independently by determining the extent to which each website fulfilled the stated criteria, and the resulting 3 scores were used to develop a mean score for each website.

Figure 2.

Ankle Sprain/Instability Quality Grading Sheet.

The accuracy of the websites was assessed using a scale from 1 to 4 based on previously validated methodology.^{11,12,15,18,23,29,30} A score of 1 represented that the reviewer agreed with only up to 25% of information available, 2 if the reviewer agreed with 26% to 50%, 3 if agreement is between 51% to 75%, and 4 if the reviewer agreed with >75%. The scores from the 3 reviewers were summed and averaged to create a final score. In addition, it was noted if the sites were authored by physicians or not. If the author was not mentioned, those websites were excluded.

The Flesch-Kincaid (F-K) scale was used to evaluate the readability of the websites as used in prior investigations.^{1,18,26,29,30} The F-K grade of any written content denotes the equivalent academic grade a person must have to be able to read and comprehend the material. Readability was defined as at or below the seventh-grade reading level recommended by the NIH.

Lastly, websites were evaluated for any commercial bias. A website was determined to have commercial bias if it contained advertisements for products or services. This was based on the assumption that the websites received payment for hosting advertisements.

Statistics

The statistical analysis was conducted with Microsoft Excel (Microsoft, Redmond, WA) and Minitab (State College, PA) statistical software. Data were described in terms of mean ± standard deviation (SD). Two-sample independent Student t test (for normally distributed data) were conducted to evaluate for differences in quality, accuracy, and readability with respect to the search term used (“ankle sprain” vs “ankle instability”), if the website was written by a physician, and if it was associated with commercial bias. Multivariable regression analysis was performed to detect if the website’s quality and/or accuracy correlated with any other factors.

Results

Half of the websites included in our study used the term “ankle sprain” whereas the other half used “ankle instability.” Twenty sites were identified using Google, 14 using Bing, 3 using Yahoo, whereas the remaining 19 appeared in multiple search engines. Sixty-nine percent of the websites (39/56) were written by physicians whereas only 21% (12/56) were associated with commercial bias.

The mean quality of the websites written above a seventh-grade level was statistically significantly higher than those at or below seventh grade (P = .01; 95% CI= –4.98 to –7.9). Websites written by physicians were shown to be of higher quality than those that were not (P = .005; 95% CI = –5.30 to –1.01). Finally, the mean quality of websites without commercial bias was statistically higher than the ones associated with commercial bias (P = .001; 95% CI= 1.49 to 5.14) (Table 1).

Table 1.

Summary of Quality Results of All Websites.

Category	Number	Mean ± SD	SE Mean	P Value
All websites	56	9 ± 3.90
Search term used
Ankle sprain	28	6.69 ± 2.88	0.54	<.05
Ankle instability	28	10.80 ± 3.73	0.70
Readability (F-K score)
≤ Seventh-grade level	6	6.17 ± 1.88	0.77	.011
> Seventh-grade level	50	9.05 ± 3.97	0.56
Commercial bias
Yes	12	6.14 ± 2.31	0.67	.001
No	44	9.45 ± 3.96	0.60
Written by physician
Yes	39	9.70 ± 3.67	0.59	.005
No	17	6.55 ± 3.59	0.87

Abbreviations: F-K, Flesch-Kincaid; SD, standard deviation; SE, standard of error.

There was no statistical significance in the accuracy of the websites regarding the search term used (P = .146). The mean accuracy of websites written by physicians was not statistically different from those not written by physicians (P = .055; 95% CI= –0.73 to 0.01). The mean accuracy of websites written above a seventh-grade level was statistically higher than websites written at or below a seventh-grade level (P = .01; 95% CI= –0.92 to –0.16). Similar to quality, the accuracy of websites without commercial bias was statistically higher than the websites with commercial bias (P = .003; 95% CI= 0.22 to 0.94) (Table 2).

Table 2.

Summary of Accuracy Results of All Websites.

Category	Number	Mean ± SD	SE Mean	P Value
All websites	56	3 ± 0.66
Search term used
Ankle sprain	28	2.90 ± 0.50	0.095	.146
Ankle instability	28	3.16 ± 0.78	0.15
Readability (F-K score)
≤seventh-grade level	6	2.55 ± 0.34	0.14	.010
>seventh-grade level	50	3.09 ± 0.67	0.097
Commercial bias
Yes	12	2.58 ± 0.49	0.14	.003
No	44	3.15 ± 0.66	0.10
Written by physician
Yes	39	3.14 ± 0.67	0.11	.055
No	17	2.78 ± 0.60	0.15

Abbreviations: F-K, Flesch-Kincaid; SD, standard deviation; SE, standard of error.

Only 6 websites (10.7%) included in this study were written at a seventh-grade level or below. The mean F-K score of all websites was 10.06. The mean F-K score for websites using the search term ankle instability was 11.17 and for websites using ankle sprain was 8.96 (P = .146). The mean F-K score of websites written by physicians was not significantly different from websites not written by physicians (9.99 versus 10.25; P = .67; 95% CI= -0.95 to 1.46). There was no difference in the mean F-K score between websites with and without commercial bias (P = .74; 95% CI= –1.84 to 1.33). The mean quality and accuracy of websites written at higher than a seventh-grade level were higher than those written at a seventh-grade level or below (Figures 3 and 4).

Figure 3.

Individual value plot graph showing the difference in means of quality of websites written at or below a seventh-grade level versus the ones written above a seventh-grade level.

Figure 4.

Individual value plot graph showing the difference in means of accuracy of websites written at or below a seventh-grade level versus the ones written above a seventh-grade level.

Multivariable regression analysis revealed the accuracy of websites was significantly associated with the F-K score (P = .002) and absence of commercial bias (P = .026).

Furthermore, the quality of websites was positively correlated with the F-K score (P = .042), use of the search term “ankle instability” (P = .023), websites written by physicians (P = .037), and absence of commercial bias (P = .043).

Discussion

Although the majority of patients now consider the Internet their primary source of obtaining health information,^5,14 there are concerns regarding the reliability of materials presented online. The current study identifies poor quality and accuracy of online information specifically pertaining to ankle sprains/instability, particularly in those written at or below the NIH-recommended seventh-grade reading level. This is consistent with previously published data, which suggested poor quality and higher reading levels of health-related information available on the Internet at a higher average reading level.^{3,7,18,19,22,25} Although the websites written by physicians had overall superior quality and accuracy, they also tended to be written at a higher reading level.

Studies have shown that patients who are able to comprehend health-related information are generally more compliant and have better outcomes.^4,10,17,28 With this in mind, the AOFAS developed the website http://footcaremd.org in an attempt to enhance the quality and accuracy of online information related to myriad of foot and ankle disorders. ¹⁶ The webpage reaches a large audience: 9.9 million page views were recorded in 2015, up from 1.1 million views in 2013. ²¹ Another study conducted a survey of national physician leaders in the field of foot and ankle surgery and found that about 50% of them directed patients to the Internet for information. ² A majority of them used the AOFAS website. The authors also recommended directing patients to the following sites: http://www.footcaremd.com, http://www.webmd.com, http://www.wheelessonline.com, and http://www.orthoinfo.aaos.org.

However, several studies have raised concerns regarding the high readability level of the content of those resources. In 2015, one study evaluated the online education material presented on the American Academy of Orthopaedic Surgeons (AAOS) website, considered to be one of the largest providers of musculoskeletal information. ¹³ The study found that the 260 sites assessed had a mean readability grade of 9.2. In fact, 97% of the sites studied had a score above a sixth-grade level. ¹³ Another study reported on the readability of the materials available on the AOFAS website. The average reading level of such content was 8.3. ⁶ Another study found similar results in the content of the AAOS, MedlinePlus, and 11 academic centers’ patient education websites. When sentences were shortened to less than 15 words (as recommended by the NIH to improve readability of online materials), the authors reported improvement in the readability of the articles by an average of 1.41 grade levels. ²⁷

The current study discovered only 10% of the websites were written at or below seventh-grade level. Using a similar study design, one study reported on the quality, accuracy, and readability of online websites with respect to hallux valgus. ²⁹ They used eighth-grade level as their desired cutoff for readability and found that nearly 80% of the websites were written at a level greater than eighth-grade. The authors found the websites to have a mean F-K score of 10.6, consistent with the results reported here. However, contrary to their findings, websites written by physicians in the current study had lower F-K scores than those not written by physicians. This may suggest that physicians are more aware of the previously reported high reading level of online websites and that physicians are making an effort to accommodate to the NIH recommendations. However, the overall readability is still not at the desired level, which could place patients at disadvantage when it comes to seeking health-related information through the Internet.

There was no significant difference in quality and accuracy of websites discovered using the term ankle sprain compared to ankle instability. Interestingly, another study showed that websites found using the colloquial term bunion was associated with significantly higher quality websites when compared to websites found using the medical term hallux valgus. ²⁹ This may be explained by instability being more of a lay term than hallux valgus. When discussing diagnoses with patients, it is important to explore the medical as well as colloquial terminology that they may come across during their web searches.

Websites associated with commercial gains were found to have lower quality and accuracy scores. This can be problematic because patients might not be able to differentiate between informational and commercial resources. Additionally, the presence of images of the products with or without supplemental videos may be more attractive to the general public. This should serve as an incentive for physicians to guide their patients to resources with the most accurate information and to be wary of websites trying to sell a product.

Many patients will not seek a doctor’s advice for an ankle sprain, instead opting to manage their injury themselves using information online. Most of these patients will have satisfactory outcomes. However, a small subset of patients will develop persistent instability, turning a simple sprain into a problem that may warrant surgical intervention. Therefore, the quality and accuracy of online information pertaining to ankle sprains is of utmost importance to ensure patients are getting the correct treatments to provide them with the best outcomes.

The current study highlights the poor quality and accuracy of online information related to ankle sprains, especially those with commercial bias. Furthermore, while websites written by or under supervision of physicians were found of superior quality, a majority of sites were found to have an unacceptably high reading level. As the importance of health literacy becomes more apparent, it is critical to remember the physician’s role in guiding patients to high-quality and accurate resources written at an appropriate level.