Abstract
Background:
The internet is a valuable tool, but concerns exist regarding the quality and accuracy of medical information available online.
Purpose:
To evaluate the source and quality of information on the internet relating to anterolateral ligament reconstruction (ALLR) compared with anterior cruciate ligament reconstruction (ACLR).
Study Design:
Cross-sectional study.
Methods:
A questionnaire was administered to 50 ACLR patients in Australia to determine their use of the internet to research their operation and their familiarity with the anterolateral ligament (ALL) of the knee. The most common search terms were determined, and the first 70 websites returned by the 5 most popular search engines were used to assess the quality of information about ACLR and ALLR. Each site was categorized by type and was assessed for quality and validity using the DISCERN score, the Journal of the American Medical Association (JAMA) benchmark criteria, and a novel specific content score for each procedure. The presence of the Health on the Net Code (HONcode) seal was also recorded.
Results:
The majority (84%) of ACLR patients used the internet to research their operation. The quality of information available for ALLR was significantly inferior to that for ACLR according to the DISCERN score (37.3 ± 3.4 vs 54.4 ± 4.6; P < .0001) and specific content score (5.3 ± 1.3 vs 11.0 ± 1.5; P < .0001). ACLR websites were predominantly physician produced, while the majority of ALLR websites were academic. In contrast to ACLR websites, the majority of ALLR websites did not provide information on the indication for treatment or potential complications. ALLR websites scored better on the JAMA benchmark criteria due to the predominance of academic websites. A greater proportion of ACLR websites (14.6%) versus ALLR websites (2.5%) provided an HONcode seal. Correlation was demonstrated between the DISCERN score and specific content scores for both ACLR and ALLR but not with JAMA benchmark criteria. The specific content score had high reliability for both ACLR and ALLR.
Conclusion:
The majority of patients undergoing ACLR in Australia used the internet to research the procedure. The quality of information on the internet relating to ALLR was significantly inferior to information about ACLR. Most ALLR websites failed to include crucial information about the indication or options for treatment, prognosis, and potential complications. Surgeons should be aware of the information to which their patients are exposed through the internet and should be proactive in directing patients to appropriate websites.
The internet is an extraordinarily valuable tool that has transformed many aspects of modern life. The capability of the internet to disseminate diverse data across populations and nations presents unique opportunities but also poses significant challenges in terms of regulation of information, particularly in the health care industry. Traditionally, health care professionals were the primary source of information for health care consumers, but in recent times the internet, social media, and print journalism have assumed an increasingly greater role. 1,8,14 Although it is important for health care consumers to be informed, a number of studies have demonstrated that many health care websites contain inaccurate or misleading information. 4,10,16,21,24 Another concern relates to regulation, as the pharmaceutical and medical devices industries are increasingly using the internet to market their products, not just to health care professionals but also directly to the general public. 19
In the setting of knee surgery, anterior cruciate ligament (ACL) injury is one of the most commonly treated conditions. 13 Previous studies have demonstrated that the internet is an important source for patients seeking information about ACL reconstruction (ACLR), although the quality of information has been reported to be variable and often commercially driven. 4,5,11 The increasing pervasiveness of the internet in the field of orthopaedics is probably best illustrated by the case example of the anterolateral ligament (ALL). Following the publication of an anatomic study by Claes et al, 7 who reported the presence of a distinct ligamentous structure on the anterolateral aspect of the knee, there was widespread interest beyond the traditional scientific publications. The findings of the study were reported on website blogs 18 and taken up by mainstream print media within weeks of release of the scientific publication. 9 Surgical techniques quickly emerged to perform ALL reconstruction (ALLR) in combination with ACLR. 22,23 The proposed benefit of performing ALLR in combination with ACLR is to provide greater rotational control of the knee than provided by ACLR alone, which theoretically has the potential to reduce graft failure. 23 Ironically, recent criticism has emerged—through the same print media that heralded the “discovery” of the ALL—that the development of ALLR has been too rapid and that proven evidence of the efficacy of surgery is lacking; the cynicism of the New York Times article is illustrated by the title: “Surgery Fixes a Ligament (If It Exists): Does It Fix the Knee?” 15
The aim of this study was to assess the source and quality of information on the internet related to ALLR compared with ACLR through use of recognized scoring systems, identification of quality markers, and pathology-specific content scores. The hypothesis was that the quality of information available would be inferior for ALLR in contrast to ACLR. A secondary aim was to survey a cohort of patients who had undergone ACLR to determine the prevalence of internet use preoperatively to research the surgery and to assess the patients’ awareness of the ALL.
Methods
Prior to the commencement of a web search, a questionnaire was administered to 50 consecutive postoperative ACLR patients in Australia (Table 1). The patient cohort consisted of 33 men and 17 women with a mean age of 25 years (range, 13-51 years). The questionnaire was designed to determine whether the patients had used the internet preoperatively to research their operation, which search terms and which search engines the patients had used, and whether they were familiar with the ALL of the knee.
Patient Questionnaire
From the questionnaire it was established that the term ACL reconstruction was the most commonly used search term for ACLR. This search term was then used to carry out an internet search. Only a minority of patients were familiar with the ALL or any relevant abbreviations. Therefore, the full term anterolateral ligament reconstruction was used for the internet search.
Consistent with previous studies, 4 the 5 most popular search engines—as determined by total market share—were used by patients: Google (65%), Bing (16%), Yahoo! (8%), Lycos (<1%), and AOL (<1%). 17 To reflect the disproportionate use of the various available search engines, and in keeping with our questionnaire findings, an analysis was performed on the first 30 of approximately of 530,000 websites returned by Google (www.google.com) and 10 each from AOL (www.aol.com), Bing (www.bing.com), Lycos (www.lycos.com), and Yahoo! (yahoo.com). Duplicate websites and those that were inaccessible were excluded from the review (Table 2). Searches were performed on January 25, 2016, and were carried out in Australia.
List of Websites Returned for Anterior Cruciate Ligament and Anterolateral Ligament Reconstruction as Sorted by Search Engine
The first part of the analysis involved separating the websites into the following categories: academic, physician, nonphysician (allied health professionals such as physical therapists, occupational therapists, and alternative medicine providers), media, commercial, social media, and advertisement. These classifications were based on those described in previous work. 4 Academic websites were defined as those with an affiliation to a university or medical society. Physician websites included professional sites for individual physicians or group practices not affiliated with an academic institution. Media websites were nonmedical news-oriented sites. Websites were considered commercial if they included products for sale. Social media included websites such as YouTube, Facebook, and Twitter. Finally, advertisement was any website that displayed advertisements.
Websites were assessed for quality and validity by use of the DISCERN score, 6 and a pathology-specific content score was used for both ACLR and ALLR to assess the quality of information in each search. In addition, the Journal of the American Medical Association (JAMA) benchmark criteria 21 and the presence or absence of a Health on the Net code (HONcode) certification were noted. 3
The DISCERN tool was designed to allow consumers and information providers to determine the quality of health information for treatment choices. 6 The tool has 16 questions; the first 8 questions relate to the reliability of the publication, and 7 questions address specific details of the information about treatment choices. The final question assesses the overall quality of the website. Scores of 60 or higher represent websites that are useful and appropriate sources of information, while scores of 30 or lower are indicative of websites with serious shortcomings that are not appropriate sources of information.
To ascertain the information value of each website specifically related to the surgical procedure in question, a pathology-specific content score was developed for both ACLR and ALLR. The ACLR-specific content score was modified from a similar tool developed and published by Bruce-Brand et al 4 (Table 3). A similar tool using the same format was designed to assess the value of information for ALLR (Table 4). One point is allocated for the presence of predefined terms that relate to general characteristics of the condition, the prognosis, options for treatment, and complications, resulting in a score between 0 and 20. This was done in consultation with a fellowship-trained orthopaedic sports surgeon (B.M.D.) and by referencing peer-reviewed literature. A higher score indicates a more informative website.
Scoring for Web Content Specific to ACL Reconstruction a
a One point was allocated for each item mentioned on the website, for a total of 20 points. ACL, anterior cruciate ligament.
Scoring for Web Content Specific to ALL Reconstruction a
a One point was allocated for each item mentioned on the website, for a total of 20 points. ALL, Anterolateral ligament.
The JAMA benchmark criteria, originally published by Silberg et al, 21 consist of 4 categories to determine whether a source of information is credible: authorship, attribution, disclosure, and currency. Authorship requires the authors and contributors to provide their affiliations and credentials. Attribution relates to the listing of references and sources as well as all relevant copyright information. Disclosure is assessed on whether website “ownership” is fully disclosed as well as any sponsorship, advertising, commercial funding, or conflicts of interest. Currency requires that the website provide the dates when the content was posted and updated. One point is allocated for each of the criteria that are met, with a maximum score of 4; a score of 4 indicates a credible source, while a score of zero indicates that the source of information is questionable.
The HONcode seal is used as a mark of accreditation for websites that comply with listed standards and publish transparent health-related information. 3 The presence or absence of this seal was assessed for each website.
Independent t tests were used to compare means between ACLR and ALLR. The Pearson correlation coefficient was used to assess convergent validity between scales, and the intraclass correlation coefficient (ICC) was used to measure interrater reliability for the new pathology-specific content scores. Statistical analysis was performed by use of R 3.3.2/RStudio 1.0.136. The level of significance was set at 5%.
Results
Patient Questionnaire
Overall, 84% of the patients (42/50) used the internet to research their procedure preoperatively. Of these, 40 (95%) used Google as the primary search engine (Figure 1). Thirty-two percent of patients (16/50) had prior knowledge of the ALL. The knowledge was from variable sources, most frequently from online information (7/16; 44%). The other sources were word of mouth (4/16), medical professionals (2/16), and academic studies (3/16).
Frequency of websites used for anterior cruciate ligament reconstruction searches.
Website Analysis
A total of 41 unique websites were analyzed for ACLR, and 40 unique websites were analyzed for ALLR (see Table 2). Figure 2 shows the breakdown according to website type. The majority of ACLR websites were physician produced (n = 23; 56%). In contrast, the majority of ALLR websites were academic (n = 18; 45%), but physician-produced websites were also numerous (n = 11; 27.5%). Figure 3 illustrates the origin of the websites from each of the searches. The majority of websites returned for ACLR were Australian (n = 23; 56%), whereas the majority of websites analyzed for ALLR were international sites (n = 37; 92.5%).
Frequency of website type returned for anterior cruciate ligament (ACL) and anterolateral ligament (ALL) reconstruction searches.
Origin of websites returned for anterior cruciate ligament (ACL) and anterolateral ligament (ALL) reconstruction searches.
DISCERN Scores
The mean DISCERN score for ACLR websites was 54.37 ± 4.6, whereas the score for ALLR websites was 37.3 ± 3.4 (P < .0001). Eighteen ACLR websites scored 60 or higher on the DISCERN tool, representing websites that are useful and appropriate sources of information. Only 2 ALLR websites scored 60 or higher. Four ACLR websites scored 30 or lower on the DISCERN tool, representing websites with serious shortcomings that are not appropriate sources of information, compared with 8 ALLR websites. Figure 4 demonstrates DISCERN scores by website type.
DISCERN scores by website type returned for anterior cruciate ligament (ACL) and anterolateral ligament (ALL) reconstruction searches.
Pathology-Specific Content Score
Figure 5 demonstrates the distribution of specific content scores according to website type for ACLR and ALLR. The highest score for an ACLR website was 18 of 20, which was recorded in both academic and physician-produced sites. The highest score for an ALLR website was 17 of 20, which was for a commercially produced website.
Pathology-specific content score by website type returned for anterior cruciate ligament (ACL) and anterolateral ligament (ALL) reconstruction searches.
Regarding ACLR websites, the greatest deficiency in terms of content was information on complications, specifically the potential to develop osteoarthritis, which was mentioned on only 4 websites. ALLR websites demonstrated a more widespread deficiency in content; only 9 websites discussed the mechanism of injury, 13 websites mentioned the requirement for treatment, and 8 websites listed a complication of any type related to the surgery.
Regarding pathology-specific content scores, ACLR websites had a mean score of 11.05 ± 1.5 (out of 20), which was significantly higher than the mean score for ALLR websites—5.3 ± 1.3 (P < .0001). Figure 6 shows the frequency distributions for pathology-specific content scores among website types. A notable difference was found in the distribution of specific content scores. For ACLR websites, 14 was the most frequent score, while for the ALLR websites the most frequent score was zero.
The frequency distribution of pathology-specific content scores among websites. (A) Anterior cruciate ligament reconstruction (ACLR)–specific content score. (B) Anterolateral ligament reconstruction (ALLR)–specific content score.
JAMA Benchmark Criteria
The mean JAMA benchmark criteria score for ACLR websites was 1.9 ± 0.4, and that for ALLR websites was 2.8 ± 0.4 (P = .007) (Figure 7). Four ACLR sites had maximal scores compared with 13 ALLR sites.
JAMA score by website type returned for anterior cruciate ligament (ACL) and anterolateral ligament (ALL) reconstruction searches.
Convergent Validity and Reliability
The ACLR-specific content scores demonstrated close correlation with the DISCERN scores (ICC, 0.58; 95% CI, –0.33 to 0.75; P < .0001) but no correlation with the JAMA benchmark criteria (ICC, 0.13; 95% CI, –0.19 to 0.42; P = .43). The ACLR-specific content scores demonstrated high intraclass correlation, reflecting excellent internal consistency of the new tool (ICC, 0.997; 95% CI, 0.994-0.998; P < .0001).
The ALLR-specific content score demonstrated correlation with the DISCERN scores (ICC, 0.49; 95% CI, 0.21-0.70; P = .0013) but no correlation with the JAMA benchmark criteria (ICC, 0.26; 95% CI, –0.06 to 0.53; P = .1085). We found 2 outliers between the DISCERN score and the ALLR-specific content score, scoring highly on the DISCERN score and poorly on the ALLR-specific content score. If these outliers are excluded, the intraclass correlation is 0.72 (95% CI, 0.52-0.85; P < .0001). The ALLR-specific content score demonstrated a high intraclass correlation of 0.877 (95% CI, 0.780-0.933; P < .001).
HONcode Certification
Six (14.6%) of the ACLR websites were HONcode certified compared with 1 (2.5%) of the ALLR websites. ACLR websites with HONcode certification demonstrated a mean DISCERN score of 72.50 (95% CI, 69.87-75.13), whereas websites without HONcode certification had a mean score of 51.23 (95% CI, 46.42-56.04; P < .0001). ACLR websites with HONcode certification demonstrated a mean JAMA score of 3.67 (95% CI, 3.13-4.21), whereas websites without HONcode certification had a mean score of 1.60 (95% CI, 1.28-1.92; P < .0001). No significant difference was found in ACLR-specific content scores between websites with HONcode certification and websites without (12.75; 95% CI, 10.77-14.73 vs 10.73; 95% CI, 8.92-12.54; P = .1006). One ALLR website contained HONcode certification.
Discussion
The main finding of this study was that the quality of information found on the internet by Australian users was significantly better for ACLR compared with ALLR over a variety of website types.
The vast majority of patients undergoing ACLR who were surveyed used the internet preoperatively to research their operation. The primary source of internet information on ACLR was from physician-produced websites, while information on ALLR was predominantly from academic websites. Despite this, only 2 of 40 websites related to ALLR were considered to be a useful and appropriate source of information, while 8 had serious shortcomings according to the DISCERN quality criteria for consumer health information. Interestingly, a marked contrast was noted in terms of the countries where the ACLR and ALLR websites were generated. The overwhelming majority of ALLR websites were international (92.5%), coming from Europe and North America, while the ACLR websites were predominantly from Australia (56%). This perhaps reflects that ALLR is not an established practice but an emerging procedure, which is being championed mainly in Europe and North America, whereas ACLR is more common in Australia and is the topic of a greater majority of local websites.
An assessment of the informational value using a pathology-specific content score for each procedure revealed that compared with ACLR websites, ALLR websites were notably lacking in information related to the indications for treatment and potential complications of the procedure. Interestingly, industry-sponsored (commercial) websites had the highest ALLR-specific content score. The content pathology-specific score used was found to be reliable and correlated well with the DISCERN scores for both ACLR and ALLR. These findings raise concerns regarding the quality of information on the internet related to ALLR.
The internet has created a unique conduit through which new research, which previously remained in the domain of academic journals and scientific meetings until its validity had been debated, is rapidly released into mainstream circulation. This poses a distinct risk that patients are exposed to incomplete information and the promise of emerging treatments that have not undergone rigorous assessment. The ubiquity of this information is borne out in the current study, where an analysis of 50 postoperative ACLR patients revealed that 32% had prior knowledge of the ALL, most of whom had found this information through the internet.
A comprehensive array of analytic methods was used to assess the quality of information on the internet for the chosen surgical techniques. This was done to account for the fact that no single assessment tool was designed to cover all areas, such as the credibility of the websites, the validity and quality of the information, and the relevance of the specific content. Interestingly, each of the analysis methods yielded unique findings that, when considered together, provided a detailed assessment of the website value. Although both ACLR and ALLR websites were considered credible sources of information according to JAMA benchmark criteria, we found a substantial gulf in quality of information provided on the websites, as assessed by DISCERN and specific content scoring systems as well as the presence of an HONcode certification. The use of ACLR as a benchmark proved appropriate in this study, as the information available on the ACLR websites was generally of high quality. In contrast, the pathology-specific content information related to ALLR contained a disproportionate amount of information on the anatomic features and function of the ALL without adequately informing the patient about the indications for treatment, treatment options other than surgery, and, critically, the potential complications of surgical reconstruction.
This lack of credible material is an area of concern and has the potential to lead to cyberchondria, which is defined as an undue level of stress or anxiety brought about through a patient’s exploration of health symptoms via the internet. 2 It has been suggested that this misinformation can affect the patient-doctor relationship, as the doctor may be required to dispel mistruths and re-educate the patient as part of a routine consultation, which is not always an easy task. 4
The internet is used not only as a source of information but also as a platform for marketing. To this end, in a very competitive marketplace, physicians are increasingly using personalized websites to promote their practices and the procedures they offer. 20 Our study clearly demonstrated this trend in the percentage of physician-produced websites providing information on ACLR (56%), which is perhaps one of the most competitive areas in orthopaedics. In an attempt to gain market share, surgeons often seek to offer new, state-of-the-art procedures with the potential for improved outcomes. 20 This was seen in our study, where the second-largest internet source of information on ALLR was from physician-produced websites (27.5%).
An assessment of the quality of information on physician-produced sites using a specific content tool revealed that although the information for ACLR was consistently of good quality, the same could not be said for ALLR. One of the major criticisms of the recent interest in ALLR is that it was potentially industry driven, without rigorous clinical outcomes or comprehensive follow-up. 15 The current study does not entirely support this claim, as industry-sponsored websites accounted for only 2 of 40 websites. Furthermore, the highest quality of information on ALLR was found on one of these industry-sponsored websites.
The poor quality of information from the most commonly accessed source of information for ALLR—the academic sites—is a cause for concern. This finding is in contrast to previous studies, which determined that academic websites were typically sources of high-quality information. 4,24 An explanation for this shortfall in quality could relate to perceived urgency to remain current and provide readable information. Studies have demonstrated that patients are more inclined to seek out information that is simply displayed and easy to understand. 12 The reader demographic should also be considered, which in the setting of ACLR is typically young patients. Küçükdurmaz et al 12 investigated the readability of internet health information on femoroacetabular impingement by using a formula that calculated sentence length and syllables per word. Their study concluded that the websites intended to attract patients searching for information are providing a highly accessible, readable information source but do not appear to be quite so rigorous when it comes to inclusion of scientific literature. The findings of the current study are consistent with those of Küçükdurmaz et al 12 and suggest that “dumbing down” information for a lay audience can result in omission of many critical elements.
In the current study, the specific content scores were found to be a valuable addition to the DISCERN score to provide a more specific analysis of the quality of website material. Although the DISCERN score is well recognized and is useful in determining the reliability and quality of the content being presented, the score does not entirely address the relevance of the information. Exploring the correlation between the 2 scoring systems highlighted this point; 2 outliers were clearly identified that scored highly on the DISCERN but scored very poorly on the ALLR-specific content score. The reason for this was that only 1 of 16 questions in the DISCERN score refers to the general relevance of the information being assessed. The websites in question contained all the appropriate headings, but the content was completely unrelated to ALLR.
ALLR is an emerging procedure and as such lacks long-term follow-up that may affect the pathology-specific content score. But one would expect that this very fact should be mentioned by a source that is providing pathology-specific content outlining the evidence for treatment or expectations from surgery. While other lateral extra-articular procedures have been described, they were not included in this study as a search term because they do not purport to reconstruct the ALL anatomically or otherwise but were used originally to control anterolateral instability. Whether the ALL alone is integral in controlling anterolateral instability remains a contentious issue. Furthermore, many of these lateral extra-articular procedures were in use long before the ALL was initially described and have not been the topic of “viral trending” in recent times or the subject of the same promotion.
Limitations
We acknowledge that this study has limitations. The assessment of quality was performed at a single time point and did not take into consideration temporal trends that exist on the internet. Also, this study was conducted in a single country; therefore, national websites may have achieved a greater representation than if the study had been performed in a different country. The specific websites related to ACLR are likely to have differed to a greater extent between countries given that the majority were Australian sites (56%). Although the search term ACL reconstruction was chosen based on the results of the patient questionnaire, we chose the search term anterolateral ligament reconstruction because the patients surveyed lacked a consistent level of knowledge of the structure. A search for ALL reconstruction would likely have resulted in a different website yield because of the ubiquity of the acronym ALL. Finally, the readability of the websites was not determined.
Conclusion
This study found that the majority of patients undergoing ACLR in Australia use the internet to research the procedure. The quality of information on the internet relating to ALLR was significantly inferior to the information regarding ACLR. The majority of ALLR websites failed to include crucial information about the indications for surgery, treatment options, prognosis, and potential complications. Given the recent viral trends of dubious information being disseminated through the internet, orthopaedic surgeons should be aware of the information to which their patients are exposed and should be proactive in directing patients to appropriate websites.
Footnotes
The authors declared that they have no conflicts of interest in the authorship and publication of this contribution.
Ethical approval was not sought for the present study.