Abstract
Increasing numbers of patients are using the Internet for medical information. Internet sources include Email, Medline, electronic journals, discussion groups, and World Wide Web (WWW) sites. Medical information is one of the most common source of queries on the Web, although the exact number of medical websites is uncertain [1].
This explosion in the use of the Internet and the number of sites can lead to problems in the quantity and quality of information [2]. There are no limitations on website accessibility or the quality of information given. Sources of health information on the Web include private companies offering medical products or information, individual patients, health professionals, health support groups for patients, professional organizations, non-government organizations, academic institutes and government agencies. The quality of this information is, however, extremely variable [3–9]. Pages often do not name authors, and there are no publishing regulations, evaluations or peer review concerning the content and accuracy of presented information.
Attempts have been made to define what constitutes a good web page [10–13]. These include stating the authorship of the page with credentials, giving references, and openly stating any sponsorship information. A balanced reporting of the evidence about a treatment is also important. The Health on the Net Foundation has issued a code of conduct for medical and health sites covering much the same area [14]. However, this is only an advisory body and cannot enforce minimum standards [14].
In the area of mental health, a study of 21 popular sites on depression found that the quality of sites was poor, lacked balance and did not provide information on treatment [15, 16].
This project aimed to assess the quality and focus of information concerning chronic fatigue syndrome (CFS) available to patients on the Internet using a simple search strategy. The study identified web sites that typical users would look at, and assessed their quality against explicit criteria using a standard pro forma. The Internet search was carried out using nine common search engines to ensure comprehensive coverage of available sites. This is because engines use different search strategies to rank identified sites, and these are not made explicit to users. Criteria include ‘net appeal’ (the level of innovation used on the page such as graphics, sounds and colours) and ‘site hits’ (the number of visitors to a site over a specified period). In addition, it is also possible to pay to have a site ranked more highly on some search engines.
Methods
Selection of website
The nine search engines available on the home page of Netscape and Microsoft Explorer, two popular Internet browsers, were used to assess the number of Internet addresses or universal resource locaters (URL) on ‘chronic fatigue syndrome’. This search phrase was chosen as a term a patient or family member might use to obtain more information. A simple search method was chosen, and performed without further refinement of initial search results, to produce a list of web sites similar to one generated by a person who has limited medical, Internet or computer knowledge. Given the potential number of sites, the study used the methodology of Soot et al. to select those sites that patients were most likely to find [7]. The 25 most highly ranked web pages identified by each of the nine search engines underwent further review. When a web page was inaccessible, the next page on the list was reviewed. This was done up to a maximum of 25 sites for each search engine. The search of the WWW was undertaken over 2 weeks in September 2000.
Inclusion and exclusion criteria
A page was defined as relevant if it was a web site containing information on the treatment of CFS. This excluded discussion groups, message boards, pages that only mentioned the phrase ‘chronic fatigue syndrome’ but gave no further information, and pages that were lists of links to other pages. A random sample was not taken because search engines attempt to order pages in terms relevant to the searcher, those more closely matched being nearer the top of the list. The search was completed within a period of two weeks, as web pages can change very rapidly.
Classification of content
Consistency with evidence-based practice: Pages were evaluated against three recent systematic reviews on CFS [17–19]. These were the most up to date reviews identified from a search of Medline, the Cochrane Database of systematic reviews, and the Internet. All made similar recommendations [17–19]. The content of pages was analysed in terms of agreement or disagreement with the three systematic reviews on the usefulness of treatments for CFS, or if a treatment was not mentioned at all.
Balance of content: Using the framework of Soot et al. [7], the treatments recommended by the top 25 pages for each search engine were also categorized as follows: ‘conventional’, ‘unconventional’, ‘misleading’ and ‘unrelated’. ‘Conventional’ therapy was defined as standard management derived from good quality randomized control trials [7]. ‘Unconventional’ recommendations were defined as those that advocated experimental therapy as the sole form of treatment, and failed to mention any of the more conventional therapeutic options [7]. ‘Misleading’ therapeutic recommendations were those that emphasized experimental therapy and did not give equal time to, or downplayed, conventional therapeutic methods [7]. ‘Unrelated’ information included meeting information, departmental information and textbook advertisements. In this case, the next page on the list was reviewed for patient-orientated content so that 25 sites were included in the review. Any differences between search engines in the number of sites recommending conventional interventions, against those with unconventional or misleading content, were analysed using the chi-squared statistic.
Presence of other quality indicators
The presence of other quality indicators was assessed against criteria derived from peer-reviewed journals [10–13]. These included a named author, presence of references, a declaration of any potential conflict of interest, the need to clarify information with an appropriate health professional, and the absence of inaccuracies. Differences between engines in the number of sites that met these criteria were analysed using the chi-squared statistic.
Classification of website information source
The reference sources for information in reviewed Web sites were analysed using the methodology of Soot et al. [7]. References were divided into ‘conventional’, ‘anecdotal’, ‘none’ or ‘unable to be referenced’. ‘Conventional’ references were those that cited literature that could be examined and validated, such as journal articles and textbooks. ‘Anecdotal’ references were authors’ own experiences or beliefs on the diagnosis and treatment of CFS. ‘None’ referred to those sites that did not state the source of their information. Web pages were considered ‘unable to be referenced’, if they presented information that could not be referenced such as meeting information, text book advertisements, or departmental biographies. Any differences between search engines in the number of sites with conventional references, against those with no references or only anecdotal references, were analysed using the chi-squared statistic.
Results
Nine search engines were used in the study: Looksmart, Google, Lycos, Excite, Microsoft Network (MSN), Netscape, Hotbot and GoTo.com. Six search engines gave summaries of the total number of sites matching the search criteria (Looksmart, Google, Lycos, Excite and Netscape). The mean number of identified sites was 89,181, but this ranged from 220 sites in the case of Looksmart to 202 000 in Google. In comparison with three other topics that have been the subject of similar reviews [5, 7, 15], means of 1.3 million sites were identified for depression, 58 271 for vascular surgery and 17 002 for measles, mumps and rubella (MMR) vaccine.
Of these, 225 websites covering the management of CFS were reviewed during the study period using the methodology of Soot et al. [7]. A further 15 sites (6.3%) were inaccessible.
Consistency of content with evidence-based medicine guidelines
Recommended interventions varied widely (Table 1). The agreement between web sites and the systematic reviews was poor (Table 2). The greatest agreement was on recommendations for graded exercise followed by the avoidance of prolonged rest (Table 2). Nearly 70% of sites identified by Google recommended graded exercise (Table 2), and at least 50% of web sites identified by Looksmart, Lycos and Excite gave similar advice. Overall agreement for other interventions ranged from 13.5 to 28.5% (Table 2). Depending on the particular intervention, agreement was as low as 4% (Table 2). Google consistently identified the greatest number of sites that agreed with evidencebased practice. However, this did not reach statistical significance except in the case of antidepressants (Table 2).
Interventions recommended on reviewed web sites
Agreement with evidence-based practice on usefulness of treatment, classification of site contents and information sources, and presence of other quality indicators — lowest and highest number of sites identified by any engine and overall totals
Balance of content
Only a minority of sites (17.7%) gave appropriate balance to the interventions that were recommended, giving appropriate emphasis to conventional therapies (Table 2). In most, the content was unconventional with unproven interventions being given equal weight to those with established efficacy. A significant minority were misleading, and only recommended unproven interventions. In the case of one search engine (Hotbot) over two thirds of sites identified recommended only unproven interventions (n = 17). By contrast, Google identified the greatest number of sites giving appropriate emphasis to conventional therapies, this reaching statistical significance (Table 2).
Other quality indicators
Although most pages had a named author (64%), only a minority contained a declaration of interest or advice to seek appropriate help (Table 2).
Barely one fifth of pages were free of inaccurate statements (Table 2). Examples included anecdotal reports that CFS could be infective and passed on to pets or other members of a patient's family [20]. Another web site warned that CFS could be spread by blood transfusions [21], while others claimed that CFS is linked to AIDS [22], polio vaccine [23, 24], a ‘stealth virus’ [25] and even to the United States biological warfare programme [24]. Allegedly, two specific kinds of viruses were developed for warfare, one fatal (AIDS) and the other disabling (CFS).
Google identified the greatest number of sites that had a declaration of interest, advised visitors to seek appropriate help and avoided inaccurate statements, this reaching statistical significance (Table 2).
Classification of website information sources
Only a small minority of sites gave conventional referencing for their information sources. The vast majority relied on anecdotal references, the balance giving no references at all. Google identified the greatest number of sites that gave conventional references against those with no references or only anecdotal references, this figure reaching statistical significance (Table 2).
Quality sites
Several search engines, including Lycos, Netscape and Hotbot gave one or two recommended sites, the most common being Web MD [26]. These sites were more likely to meet minimum criteria for balance and quality of content. Snap had a much longer list of recommended sites but the 15 pages reviewed did not differ from Snap's nonrecommended sites in terms of balance or quality of content. The following sites adhered closely to recommendations of evidence-based reviews of the subject, provided balanced content and used conventional references.
http://webmd.lycos.com/topic__summary/1654
http://www.noah-health.org/english/illness/neuro/cfs.html
http://www.medicineau.net.au/clinical/internalmedicine/internalmed1503.html
http://www.cdc.gov/ncidod/diseases/cfs/index.htm
http://www.niaid.nih.gov/factsheets/cfs.htm
Discussion
With the arrival of the Internet, anyone with a computer and modem can disseminate information widely, and at times indiscriminately. Search engines will find web pages from both reputable sources such as the Centre for Disease Control, alongside other sites containing only anecdotal experience or recommending specific forms of intervention to the exclusion of all others.
Because of the speed of change on the Internet [27], the assessment of sites was carried out over 2 weeks. Sites were evaluated by one assessor against explicit evidence-based guidelines and criteria for balance and quality of content. This ensured that the results were reliable and collected within the 2-week period. However, the use of only one assessor may have introduced bias. Abbott has suggested that pages identified in a search are downloaded onto a disc, and the subsequent analysis carried out on the downloaded copies [5]. This would allow more time and the validation of a single rater's assessments by one or more other assessors.
The search strategy used in this paper is only one of a number of different strategies including the use of ‘metasearch’ search engines. Nevertheless, given the number of sites reviewed in this paper, it is unlikely that the findings would have been very different had a different strategy been used.
Balanced content consistent with evidence-based practice
Websites are often included on a search engine index on the basis of ‘net appeal’ referring to the level of innovation used on the page such as graphics, sounds and colours. It does not depend on assessment of the quality or content.
In line with other work, the poor quality and lack of balance in the quality of information on the Web shown in this study is of concern. While doctors practice evidencebased medicine, patients are reading material that shows little agreement with scientific literature. Even in areas where sites were mostly likely to give accurate advice such as the benefit of graded exercise and harmfulness of prolonged rest, overall agreement with systematic reviews was barely 50%. In other areas such as the usefulness of cognitive–behavioural therapy, agreement with the scientific literature occurred in only a quarter of cases.
Eighty per cent of sites contained one or more inaccurate statements. The figure for the percentage of websites with misleading information or which advocated unconventional therapy is higher in this study than previous work on vascular surgery [7], but is similar to a study of MMR vaccine [5]. When considering the quality of information on the Web, there may be differences in accuracy depending on the medical condition of interest. Medical management will vary in controversy and these differences will be reflected on the Internet. In terms of differences between the search engines, there were little differences between the quality of sites identified.
Other quality indicators
In terms of other quality indicators, most pages had a named author (64%). However, only a minority contained a declaration of interest or advice to seek appropriate help. Only one fifth of sites reviewed in this study used conventional sources of information. Even the presence of references to peer reviewed journals does not necessarily correlate with quality. A previous study identified a page citing any papers from peer reviewed journals which either did not support the claims of the page's authors, or which had been selectively quoted out of context [5].
These results are consistent with previous findings that the reliability, accessibility and completeness of information and advice on the Internet is extremely variable, ranging from useful to dangerous. The Federal Trade Commission and the US Science Panel on Interactive Health Communication have repeatedly warned that much information on the Web is misleading or positively harmful [3, 6].
Google tended to identify the greatest number of sites that gave appropriate emphasis to conventional therapies, were consistent with evidence-based practice, contained a declaration of interest, and gave conventional references. Google ratings are based on the quality of the site as well as complex algorithms for counting ‘hits’ to a site. Unlike some other engines, ranks cannot be purchased or commercially altered [28].
Implications
Clearly, doctors need to be aware of the sort of information that the patients are accessing on the Internet and be in a position to offer guidance on what patients are reading. There are a number of ways of filtering information [1]. If quality control at the website is not possible or desirable, alternatives are ‘downstream filtering’ by Internet users themselves, or the award of a seal of approval by an intermediary (‘upstream filtering’). However, these do not always measure key areas and only give users a remote idea of the reliability of the web site. Filtering by the Internet user (manual downstream filtering) requires the dissemination of criteria to help Internet users select information on their own. Organizations such as medical colleges could provide selection criteria for Internet users to identify Internet sites meeting evidence-based and ethical guidelines. In addition, there are generic checklists to help patients assess interventions that are recommended on the WWW [29, 30].
At the individual level, doctors should advise patients to be wary of sites that recommend one intervention to the exclusion of all others, particularly if they have a commercial interest in so doing. Doctors might also suggest sites that meet evidence-based and ethical guidelines such as those in the above list. Generally, sites that have been recommended by a search engine are more reliable than those obtained from simple searches.
At a collective level too, the medical profession needs to ensure that websites meet evidence-based and ethical guidelines. One possibility would be a database of systematic reviews of web sites using explicit inclusion and exclusion. These reviews would give ratings for the quality and balance of content, similar to those provided by the Cochrane Collaboration for peerreviewed journals.
In the longer term, software may be developed that will allow Internet users to select and filter websites according to quality criteria chosen by the user (automatic downstream filtering) [1]. Internet information would have to be labelled with ‘meta-data’ in a standardized format to allow software to select information that is suitable for an individual user. In addition to peer review, it has been suggested that other indirect quality indicators could be used for automatic selection by software. This would include a ‘website index’ which could be compiled from the absolute numbers of hyperlinks to a certain website, or new hyperlinks established over a period of time. The number of visitors a day as determined by an independent party might also be useful. Although these indicators are easy to obtain, they may have poor specificity as a popular website with many users may still contain unreliable information [1].