Abstract
Background:
Although numerous websites for patients with chronic kidney disease (CKD) are available, little is known about their content and quality.
Objective:
To evaluate the quality of CKD websites, and the degree to which they align with information needs identified by patients with CKD.
Methods:
We identified websites by entering “chronic kidney disease” in 3 search engines: Google.com (with regional variants for Australia, Canada, the United Kingdom, and the United States), Bing.com, and Yahoo.com. We included the first 50 unique English-language sites from each search. We evaluated website content using a 30-point scale comprising 8 priority content domains identified by patients with CKD (understanding CKD, diet, symptoms, medications, mental/physical health, finances, travel, and work/school). We used standardized tools to evaluate usability, reliability, and readability (DISCERN, HONcode, LIDA, Reading Ease, and Reading Grade Level). Two reviewers independently conducted the search, screen, and evaluation.
Results:
Of the 2093 websites identified, 115 were included. Overall, sites covered a mean (SD) of 29% (17.8) of the CKD content areas. The proportion of sites covering content related to understanding CKD, symptoms, and diet was highest (97%, 80%, and 72%, respectively). The proportion of sites covering travel, finances, and work/school content was lowest (22%, 12%, and 12%, respectively). The mean (SD) scores for DISCERN, LIDA and HONcode were 68% (14.6), 71% (14.4), and 75% (17.2), respectively, considered above average for usability and reliability. The mean (SD) Reading Grade Level was 10.6 (2.8) and Reading Ease was 49.8 (14.4), suggesting poor readability.
Conclusions:
Although many CKD web sites were of reasonable quality, their readability was poor. Furthermore, most sites covered less than 30% of the content patients identified as important for CKD self-management. These results will inform content gaps in internet-accessible information on CKD self-management that should be addressed by future eHealth web-based tools.
What was known before
Patients often visit websites to access health information; however, little is known about the content and quality of online information available for patients with chronic kidney disease (CKD). Previous work has highlighted gaps in quality and reported poor readability; these studies have primarily focused on appraising some quality measures, including readability, in a limited number of websites. A comprehensive evaluation of both content and quality of information on websites is needed.
What this adds
Our study provides an assessment of local and international CKD website quality and summarizes the availability of information content areas identified as important by patients with CKD. This study highlights content and quality gaps in internet-accessible information on CKD that should be addressed by future web-based tools.
Introduction
Chronic kidney disease (CKD) affects approximately 10% of adults in Canada and contributes to significant morbidity, mortality, and health care expenditures. 1 The trajectory of CKD progression may be improved by lifestyle modifications, such as diet and blood pressure management, and as a result, there is considerable interest in encouraging patients to self-manage aspects of their lifestyle, medical treatments, and symptom monitoring and management, with the goal of slowing disease progression.2-4 Numerous websites have been developed to support patient education for CKD; however, little is known about the content and quality of these websites, or the extent to which they address information needs identified by patients. Previous work evaluating CKD websites was limited to first page search results, is out-of-date, or was limited in scope (focusing on a small number of unique websites or limited to a specific geographic region or interest area such as living donor transplantation).5-10
The main objectives of this environmental scan were to (1) identify and evaluate CKD website quality (usability, reliability, and readability), and (2) summarize CKD content relative to self-management needs identified by CKD patients in our prior work. 11
Methods
Search Strategy
Given the diversity in websites and data, an environmental scan was determined to be the optimal method to synthesize knowledge regarding the content and quality of websites for patients with CKD. We consulted with a medical library specialist at the University of Calgary to finalize our search strategy and search terms. We systematically searched the 3 most popular English-language search engines: GoogleTM, BingTM, and YahooTM. 12 To account for potential geographic variation in search results, we included regional variants for the leading search engine, Google, which accounted for 71% of desktop and 89% of mobile traffic worldwide in the year preceding data collection. 12 We searched “chronic kidney disease” in each of the search engines: Google.com (four separate searches with preferences set to Australia, Canada, the United Kingdom, and the United States), Bing.com, and Yahoo.com. “Chronic kidney disease” is a term commonly used by patients in website searches. 5 In search engines, each search term is treated as a distinct keyword; therefore, while the search would prioritize sites with all 3 terms (and particularly those sites that repeat the key word[s]), it would also include sites with an individual term. Thus, “kidney disease” would capture the majority of sites, and inclusion of “chronic” would provide additional specificity.
Web traffic data, including Alexa Rank, was collected alongside the search and screen using a free chrome plug-in, SEOquake, which reports search engine optimization metrics for each website listed in the search results in the 3 search engines used in this study. 13 Alexa rank is a metric of global website popularity relative to all other websites available and is calculated based on daily website traffic and page views over a 3-month period. 14
Inclusion and Exclusion Criteria
We included only English-language websites. Websites that had nonfunctional links, required subscription to access content, or were discussion forums, blogs, scholarly journals or news articles were excluded. Eligible websites were recorded in a spreadsheet for further evaluation.
Website Screening
Two reviewers (M.D.S. and S.G.) independently searched and screened websites in September 2018 for inclusion. We followed the Canadian Agency for Drugs and Technologies in Health guidelines 15 for systematically searching the internet. Websites were reviewed in descending order until 50 unique, eligible websites were identified from each search engine (ie, we did not include subsites of URLs already cataloged), or until at least 200 sites were screened for eligibility in each search. Websites were included if both reviewers agreed on their inclusion; conflicts and uncertainty were resolved by discussion and involvement of a third reviewer (B.R.H.).
Website Review and Data Extraction
The review occurred in 2 phases: (1) quality (usability, reliability, and readability) and (2) assessment of CKD content. Phase 1 was completed independently by 2 reviewers (M.D.S. and S.G.) for all sites. For Phase 2, each website was reviewed by 1 reviewer (divided equally between reviewers).
Phase 1: Quality assessment (usability, reliability, and readability)
We assessed eligible sites for quality using the validated DISCERN,16,17 Health on the Net (HON) Code of Conduct, 18 and MinervaLIDAtion (LIDA) 19 instruments. These tools are freely available online and have been used extensively to evaluate health website quality.20-28
The DISCERN tool includes 16 questions that evaluate reliability and quality of written health information. Similar to Lutz et al, 5 we used the DISCERN tool without the 7-item subscale related to treatment choices, as treatment choices for CKD (dialysis, transplant) were not the focus of our review. Each question was rated on a 5-point Likert scale; the minimum overall score (excluding the treatment subscale) was 9 and the maximum was 45. Higher scores indicate greater quality and reliability. The DISCERN tool was originally developed for a “consumer,” rather than academic audience, although it has been used primarily by academic researchers to appraise written health information. 29
The HONcode includes 14 questions that evaluate 8 overarching principles relating to justifiability, transparency, and financial disclosure, among other items. Each question is rated as present (1)/absent (0), with higher scores indicating greater compliance with HONcode standards. The HON Foundation has provided official certification for qualified websites since 1995 and the code is broadly known as an ethical standard for health websites. 30
The LIDA instrument is a tool both to guide the development of and to evaluate health websites. The LIDA instrument includes 41 questions that assess accessibility, usability, and reliability. We used the LIDA instrument without the accessibility subscale, as the online accessibility checker tool is no longer available and some of the questions are not applicable to more recent website development advancements. For the usability and reliability sections, each question is rated on a 3-point scale; the minimum overall score for these sections is 0 and the maximum is 81. Higher scores indicate greater usability and reliability.
Readability was assessed using the Flesch Reading Ease (FRE) scale and Flesch-Kincaid Reading Grade Level (FKRGL).31-33 Similar to other studies5,9,10 we imported 200 to 400 words from each site into Microsoft Word and assessed the text using the readability statistics function, which reports both the FRE and FKRGL scales (MS Word 10, Microsoft Inc., Redmond, Washington). Both scales are calculated using word length, number of syllables per word, and number of words per sentence. FKRGL is reported as a numerical grade level and FRE is scored on a scale of 0 to 100, with lower scores suggesting poor readability. In general, a grade 6 reading level is recommended for patient education materials. 34
Phase 2: Assessment of CKD content
We assessed all sites for specific content areas relating to information needs identified by CKD patients and their caregivers from a National CKD self-management study. 11 Content areas included overarching categories of understanding CKD, diet, symptoms, medications, finances, mental/physical health, travel, and work/school. Several subcategories were identified under each broad category, resulting in 30 content areas in total.
We also categorized websites according to the audience(s) the site appeared to be tailored to (patient-focused, clinician-focused, or both) and by site type as follows: (1) academic/professional (universities, hospitals, and research centers); (2) nonprofit organizations, foundations, and charities; (3) commercial and pharmaceutical companies; and (4) other.
Rater Calibration
The 2 reviewers independently screened a convenience sample of 20 consecutive websites using the inclusion and exclusion criteria. After 2 rounds (40 websites), good agreement was achieved (κ = 0.92). Following the website screening activity, the 2 reviewers piloted the evaluation process by independently applying the assessment instruments to 5 randomly selected CKD sites. This process familiarized the reviewers to the instruments and identified interpretive differences for further discussion. Scoring differences were resolved by discussion. The reviewers conducted 3 sequential pilots (15 websites total). Following the first round, the percent agreement for the CKD content checklist was >90% and by the third round was ≥90% across all instruments (LIDA = 93.1%, κ = 0.52; HONcode = 97.7%, κ = 0.81; DISCERN = 90%, κ = 0.68).
Data Synthesis/Analysis
We used descriptive statistics (mean [percentage], SD, frequencies) to summarize quality and content scores. Scores demonstrating greater than 20% difference between raters were reviewed a second time and a final score was assigned by consensus. We reported the mean of the 2 raters’ results for scores with differences of less than 20%.
Results
Search Results and Website Characteristics
A total of 2093 sites were identified by the 2 reviewers (Figure 1). After ineligible sites and duplicates were removed from individual searches, 115 websites were eligible and included in the evaluation. The most common reasons for exclusion were duplicate websites, academic journals, and news articles. The number of websites identified in each search engine and the number of search engines each site appeared in are summarized in Figure 2. There was overlap in website identification across each of the searches, with some websites appearing in more than 1 search engine; however, only approximately 35% of websites were repeatedly identified in more than 3 search engines and 41% of websites appeared in a single search engine only. Websites were primarily targeted to a patient (56%) or patient and clinician audience (39%) and the majority of sites were developed by academic/professional centers (universities, hospitals, and research centers) (39%), followed by commercial and pharmaceutical companies (30%), nonprofit organizations, foundations and charities (29%), and other/unknown (1%).

CKD website review PRISMA flow diagram.

Search engine results.
CKD Content
Overall, sites covered a mean (SD) of 29% (17.8) of the 30 identified CKD content areas (Tables 1 and 2). The proportion of sites covering at least 1 content topic related to understanding CKD, symptoms, and diet was the highest (97%, 80%, and 72% respectively), and approximately half included content on mental/physical health (53%) and medications (52%). The proportion of sites covering travel, finances, and work/school content was low (22%, 12%, and 12%, respectively). There were a number of patient-identified information needs found in less than 10% of websites. This included those related to diet (options for eating out, practical tools for diet tracking), mental and physical support (screening tools for depression, how to communicate with others about mental health), finances (cost considerations for medications, food etc), travel (accessing health care abroad, what to bring), and work and school (supports and considerations for returning to work/retraining, how to discuss potential limitations with employer or school, and what should be disclosed and to whom). When stratified by website category (academic/professional, commercial/pharmaceutical, nonprofit/charity) or website audience (patient, clinician, both), there was minimal difference in the proportion of content topics covered by websites (Supplemental Table 1).
Proportion of Websites Addressing Specific CKD Content (n = 115).
Note. CKD = chronic kidney disease.
Website details including audience, rank, content and quality (n=115).
Note. CKD = chronic kidney disease; FRE = Flesch Reading Ease; FKRGL = Flesch-Kincaid Reading Grade Level; eGFR = estimated glomerular filtration rate; CDC = Centers for Disease Control and Prevention; CESPHN = Central and Eastern Sydney Primary Health Network; DPC = Dialysis Patient Citizens; VMC = Virtual Medical Center; NIDDK = National Institute of Diabetes and Digestive and Kidney Diseases; NHS = National Health Service.
Website became unavailable during evaluation.
Website link no longer accessible on 9 July 2019.
The top 10 websites covering the greatest proportion of CKD content areas were Davita (93.3%), National Kidney Foundation (76.7%), Fresenius Kidney Care (66.7%), Kidney Foundation of Canada (63.3%), Kidney Care UK (60%), Manitoba Renal Program (60%), National Institute of Diabetes and Digestive and Kidney Diseases (60%), Renal Support Network (60%), The Renal Association (60%), Life Options (56.7%), and NHS Inform (56.7%).
Website Quality (Reliability, Usability, and Readability)
Most websites demonstrated moderate to high reliability and usability (Table 3); however, there was considerable variability across all websites, with scores as low as 35% in some cases. There was little variability when comparing mean scores of each instrument. The mean (SD) DISCERN score was 68.1% (14.6), LIDA was 71.0% (14.4), and HONcode was 74.6% (17.2). Although there was only slight variation in most scores, 2 usability measures were comparatively low across all sites: engagability (57%) and currency of presented information (51%). There was considerable variation in readability across all websites; the mean (SD) FKRGL was grade 10.6 (2.8) and FRE was 49.8 (14.4), suggesting poor readability. When stratified by website category (academic/professional, commercial/pharmaceutical, nonprofit/charity) or website audience (patient, clinician, both), there was minimal difference in quality scores across most quality metrics (Supplemental Table 1). The DISCERN, HONcode, and LIDA Reliability scores were slightly higher for websites geared toward a clinician audience; however, there were only 3 clinician-specific websites included in our evaluation. Readability did not differ based on website category or audience.
Mean Quality and Readability Assessment Scores for Reviewed Sites (n = 115).
Note. FRE = Flesch Reading Ease; FKRGL = Flesch-Kincaid Reading Grade Level.
The top 10 websites for each quality metric are included in Table 4. There was some variability in the websites achieving the highest quality scores across each quality metric; however, several sites scored in the top 10 across multiple quality metric categories (including American Kidney Fund, Kidney Care UK, Medical news Today, My Doctor, Family Doctor, MyHealth Alberta, UpToDate, and Very Well Health).
Top 10 Websites According to Quality Criteria a .
Note. CDC = Centers for Disease Control and Prevention; NIDDK = National Institute of Diabetes and Digestive and Kidney Diseases.
HON Code—12 websites scored 100.0%: Cleveland Clinic; Drugs.com; eMedicine Health; Family Doctor; Lab Tests Online; Mayo Clinic; Medical News Today; Medicine Net; My Doctor; Patient; Very Well Health; WebMD.
Website Traffic
There was considerable variability in website traffic measured using Alexa Rank. The Alexa Rank ranged from 5 (Wikipedia) to 14.4M (Kidney Support Network). Websites with the highest traffic (Table 5) tended to be general health websites offering information on multiple medical conditions. One exception was the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK), which ranked second overall for website traffic in our results (Alexa Rank 153). Most websites with the highest web traffic covered less than 40% of content topics (ranging from 6.7% [Wiki How] to 60% [NIDDK]), had quality scores above 70% in most cases (ranging from 44.4 [Medical Dictionary] to 100 [WebMD, Mayo Clinic, Medical News Today]), and readability ranged from grade 6.7 (NIDDK) to grade 14.5 (Wikipedia).
Top 10 Websites According to Website Traffic (Alexa Rank).
Note. NIDDK = National Institute of Diabetes and Digestive and Kidney Diseases; CKD = chronic kidney disease; CDC = Centers for Disease Control and Prevention.
Discussion
We completed a comprehensive review of CKD websites to evaluate their quality as well as content relative to 8 overarching information needs identified by CKD patients in our prior work. 11 Of the 115 websites included, we found that they were of reasonably high quality, as assessed using validated instruments. However, their mean reading grade level was grade 11, much higher than recommended for a general patient audience, and information related to travel, finances, and work/school in particular was very limited.
We found considerable variation in content covered across all websites, with the majority of websites including information on less than 30% of the information needs expressed by patients as important for self-management. 11 Information related to understanding CKD, symptoms, and diet were included across most of the websites; however, content related to travel, finances, and work/school considerations was particularly lacking and, where available, was not tailored to a Canadian audience. Our review also highlighted deficiencies in content related to mental health, including screening tools for depression and strategies to communicate with others about mental health, and content related to dietary and self-management needs, such as practical tools for tracking diet and laboratory data and guidance on eating in restaurants. In addition, the website content was often difficult to navigate and required considerable time searching for the topics listed in our content checklist.
We also found considerable variability in website traffic measured using Alexa Rank, with the highest ranking (correlating to the greatest traffic) being general health websites and not specific to CKD. In fact, the websites with the highest traffic covered less than 40% of the content areas and information needs identified by patients with CKD. Furthermore, both quality and readability were similar irrespective of the website category or audience.
While previous CKD website reviews focused on quality and readability assessment,5,10 a recent evaluation 35 of CKD mobile applications available in the Apple Store, Google Play, and 360 Mobile Assistant included a review of content topics identified as important to CKD self-management. 36 Similar to our review, Lee et al found that most of the mobile applications contained less than 60% of the information topics assessed and found considerable content gaps relating to managing medications (11.8% of apps), engaging and sustaining social support (6.4% of apps), and maintaining social and occupational roles (0% of apps). 35 These findings reveal a consistent need to improve the comprehensiveness of content relating to patient-identified self-management information needs across all eHealth platforms.
When compared with earlier studies,5,10 our evaluation revealed some improvement in website quality, particularly with regard to referencing the source of content and discussing areas of uncertainty, although this finding could be due to the larger number of websites evaluated in our review. Readability, however, continues to be a significant issue with the majority of sites written for a grade 10, or higher, audience. Weiss 34 recommends that written health information be targeted to a grade 6 audience to ensure that patients with a range of health literacy are able to understand presented information. Similar to other studies appraising CKD websites,5,10 we found that most CKD websites had a reading grade level greater than the 10th grade, or rated “difficult” to “very difficult.” As patient-focused health resources are increasingly developed for and accessed from an online environment, 37 this finding highlights the ongoing need to reduce the complexity of online health information and improve the accessibility of the language used.
Strengths of our study include a broad search methodology and inclusion of 1 popular search engines encompassing results from Australia, Canada, the United Kingdom, and the United States. We used 3 different measures of usability and reliability, with 2 reviewers completing the evaluation independently. The 2 reviewers also independently measured readability using 2 different methods for all sites.
Our study does, however, have limitations. We limited our review to English-language websites, and although we screened >2000 websites and included 115 in our evaluation, our search terms were limited to “chronic kidney disease” so it is possible we inadvertently missed CKD websites that did not include these terms, although these are the terms most commonly used by patients with CKD. 5 In addition, websites may change rapidly over time and the content and quality may change as websites are updated and new content and features are added; therefore, the results are reflective of the website content and quality at the time of our review.
Our study provides a comprehensive review of CKD websites and includes regional variants within the search strategy, allowing broad identification and evaluation of international sites. Our results highlight the lack of tailored, comprehensive online information for CKD, and identify specific information and quality gaps in need of attention. These results will inform content gaps in internet-accessible information on CKD self-management that should be addressed by future eHealth web-based tools.
Supplemental Material
SuppTable_1._Mean_score_based_on_website_category_and_audience_17May2019 – Supplemental material for Content and Quality of Websites for Patients With Chronic Kidney Disease: An Environmental Scan
Supplemental material, SuppTable_1._Mean_score_based_on_website_category_and_audience_17May2019 for Content and Quality of Websites for Patients With Chronic Kidney Disease: An Environmental Scan by Michelle Smekal, Sarah Gil, Maoliosa Donald, Heather Beanlands, Sharon Straus, Gwen Herrington, Dwight Sparkes, Lori Harwood, Allison Tong, Allan Grill, Karen Tu, Blair Waldvogel, Chantel Large, Claire Large, Marta Novak, Matthew James, Meghan Elliott, Maria Delgado, Scott Brimble, Susan Samuel and Brenda R. Hemmelgarn in Canadian Journal of Kidney Health and Disease
Footnotes
Acknowledgements
The authors would like to acknowledge James Wick for Stata technical support.
Ethics Approval and Consent to Participate
Not applicable, as this study did not involve human subjects or information.
Consent for Publication
All authors consent to the publication of this study.
Authors’ Note
The results presented in this article have not been published in whole or part elsewhere.
Author Contributions
All authors in this study have contributed to this article and approve of this submission. M.D.S., S.G., and B.R.H. contributed to the study design and drafted the article. All authors contributed to the design and provided critical revisions to this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by the Can-SOLVE CKD Network, which is supported by The Canadian Institutes of Health Research (CIHR) under Canada’s Strategy for Patient-Oriented Research (SPOR). Dr Hemmelgarn is supported by the Roy and Vi Baay Chair in Kidney Research. Dr Tu is supported by a research scholar award from the Department of Family and Community Medicine at the University of Toronto.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
