Characterization and reliability of internet resources on pulmonary rehabilitation for individuals with chronic lung disease

Abstract

Background

Individuals with lung disease commonly use the internet as a source of health information on pulmonary rehabilitation (PR). The objective of this study was to characterize internet resources on PR, and to assess the content, readability, and quality of patient-directed PR resources.

Methods

The first 200 websites for the search term ‘pulmonary rehabilitation resources and exercise’ were analyzed on Google, Yahoo, and Bing. Website content was assessed based on 30 key components of PR from the 2013 and 2021 international consensus statements. Website quality was determined using DISCERN, JAMA benchmarks, and Global Quality Scale (GQS).

Results

66 unique PR websites were identified with the two most common categories being scientific resources (39%) and foundation/advocacy organizations (33%). The average reading level of websites was 11 ± 3. PR content varied significantly across websites (mean range 13.4–21.5). Median DISCERN total score and GQS score were 4 (IQR 3-4) and 3.5 (IQR 2-4), respectively, representing moderate-good quality. Foundation/advocacy websites had higher DISCERN and GQS scores compared to other websites.

Conclusion

Foundation/advocacy websites had the highest quality and reliability metrics; however, the higher-than-recommended reading levels may compromise patient comprehension and utilization. This study provides critical insight on the current state of online PR health-related information.

Keywords

pulmonary rehabilitation exercise training chronic lung disease internet resources health education

Introduction

Pulmonary rehabilitation (PR) is a multidisciplinary program of patient-tailored therapies including exercise training, education, and behaviour change that aims to reduce dyspnea, improve exercise tolerance, and enhance health-related quality of life (HRQL) in individuals with chronic lung disease.¹ PR remains underutilized worldwide despite being an integral component in the clinical management of individuals with chronic lung disease.²

The internet is commonly used as a source of health information, with about 93% of North Americans having internet access.³ The internet allows easy and immediate access to health information, where online patient education materials can be easily updated as knowledge regarding a chronic disease evolves and new clinical consensus guidelines are developed. While patient education about a chronic condition can empower patients to participate in self-management behaviours and achieve better health outcomes,⁴ the accuracy and reliability of online health information varies widely.^5–7

Clinical practice guidelines on PR were published in 2013, with a more recent consensus report in 2021 identifying the essential components of PR.^1,2 Despite these guidelines, individuals with chronic lung disease may access information from online resources that are incomplete, biased, and potentially inaccurate.^5,8 In addition, online resources are often written at higher than the recommended sixth-grade reading level for patient education materials, which can compromise comprehension and utilization.⁹

Since the start of the COVID-19 pandemic, the transition to virtual PR programs has led many individuals with lung disease to rely on online resources for exercise training. This reliance is expected to increase in post-pandemic conditions;^10,11 however, the quality and reliability of these online patient-directed PR resources has not been evaluated. Thus, the objective of this study was to characterize internet resources on PR, and to assess the content, readability, quality, and reliability of websites compared with international PR consensus guidelines.² We hypothesized that available online PR resources would be predominately from scientific resources and foundation/advocacy organizations, and that these resources would score higher on website content and quality metrics than websites from other sources.

Methods

Search strategy

We analyzed internet resources on PR using the top three search engines − Google, Yahoo, and Bing. An internet search was conducted using the search term ‘pulmonary rehabilitation resources and exercise’ on 24 July 2021. An updated search was conducted on 3 December 2022 to identify any new or relevant websites. A United States (US) internet protocol address and virtual private network were used for each search following the removal of web browsers’ history and cookies.⁵

Study selection

The first 200 identified websites for each search engine were screened for eligibility for the initial search. For the updated search, the first 50 websites for each search engine were screened for eligibility. Websites were included if they described the concept of PR and/or exercise training and its benefits, and/or provided an instructional portion to serve as a rehabilitation program for individuals with chronic lung disease. Additional websites directly identified from the online page of eligible websites were also reviewed and included, if they met criteria. Websites were excluded for any of the following reasons: (1) duplicate records; (2) non-English; (3) scientific journal articles; (4) dedicated solely for patient enrollment in PR programs; (5) exclusively video-based (e.g. YouTube); (6) requiring a fee or health care provider credential to enroll and access health information; (7) unrelated to PR and/or exercise training; or (8) specifically targeting rehabilitation in patients recovering from COVID-pneumonia. Websites were not excluded based on intended audience as a survey conducted by the Health on the Net found that 62% of Internet users seek medical literature when searching for health information.¹² Website eligibility was determined by the primary reviewer (TDS) according to the predefined criteria. Websites with unclear eligibility were reviewed by secondary reviewers (AL and NAK). Ongoing discrepancies were assessed by a third reviewer (DR).

Given that most websites contained links to additional resources, we only assessed the main website unless the link resulted in a relevant article or video that remained within the original domain. Other forms of content such as downloadable booklets, leaflets, and pamphlets were also included. The URLs from all eligible websites are provided in Supplementary Table 1.

Data abstraction

Website characterization

The following data were abstracted from each website: publication date, geographic location of website origin, website category, and target population. Websites were classified into five main categories: scientific resources (i.e. academic institutions and governmental organizations), foundation/advocacy organizations, industry or for-profit, personal commentary (i.e. personal blog), and PR enrollment websites.⁵ The industry/for-profit and personal commentary/blog websites were grouped into a single category for statistical modelling purposes given limited numbers in each category.

Website evaluation

Website readability was evaluated using the Flesch Reading Ease Score (FRES) and the Flesch-Kincaid Grade Level from the built-in readability statistics function of Microsoft Word 2018^TM, as further outlined in the Supplemental Material.^13,14 Website content was evaluated based on a predefined scoring system of the 30 key components of PR, as described in the 2013 and 2021 consensus statements capturing the major elements of PR,^1,2 as well as additional components deemed important by our team based on the literature (Supplementary Table 2). Website content was scored a “yes” (1) indicating the item was addressed or “no” (0) as not addressed. Videos contained within websites were evaluated for website content only and not for quality.

Website quality was assessed using the DISCERN instrument, Journal of the American Medical Association (JAMA) benchmarks, and the Global Quality Scale.^15–17 DISCERN is a validated instrument that provides a reliable way of assessing the quality of written information on the internet (Supplementary Table 3).¹⁵ The JAMA benchmarks score uses four criteria for the qualitative assessment of websites: authorship, attribution, currency, and disclosure.¹⁶ Global Quality Scale (GQS) is a 5-point scale that assesses website accessibility, quality, and overall flow of information, and the usefulness of that website to patients.¹⁷ Health on the Net (HON) certification status [https://www.hon.ch/HONsearch/Patients/medhunt.html] was also determined for each site.¹⁸

Each website was independently scored for content and quality by two reviewers (TDS, and NAK or AL). Scores within 1 point were considered in agreement for DISCERN questions 1 to 15. In these cases, the initial score provided by the primary reviewer was used. Overall quality score (i.e. DISCERN question 16) was determined by using the median score of questions 1 to 15. Discrepancies were resolved using a consensus score between the two reviewers for the overall website content score, JAMA, and GQS quality scores. Any ongoing discrepancies were resolved by a third reviewer (DR).

Statistical analysis

Descriptive statistics were used to characterize available internet resources. Content and quality metrics were compared across website categories using one-way ANOVA, Kruskal-Wallis, and Wilcoxon rank sum as appropriate. Inter-rater reliability was assessed using Cohen’s kappa. Multivariable linear regression models were used to evaluate the associations between website characteristics and website content, DISCERN, and GQS total scores. Predefined model variables included target population, geographic location of website origin, website category, and HON status.⁵ Statistical analyses were performed using SPSS (version 23.0; IBM, Armonk, NY, USA), with two-tailed p < 0.05 considered significant.

Results

Website characteristics

The search yielded 66 unique websites meeting eligibility criteria (Figure 1). A list of the eligible websites and their rank in each search engine are provided in Supplementary Table 1. Website characteristics are summarized in Table 1, with the two most common categories being scientific resources (39%) and foundation/advocacy organizations (33%). Most PR websites addressed all types of chronic lung diseases (55%), whereas others focused on specific lung diseases such as chronic obstructive pulmonary disease (30%), interstitial lung disease (7.5%), and others (7.5%; pulmonary hypertension, lung transplantation, and bronchiectasis). HONcode certification was present in only 8% of the websites. The average reading grade level of PR websites was equivalent to grade 11 (11.2 ± 3.3, ages 16–17) and the FRES was 42.4 ± 18.6, corresponding to a difficult readability level.

Figure 1.

Flow diagram of the search results and study selection. *Other: PR websites identified directly through additional resources from eligible websites outside of the original search.**Search was updated on 3 December 2022; 6 websites were removed as content was no longer available. Abbreviations: PR: pulmonary rehabilitation.

Table 1.

Website characteristics.

Website characteristics	Overall (n = 66)
Website category
Scientific resource	26 (39%)
Foundation/advocacy organization	22 (33%)
PR enrollment websites	11 (17%)
Industry/for-profit	3 (5%)
Personal commentary/blog	4 (6%)
Target population
Chronic lung disease (general)	36 (55%)
COPD	20 (30%)
ILD	5 (7.5%)
Other^*	5 (7.5%)
Continent of origin
North America	54 (82%)
Europe	8 (12%)
Australia	4 (6%)
Other	0 (0%)
HON Certification status	5 (8%)
Flesch-Reading Ease Score^a	42.4 ± 18.6
Flesch-Kincaid grade level	11.2 ± 3.3
Content total score^b	18.0 ± 5.5
DISCERN total score^c	4 [3-4]
GQS total score^c	3.5 [2-4]
JAMA total score^d	2 [2-3]

Data are shown as n (%), median [interquartile range], or mean ± SD. The content total score was based on a predefined scoring system of 30 key components of pulmonary rehabilitation as described in the 2013 PR guidelines and the 2021 PR international consensus document.^1,2

Abbreviations: COPD: chronic obstructive pulmonary disease; GQS: Global Quality Scale; HON: Health on the Net Foundation code of conduct; ILD: interstitial lung disease.

*Other includes pulmonary hypertension (3%), lung transplantation (1%) and bronchiectasis (1%).

^aFlesch-Reading Ease Score rated on a scale from 0-100 with lower scores indicating greater difficulty.

^bContent total score rated on a scale from 0-30 with higher scores indicating greater content scores.

^cDISCERN and GQS total score rated on a scale from 1-5 with higher scores indicating greater quality.

^dJAMA total score rated on a scale from 0-4 with higher scores indicating greater quality.

Website content

There was significant variability across websites with respect to PR content (18.0 ± 5.5 content items out of a maximum of 30 items; Figure 2). The two major components of PR were described in almost all websites (exercise training: 100%, education: 95%), with less emphasis placed on the concept of behaviour change (44%). Most websites focused on traditional aerobic (95%), resistance (86%), and flexibility/stretching (76%) training modalities, in contrast to balance exercises (15%) and inspiratory muscle training (20%). Slight majority of the websites included the FIT principle for exercise training (Frequency: 61%, Intensity: 55%, and Time: 59%, as described in Figure 2). Falls risk as a safety consideration was only mentioned in 3% of websites. Several websites provided education focused on smoking cessation (59%), breathing strategies (61%) and nutrition (58%), with fewer websites addressing physical activity (39%), self-efficacy (42%), and motivation (18%). The importance of adherence and maintenance in PR was discussed in most websites (64%). Website content was significantly different across website categories (p < 0.001; Table 2). Pairwise comparison revealed higher content scores across foundation/advocacy websites (content score 21.5 ± 3.6 out of 30) as compared to scientific websites (content score 16.2 ± 5.3; p = 0.003) and industry/for-profit and personal commentary/blogs (13.4 ± 6.2; p = 0.002).

Figure 2.

Content scores across website categories. Each row represents an individual website. The shade of each cell corresponds to the content score, ranging from 0- not addressed (light gray) to 1-adequately addressed (blue). Abbreviations: PR = pulmonary rehabilitation; FIT = Frequency, Intensity, Time.

Table 2.

Website content, reliability, and quality across website categories.

Questionnaire items	Website category
Questionnaire items	Overall (n = 66)	Academic/Scientific resources (n = 26)	Foundation/Advocacy organizations (n = 22)	PR enrollment websites (n = 11)	Industry/For-profit and personal commentary/Blog (n = 7)	p-value
Content total score	18.0 ± 5.5	16.2 ± 5.3	21.5 ± 3.6	18.4 ± 5.0	13.4 ± 6.2	<0.001^*†
Discern Reliability (Q1-8)	3.0 [3.0–4.0]	3.5 [3.0–4.0]	4.0 [3.0–4.25]	3.0 [3.0–3.0]	2.0 [2.0–3.0]	0.007^†
Discern Quality (Q9-15)	4.0 [3.0–4.5]	4.0 [2.5–4.13]	4.0 [4.0–5.0]	4.0 [1.5–4.0]	2.0 [1.5–4.0]	0.011^*†
Discern Overall score (Q16)	4.0 [3.0–4.0]	4.0 [3.0–4.0]	4.0 [4.0–5.0]	3.0 [2.0–4.0]	2.0 [1.0–3.0]	<0.001^†‡
GQS score	3.5 [2.0–4.0]	3.0 [2.0–4.0]	4.0 [3.75–5.0]	3.0 [2.0–4.0]	1.0 [1.0–3.0]	<0.001^*†‡
JAMA total score	2.0 [2.0–3.0]	2.5 [2.0–3.25]	2.0 [2.0–3.0]	2.0 [2.0–2.0]	2.0 [2.0–2.0]	0.027

Data are shown as median [IQR] or mean ± SD.

Abbreviations: GQS = Global Quality Scale; JAMA = Journal of the American Medical Association.

p-value column represents one-way ANOVA and Kruskal–Wallis tests comparing each item across all website categories for normally distributed and non-normally distributed data, respectively.

^*p < 0.04 on pairwise comparison of foundation/advocacy organization versus academic/scientific resources.

^†p < 0.02 on pairwise comparison of foundation/advocacy organization versus industry/for-profit and personal commentary/blog.

^‡p < 0.02 on pairwise comparison of foundation/advocacy organization versus PR enrollment websites.

Website quality

Website quality and reliability, as measured by the median DISCERN total score, was 4 (interquartile range [IQR], 3-4; Table 1). The scores for individual DISCERN questions are shown in Supplementary Figure 3. Questions 10 and 13 received the highest ratings, which respectively describe the benefits of PR and how PR affects overall quality of life. The lowest scores were for questions 4 and 5, which address the clear use and date of sources, and question 8, which refers to areas of uncertainty in PR. The median DISCERN total score varied across website categories (p < 0.001; Table 2), with higher scores for foundation/advocacy websites compared to both PR enrollment websites (p = 0.002) and industry/for-profit and personal commentary/blog websites (p = 0.002). Individual scores for DISCERN reliability (questions 1-8) were higher for foundation/advocacy organizations compared to industry/for-profit and personal commentary/blog websites (Table 2). Individual scores for DISCERN quality (questions 9-15) were higher for foundation/advocacy organizations compared to scientific websites (Table 2). The inter-rater reliability (Cohen’s kappa) for the overall median DISCERN score was 0.65 (95% CI [0.51,0.78]; 73% agreement across websites), with 100% consensus reached between the two reviewers.

The median GQS was 3.5 (IQR, 2-4; Table 1 and Supplementary Table 4). The GQS score was significantly different across website categories (p < 0.001; Table 2), with higher scores for foundation/advocacy websites compared to all other website categories (scientific websites p = 0.02, PR enrollment website p = 0.02, industry/for-profit organization and personal commentary/blog p = 0.002; Table 2). The inter-rater reliability (Cohen’s kappa) for the total GQS score was 0.75 (95% CI [0.63, 0.87]; 80% agreement across websites), with 100% consensus reached between the two reviewers.

The median JAMA benchmark criteria score was 2 (IQR 2-3; Table 1). A large proportion of websites contained disclosure of ownership and conflicts of interest (98%), and were updated with the latest information (currency, 92%; Supplementary Table 5). However, only 20% and 35% of websites included authorship and attribution, respectively. JAMA benchmark criteria score did not differ across website categories in the post-hoc test, even though significant with Kruskal-Wallis test (p = 0.027; Table 2).

Associations with PR website content and quality metrics

Total PR content score was associated with website category, with higher content scores in industry/for-profit and personal commentary/blog (4.90 points, 95% CI [0.11, 9.67], p = 0.045, Supplementary Table 6) compared to PR enrollment websites. DISCERN total score was 1.34 points (95% CI [0.55, 2.13]) higher for foundation/advocacy websites (vs. industry/for-profit and personal commentary/blog websites (p = 0.001). GQS was independently associated with website category, with higher GQS scores in foundation/advocacy websites (1.37, 95% CI [0.38, 2.36]) compared to industry/for-profit and personal commentary/blog websites (p = 0.007). HON status, continent of origin, and target population were not associated with website content, DISCERN, or GQS scores (Supplementary Table 6).

Discussion

To our knowledge, this is the first study to characterize internet resources on PR, and to assess the content, readability, quality and reliability of patient-directed online resources for individuals with chronic lung disease. We observed that foundation/advocacy websites had the highest quality and reliability metrics; however, the higher-than-recommended reading levels may compromise patient comprehension and utilization.

PR content varied substantially across websites, and only partially captured the items outlined in international consensus guidelines. Nearly all websites described exercise training and education as essential components of PR; however, most websites did not acknowledge the central role of behaviour change in promoting long-term adherence to health-enhancing behaviours.^1,19 Moreover, a significant proportion of websites did not include the Frequency, Intensity, Time (FIT principle) that enables PR uptake by patients.²⁰ In addition, non-traditional exercise training modalities (e.g. balance exercises, inspiratory muscle training), and education surrounding physical activity, self-efficacy, and motivation were not discussed. The inclusion of different training modalities across PR websites aligns with the contemporary principle of personalized PR, in which treatment choices are made on the basis of each individual’s needs, goals, and preferences.² Furthermore, self-efficacy is a critical motivating factor in the adherence and maintenance of health-enhancing behaviours post-PR.²¹ Therefore, it is important to educate around the principles of self-efficacy in patient-directed PR websites to promote behaviour change and improve health outcomes.^19,21,22 There is also an important absence of discussions regarding safety considerations, including falls risk and monitoring of exertional tolerance. These findings indicate that most websites did not include all the essential components of PR as highlighted in the ATS/ERS PR statements.^1,2

Website quality was moderate-to-good, suggesting that most websites were useful for patients.¹⁵ Websites frequently did not provide clear sources and dates, or refer to areas of uncertainty in PR management such as the optimal maintenance of program or timing of PR initation (only 5 out of 66 websites). Addressing areas of uncertainty or gaps in knowledge regarding treatment choices informs decision-making.²³ However, these issues are often complex and require in-depth discussions with a healthcare provider, and thus may not necessarily enhance the quality of websites. Further, JAMA benchmark criteria scores also revealed lack of authorship and attribution, indicating a lack of transparency in PR-related resources. Patient-directed websites would benefit from identifying references and sources of information to ensure compliance with quality standards and promoting transparency in scientific content.

Individuals with chronic lung disease use health-care providers as their primary source of health information, followed by the internet.^24,25 However, medical appointments often do not allow sufficient time to adequately address patient concerns and provide education.²⁶ Internet resources can be used to fill educational gaps and facilitate information sharing and learning. Thus, it is critical for healthcare providers to be knowledgeable about the health information available online. Foundation/advocacy websites ranked the highest in terms of quality and reliability, and provided the most complete and comprehensive information on PR and exercise training. These findings are consistent with previous studies reporting higher quality and reliability metrics across foundation/advocacy resources, which have an interest in promoting patient education and lung health.^5,8 Therefore, healthcare providers may consider directing patients towards online PR resources from foundation/advocacy organizations.

Health literacy is a major determinant of health.⁹ Limited health literacy is associated with worse health outcomes and lower adherence to self-management in many chronic diseases.⁹ Specifically, in chronic lung disease, lower health literacy has been associated with greater symptom severity, worse respiratory-specific health-related quality of life, poor inhaler technique, and higher likelihood of health-care utilization.²⁷ According to the American Medical Association and the National Institutes of Health, the readability of patient education materials should not exceed grade 6 reading level. The average reading level of PR websites was equivalent to grade 11, with only 3 (5%) websites written at a grade 6 level. This is comparable to other studies evaluating the readability of patient education materials in chronic lung disease and other medical conditions.^5,6,8 The higher reading level of patient-directed resources compromises patient comprehension and utilization, and contributes to further health literacy disparities. The inclusion of pictures, diagrams, and/or videos can improve information retention and comprehension, and adherence to health information.^28,29 These findings highlight the importance of targeted strategies that consider health literacy when developing patient resources.

Patients with chronic lung disease often have limited confidence in their ability to distinguish between high- and low-quality health resources on the internet.³⁰ The HONcode certification is an ethical standard for the identification of reliable and trustworthy online health information; however, only 5 (8%) of patient-directed PR websites were HON certified, which is similar to the rates observed across chronic disease states.^5,8,31,32 The certification process requires the website applicant to voluntarily submit a request for HON review and may be associated with a cost.³³ A certification system that automatically identifies websites conforming to the HONcode principles may be helpful to facilitate patient appraisal of online health information.^33,34

The following limitations should be considered. First, the searches were restricted to websites published in English and limited to the first 200 websites for each search engine. This restriction was justified given that most individuals typically examine the first page of a search engine when searching for online health information.³⁵ Second, we used a U.S. Internet Protocol (IP) address for our search, and thus results could potentially be different using other country-specific IP addresses. Third, our inter-rater reliability was moderately acceptable; however, we ensured that consensus was reached between the two reviewers. Lastly, our search was mainly limited to text-based online health information and thus did not include websites that were exclusively video-based. Future studies should evaluate other sources of online health information about PR, including videos. Despite these limitations, this is the first comprehensive assessment of patient-directed online health information on PR and exercise training.

Conclusion

PR content varied significantly across websites, and only partially captured items described in PR international consensus guidelines. Foundation/advocacy websites ranked the highest in terms of quality and reliability. Targeted strategies are needed that consider health literacy when developing PR resources for individuals with chronic lung disease. These findings have important implications as we emerge from the COVID-19 pandemic with increased reliance on virtual and hybrid PR programs, and on the internet as a source of health information.

Supplemental Material

Supplemental Material - Characterization and reliability of internet resources on pulmonary rehabilitation for individuals with chronic lung disease

Supplemental Material for Characterization and reliability of internet resources on pulmonary rehabilitation for individuals with chronic lung disease by Tania Da Silva, Ashira Lokhandwala, Noor Al Kaabi, Julie Semenchuk, Gillian C. Goobie, Encarna Camacho, W. Darlene Reid, Jolene H. Fisher, Christopher J. Ryerson and Dmitry Rozenberg in Chronic Respiratory Disease

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: DR receives research support from the Sandra Faire and Ivan Fecan Professorship in Rehabilitation Medicine.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Sandra Faire and Ivan Fecan Professorship in Rehabilitation Medicine and the University of Toronto CREMS Summer Program.

ORCID iDs

W. Darlene Reid

Dmitry Rozenberg

Supplemental Material

Supplemental material for this article is available online.

References

Spruit

Singh

Garvey

, et al. An official American Thoracic Society/European Respiratory Society statement: key concepts and advances in pulmonary rehabilitation. Am J Respir Crit Care Med 2013; 188(8): e13–e64. DOI: 10.1164/rccm.201309-1634ST

Holland

Cox

Houchen-Wolloff

, et al. Defining modern pulmonary rehabilitation. an official American Thoracic Society workshop report. Ann Am Thorac Soc 2021; 18(5): e12–e29. DOI: 10.1513/AnnalsATS.202102-146ST

Pew Research Center . Americans’ Internet access: 2000-2021. Washington, DC: Pew Research Center. (Accessed June 2, 2021).

Mahler

Petrone

Krocker

, et al. A perspective on web-based information for patients with chronic lung disease. Ann Am Thorac Soc 2015; 12(7): 961–965.

Fisher

O'Connor

Flexman

, et al. Accuracy and reliability of internet resources for information on idiopathic pulmonary fibrosis. Am J Respir Crit Care Med 2016; 194(2): 218–225. DOI: 10.1164/rccm.201512-2393OC

Storino

Castillo-Angeles

Watkins

, et al. Assessing the accuracy and readability of online health information for patients with pancreatic cancer. JAMA Surg 2016; 151(9): 831–837. DOI: 10.1001/jamasurg.2016.0730

Hirsch

Aggarwal

Barker

, et al. Googling endometriosis: a systematic review of information available on the Internet. Am J Obstet Gynecol 2017; 216(5): 451–458. DOI: 10.1016/j.ajog.2016.11.1007

Saleh

Fisher

Provencher

, et al. A systematic evaluation of the quality, accuracy, and reliability of internet websites about pulmonary arterial hypertension. Ann Am Thorac Soc 2022; 19: 1404–1413.

Berkman

Sheridan

Donahue

, et al. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med 2011; 155(2): 97–107. DOI: 10.7326/0003-4819-155-2-201107190-00005

10.

Dechman

Aceron

Beauchamp

, et al. Delivering pulmonary rehabilitation during the COVID-19 pandemic: a Canadian Thoracic Society position statement. Can J Respir Crit Care Sleep Med 2020; 4(4): 232–235.

11.

American Thoracic Society . What to do when pulmonary rehabilitation (PR) is unavailable. New York, NY: American Thoracic Society. (Accessed June 7, 2021).

12.

Pletneva

Cruchet

Simonet

M-A

, et al. User centred networked health care. IOS Press, 2011, pp. 73–77.Results of the 10 th HON survey on health and medical Internet use.

13.

Flesch

. A new readability yardstick. J Appl Psychol 1948; 32(3): 221–233. DOI: 10.1037/h0057532

14.

Kincaid

Fishburne

Jr Rogers

, et al. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Alexandria, VA: University Libraries, 1975.

15.

Charnock

Shepperd

Needham

, et al. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999; 53(2): 105–111.

16.

Silberg

Lundberg

Musacchio

. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. JAMA 1997; 277(15): 1244–1245.

17.

Bernard

Langille

Hughes

, et al. A systematic review of patient inflammatory bowel disease information resources on the World Wide Web. Am J Gastroenterol 2007; 102(9): 2070–2077. DOI: 10.1111/j.1572-0241.2007.01325.x

18.

Health on the Net Foundation . Health on the Net Foundation, https://www.hon.ch/en/. (Accessed June 7, 2021).

19.

Bourbeau

Nault

Dang-Tan

. Self-management and behaviour modification in COPD. Patient Educ Couns 2004; 52(3): 271–277.

20.

Garber

Blissmer

Deschenes

, et al.. American College of Sports Medicine position stand. Quantity and quality of exercise for developing and maintaining cardiorespiratory, musculoskeletal, and neuromotor fitness in apparently healthy adults: guidance for prescribing exercise. Med Sci Sports Exerc 2011; 43(7): 1334–1359.

21.

Selzler

A-M

Rodgers

Berry

, et al. The importance of exercise self-efficacy for clinical outcomes in pulmonary rehabilitation. Rehabil Psychol 2016; 61(4): 380–388.

22.

Rice

Bourbeau

MacDonald

, et al. Collaborative self-management and behavioral change. Clin Chest Med 2014; 35(2): 337–351.

23.

Han

PKJ

Klein

WMP

Arora

. Varieties of uncertainty in health care: a conceptual taxonomy. Med Decis Making 2011; 31(6): 828–838.

24.

Martinez

St Jean

Plauschinat

, et al. Internet access and use by COPD patients in the National Emphysema/COPD Association Survey. BMC Pulm Med 2014; 14(1): 66–111.

25.

Mahler

Cerasoli

Della

, et al. Internet health behaviors of patients with chronic obstructive pulmonary disease and assessment of two disease websites. Chronic Obstr Pulm Dis 2018; 5(3): 158–166.

26.

Linzer

Bitton

S-P

, et al. The end of the 15–20 minute primary care visit. Berlin: Springer, 2015, pp. 1584–1586.

27.

Omachi

Sarkar

Yelin

, et al. Lower health literacy is associated with poorer health status and outcomes in chronic obstructive pulmonary disease. J Gen Intern Med 2013; 28(1): 74–81.

28.

Schubbe

Scalia

Yen

, et al. Using pictures to convey health information: a systematic review and meta-analysis of the effects on patient and consumer health behaviors and outcomes. Patient Educ Couns 2020; 103(10): 1935–1960.

29.

Sudore

Schillinger

. Interventions to improve care for patients with limited health literacy. J Clin Outcomes Manag 2009; 16(1): 20–29.

30.

Stellefson

Shuster

Chaney

, et al. Web-based health information seeking and eHealth literacy among patients living with chronic obstructive pulmonary disease (COPD). Health Commun 2018; 33(12): 1410–1424.

31.

Lee

Ban

Park

, et al. Accuracy and reliability of internet resources providing information on obstructive sleep apnea. J Clin Sleep Med 2018; 14(10): 1717–1723.

32.

Grewal

Alagaratnam

. The quality and readability of colorectal cancer information on the internet. Int J Surg 2013; 11(5): 410–413.

33.

Boyer

Dolamic

. Automated detection of HONcode website conformity compared to manual detection: an evaluation. J Med Internet Res 2015; 17(6): e135.

34.

Boyer

Frossard

Gaudinat

, et al. How to sort trustworthy health online information? Improvements of the automated detection of HONcode criteria. Procedia Comput Sci 2017; 121: 940–949.

35.

Eysenbach

Köhler

. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. Bmj 2002; 324(7337): 573–577.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.40 MB