Quality and readability of online information on hand osteoarthritis

Abstract

Objectives: This study aimed to evaluate the quality and readability of web pages providing information about hand osteoarthritis using several authorized methods.

Methods: A web page exploration was performed using the Google internet search engine. The three search terms, “hand osteoarthritis,” “finger osteoarthritis,” and “hand OA,” were used and the top 100 ranked websites were selected and divided into six categories. The Health on the Net Foundation (HON) grade scale, an instrument for judging the quality of written consumer health information on treatment choice (DISCERN instrument), and the Ensuring Quality Information for Patients (EQIP) score were used to evaluate the quality of each website. The Flesch-Kincaid reading ease (FRE) score, Flesch-Kincaid grade (FKG) level, Gunning-Fog index, and Simple Measure of Gobbledygook grade level were used to evaluate website readability.

Results: Among 300 websites, 57 websites were selected following exclusion criteria. News portal websites, including the online version of newspapers and periodicals, showed the highest score in all three quality evaluation tools. Only four websites were regarded as high-quality websites based on the HON grade scale (n = 3) and the EQIP score (n = 1). Each type of website showed an average FKG level higher than 7^th grade and obtained an average FRE score of less than 80 points, indicating an inappropriate level for a layperson to read.

Conclusions: The online information about hand osteoarthritis is low quality and difficult to read for the general public. There is a need to enhance the quality and readability of web-based information related to hand osteoarthritis for patients to obtain credible information and receive proper treatment for the disease.

Introduction

Osteoarthritis is a disease that is gradually increasing in prevalence in an aging society. ¹ Hand osteoarthritis, in particular, is known to have a higher prevalence than other types of osteoarthritis, ² and more than 60% of the general population over 55 years of age showed evidence of arthritis in at least one hand joint on X-ray. ³ Although, hand osteoarthritis may be asymptomatic despite findings on imaging, it may affect hand function and activities of daily living, as well as increase the socioeconomic burden. ⁴ Therefore, evidence-based education is required to improve understanding of hand osteoarthritis.

In this modern age, the internet has become an indispensable tool for information retrieval and knowledge creation. In the past, most knowledge could be acquired through books or printed documents, but now it can easily be acquired through web-based digital information. In particular, due to the COVID-19 pandemic, most scientific conferences, as well as classes and lectures for the general public, are being conducted on online platforms. Thus, the internet plays a crucial role in modern-day learning.

Such web-based information is important not only in quantity but also in terms of quality. The term “Quality” of web-based information refers to the accuracy and transparency of website data, as well as the availability of the best and most up-to-date scientific evidence. ⁵ It also refers to the appropriateness of website contents and the well-organized pictures and diagrams of the websites. Unlike scientific publications, web-based content is often posted on the web without being verified. Moreover, content in medical field is difficult for general public to understand and evaluate, and the possibility of errors in the content is high. Thus, there is a need for web-based information with accurate and appropriate content that the general public can easily read.

Despite the high prevalence and importance of hand osteoarthritis, only one study assessed the quality and readability of online information about the topic. ⁶ The study, however, had several limitations. First, it only addressed one of several joints in the hand, the thumb carpometacarpal joint (CMCJ). Second, it failed to employ a reliable evaluation tool to assess the readability and quality of online content. Third, the study did not include the most recently updated website. Accordingly, the study covering all hand joints and using various validated quality and readability evaluation tools had become necessary.

This study aimed to evaluate the quality and readability of web pages providing information about hand osteoarthritis using several authorized methods.

Materials and methods

Website collection

This was an observational study that did not handle human participants or human-derived materials. Thus, approval from the institutional review board was not required. We used the Google Chrome browser and Google (https://www.google.com/) internet search engine to find the website. Google is the most widely used internet search engine, which comprises over 90% of the total internet search engine market worldwide. ⁷ We searched for target websites using three search terms, “hand osteoarthritis,” “finger osteoarthritis,” and “hand OA,” on 10 July 2021. The top 100 ranked websites found using the three search terms were selected. Because previous website search history can affect website searching, cookies, web caches, and webpage browsing histories were deleted before the study and between each search session. Inaccessible websites, duplicate websites, non-English websites, websites that contained information irrelevant to hand osteoarthritis, websites that were linked to scientific articles or video clips, and websites that required registration/subscription to access their content were excluded from this study. Among a total of 300 websites, 57 websites were finally selected (Figure 1).

Figure 1.

A flow diagram of the website selection and evaluation process.

The quality of all selected websites was individually scored by three experts of the upper extremity musculoskeletal system (SWH, EK, HJP), who were blinded to each other using three website quality evaluation tools. The quality scores of each website were evaluated twice by each of the three experts. If all six scores matched, the scores were adopted. Otherwise, each quality score was decided via agreement by the three experts. Inter-observer reliabilities of the assessments were evaluated by an intra-class correlation coefficient (ICC) using 2-way random effects and absolute agreement with the mean of the multiple measurements model (ICC [2, k]). On the other hand, intra-observer reliabilities were evaluated by an ICC using 2-way random effects and absolute agreement with the single measurement model (ICC [2, 1]). For intra-observer reliability, each quality score was evaluated at a 2-weeks interval to ensure that the assessments were independent.

Website categorization

The website was divided into six categories (i.e., academic, news portal, healthcare professional, non-expert, commercial, and non-profit) via consensus between two authors (SWH and JHK). Academic websites were those affiliated with a hospital, graduated school, medical society, or medical publisher. News portal websites were those that provided news or information including the online version of newspapers or periodicals. Healthcare professional websites were those operated by healthcare professionals such as physicians and physical therapists. Non-expert websites were those managed by laypersons that were not associated with healthcare providers. Commercial websites were those that sell products, display advertisements, or provided services to make a profit. Non-profit websites were those operated by the government or supported by donations and were not established for the purpose of profit-making.

Website quality evaluation

The Health on the Net Foundation (HON) grade scale, an instrument for judging the quality of written consumer health information on treatment choice (DISCERN instrument), and the Ensuring Quality Information for Patients (EQIP) score were used to evaluate the quality of each website. The HON criteria were developed to monitor the transparency of information and the purpose of websites in addressing their healthcare content. The HON grade scale is a 16-point scoring system which was designed to evaluate the key elements of the HON criteria. ⁸ The DISCERN instrument was developed to provide internet users a confidential way to rate the quality of written health information. ⁵ It consisted of 16 questions that were rated on a scale of 1 (poor) to 5 (good quality). Website quality is measured by adding the scores of 16 questions and can have a total of 80 points. The EQIP score is a 20-item practical measurement tool for written healthcare information, and its validity and reliability have been proven. ⁹ The EQIP is graded from 0% to 100%, with a higher percentage indicating higher quality information. Websites that received more than 75% of the maximum achievable score for each evaluation tool were considered high-quality.^8,10

Website readability evaluation

The Flesch–Kincaid reading ease (FRE) score, Flesch–Kincaid grade (FKG) level, Gunning–Fog index, and Simple Measure of Gobbledygook (SMOG) grade level were used to evaluate website readability. An online readability calculator (https://www.webfx.com/tools/read-able/) was used to evaluate the scores of each readability tool. For a comprehensive analysis, we used all written information on the website to evaluate readability. The FRE score is a well-known readability evaluation tool with scores ranging from 0 to 100, with a high score indicating better readability. The FKG level is a modified version of the FRE score which represents the average US school grade level that can read the given text. The FRE scores ranging between 90–100, 60–70, and below 30 indicate that the text is suitable for reading by 5^th grade, 8^th to 9^th grade, and graduate-level students (i.e., very difficult to read), respectively. ¹¹ (Table 1) Generally, text that is readable for the 6^th grade and below is recommended for the optimal level of health literacy.^12,13 Therefore, a website with an FKG level of less than 7^th grade and an FRE score of more than 80 was considered to have acceptable readability in this study. The Gunning-Fog index and SMOG grade level also measured the average US school grade.

Table 1.

Interpretation of Flesch reading ease score and Flesch-Kincaid grade level.

FRE score	Reading difficulty	FKG level
91–100	Very easy	5
81–90	Easy	6
71–80	Fairly easy	7
61–70	Standard	8–9
51–60	Fairly difficult	10–12
31–50	Difficult	13–16
0–30	Very difficult	College graduate

Statistical analysis

A Shapiro–Wilk normality test showed that the data from the present study were not normally distributed. Therefore, nonparametric tests were adopted. The Kruskal–Wallis test was used to compare the differences in quality evaluation scores and readability scores according to each website type, while the Dunn–Bonferroni test was used for post-hoc analysis. Pearson’s correlation analysis was used to estimate correlations among seven evaluation scores. Because 7, 10, and 21 simultaneous comparisons were performed in the Kruskal–Wallis test, in the post-hoc analysis, and in the Pearson’s correlation analysis, respectively, a Bonferroni correction was applied. An adjusted p < 0.007 (0.05/7, Kruskal–Wallis test), p < 0.005 (0.05/10, post hoc analysis), and p < 0.002 (0.05/21, Pearson’s correlation analysis) were taken to indicate statistical significance. All statistical analyses were performed using the SPSS software (ver. 24.0; SPSS Inc., Chicago, IL, USA).

Results

Classification of website and reliability evaluation

A total of 57 websites were classified into six academic websites, 10 news portal websites, 12 healthcare-professional websites, 12 commercial websites, and seven non-profit websites (Table 2). The ICC (2, 1) for intra-observer reliabilities of the three quality evaluation items measured by each observer and the ICC (2, k) for inter-observer reliabilities of the three quality evaluation items are shown in Table 3. Since ICC values greater than 0.75 were considered to indicate good reliability, ¹⁴ the intra- and inter-observer reliabilities of the quality evaluations were acceptable.

Table 2.

Quality and readability evaluation scores of each website.

Type of website	Number of websites	Quality evaluation scores			Readability evaluation scores
		HON grade scale	EQIP score (%)	DISCERN instrument	FRE score	FKG level	Gunning-fog index	SMOG grade level
Academic	6 (10.5%)	5.50 (3.75–6.50)	35.00 (22.50–47.50)	32.00 (27.25–41.25)	57.45 (46.25–64.58)	8.75 (7.95–10.55)	11.80 (10.30–14.10)	8.90 (7.85–10.33)
News portal	10 (17.5%)	10.00 (8.50–11.00)	46.25 (42.50–50.63)	39.50 (31.00–43.50)	44.30 (32.78–53.23)	12.45 (9.28–14.53)	16.10 (12.50–17.48)	11.50 (9.38–12.93)
Healthcare professional	12 (21.1%)	5.00 (4.25–6.00)	31.25 (25.00–38.75)	30.00 (27.25–35.00)	56.45 (48.25–59.93)	10.25 (9.25–11.63)	13.5 (12.33–14.78)	9.80 (9.20–10.90)
Non-expert	0 (0%)	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Commercial	22 (38.6%)	6.00 (4.00–8.00)	32.50 (29.38–41.25)	32.00 (29.75–37.25)	48.45 (36.95–55.03)	11.35 (9.78–12.85)	14.40 (11.90–15.90)	10.60 (9.25–11.83)
Non-profit	7 (12.3%)	5.00 (4.00–7.00)	27.50 (25.00–50.00)	36.00 (29.00–42.00)	56.30 (47.50–61.60)	9.70 (9.00–11.30)	13.10 (11.00–14.20)	9.90 (8.50–10.60)
Overall median value	57 (100%)	6.00 (5.00–8.00)	35.00 (27.50–45.00)	32.00 (29.00–39.50)	52.20 (43.85–58.95)	10.30 (9.20–12.25)	13.60 (11.90–15.50)	10.00 (9.10–11.60)
p value for kruskal-wallis test		0.002 *	0.079	0.162	0.066	0.051	0.075	0.076
Post hoc anlaysis		News portal –Healthcare professional	—	—	—	—	—	—

HON grade scale, The Health on the Net Foundation grade scale; EQIP score, the Ensuring Quality Information for Patients score; DISCERN instrument, an instrument for judging the quality of written consumer health information on treatment choice; FRE score, Flesch-Kincaid reading ease; FKG level, Flesch-kincaid grade level; SMOG grade level, Simple Measure of Gobbledygook grade level.

^*p < 0.007 by Kruskal-Wallis test.

*

Descriptive values are shown as median (interquartile range) or number of cases (proportion).

Table 3.

Intra-observer and inter-observer reliabilities of the quality evaluation of the websites.

Items	ICC (2, 1) for intra-observer reliability			ICC (2, k) for inter-observer reliability
	Observer 1	Observer 2	Observer 3
HON grade scale (16 points)	0.929	0.940	0.885	0.919
EQIP score	0.960	0.972	0.912	0.948
DISCERN instrument	0.957	0.981	0.892	0.939

ICC (2, 1), intra-class correlation coefficient (ICC) using 2-way random effects and absolute agreement with the single measurement model; ICC(2, k), ICC using 2-way random effects and absolute agreement with the mean of multiple measurements model.

Website quality and readability evaluation

The results of the quality evaluation score according to each website type are shown in Table 2. News portal websites showed the highest scores in terms of the HON grade scale, EQIP score, and DISCERN instrument. In addition, these had a statistically significant difference in the HON grade scale. A total of four websites were considered high-quality based on the HON grade scale (n = 3) and EQIP score (n = 1). No website was considered high-quality based on the DISCERN instrument (Table 4). Across all websites, the median FRE score and FKG level were 52.20 [Interquartile range (IQR), 43.85–58.95] and 10.30 (IQR, 9.20–12.25), respectively, whereas the mean Gunning–Fog index and SMOG grade level were 13.60 (IQR, 11.90–15.50) and 10.00 (IQR, 9.10–11.60), respectively (Table 2). Each type of website showed an average FKG level of higher than 7^th grade and obtained an average FRE score of less than 80 points, indicating difficult readability. The differences in readability scores according to website type were not statistically significant. Statistically significant correlations were founded between each three quality scores, and four readability scores were also significantly correlated with each other. However, three quality scores were not significantly correlated with four readability scores except between the HON grade scale and the SMOG grade level (Table 5).

Table 4.

Number of high-quality websites.

Type of website	Number of websites	Number of high-quality websites
		HON grade scale	EQIP score	DISCERN instrument
Academic	6	0	0	0
News portal	10	2	0	0
Healthcare professional	12	0	0	0
Non-expert	0	0	0	0
Commercial	22	1	1	0
Non-profit	7	0	0	0
Total	57	3	1	0

HON grade scale, The Health on the Net Foundation grade scale; EQIP score, the Ensuring Quality Information for Patients score; DISCERN instrument, an instrument for judging the quality of written consumer health information on treatment choice.

Table 5.

Pearson’s correlation coefficient between quality and readability evaluation scores.

	HON grade scale	EQIP score (%)	DISCERN instrument	FRE score	FKG level	Gunning-fog index	SMOG grade level
HON grade scale	1.000	0.619*	0.719*	−0.366	0.387	0.399	0.420*
EQIP score (%)		1.000	0.818*	−0.264	0.210	0.223	0.262
DISCERN instrument			1.000	−0.093	0.072	0.098	0.117
FRE score				1.000	−0.929*	−0.885*	−0.925*
FKG level					1.000	0.972*	0.974*
Gunning-fog index						1.000	0.978*
SMOG grade level							1.000

HON grade scale, The Health on the Net Foundation grade scale; EQIP score, the Ensuring Quality Information for Patients score; DISCERN instrument, an instrument for judging the quality of written consumer health information on treatment choice; FRE score, Flesch-Kincaid reading ease; FKG level, Flesch-kincaid grade level; SMOG grade level, Simple Measure of Gobbledygook grade level.

^*p < 0.002 by Pearson’s correlation coefficient.

Discussion

Hand osteoarthritis is a representative degenerative disorder characterized by pain and decreased range of motion of the hand joints. It requires comprehensive management because it considerably affects the quality of life. However, since hand osteoarthritis is considered an inevitable result of aging, ¹⁵ patients may neglect its symptoms and end up not receiving appropriate treatment. ¹⁶ To overcome these limitations and provide appropriate treatment, accurate information about hand osteoarthritis should be conveyed to patients. Due to the increasing internet utilization rate of patients over 50 years (the main morbid age group for hand osteoarthritis), ¹⁷ there is also an increasing need to provide information and education on hand osteoarthritis through the internet. Since web-based information is a highly effective tool for patient education, ¹⁸ it is important to manage and evaluate its quality and readability.

According to our evaluation, two news portal websites and two commercial websites were high-quality websites. Furthermore, most websites presented with a low degree of credibility and had an unacceptable quality of information. These may be related to the lack of providing well-organized information related to treatment for hand osteoarthritis, as well as the lack of citations and the time of production. If patients with hand osteoarthritis use such poor-quality information, they may have difficulty in understanding their disease and in determining the next steps to take, such as seeking a consult for proper treatment.

Websites operated by healthcare providers had lower quality evaluation scores than news portals or commercial websites. This is a different result from that of previous studies which evaluate the quality of information about the disease other than hand osteoarthritis such as diabetic retinopathy and clavicle fracture.^19,20 In these studies, websites operated by healthcare providers had better website quality than other types of websites. Generally, healthcare providers provide better quality and more accurate health-related information. However, in this study, the news portal websites and commercial websites received higher scores in terms of information delivery and ease of understanding information than websites operated by healthcare providers. Therefore, webpages authored by healthcare providers need to improve in both quality and readability for online information on hand osteoarthritis. Furthermore, the disparity in results between this study and previous studies could be attributed to the use of different website quality evaluation tools. The evaluation items for each quality evaluation tool are quite different, as are the weights of the evaluation items.

While a readability level of 6^th grade or below is recommended for optimal health literacy, the current study found that all types of websites had a readability level of 7^th grade or higher. Therefore, all types of websites were written with difficult-to-read content that was not suitable for conveying medical information to the general public. This result was not different from that of previous studies that dealt with the readability of various medical information such as distal radius fracture, dupuytrens’ contracture, and femoroacetabular impingement.^21–23 The use of longer sentences for the expression of medical information seems to complicate information which contributes to lower readability scores. Moreover, the predominant age group for hand osteoarthritis is the elderly population, and this age group tends to have limited literacy. Therefore, when creating web content, concise and well-organized writing should be considered and health professionals should provide evidence-based information that is structured and has improved readability.

Except for one relationship between the HON grade scale and SMOG grade level, the results showed no significant correlations between three readability scores and four quality scores. Detailed medical explanations would be required to create high-quality medical webpage as the accuracy and appropriateness of the web contents influence the quality of webpage. During this process, difficult medical terms and longer sentences that reduce readability may be used. Therefore, several efforts, such as developing simple medical terminology and simplifying sentence structure would be required to improve web content readability while maintaining appropriate web content quality.

The results from the present study showed that there were strong and statistically significant correlations between readability indices. These findings could suggest that, although, there are differences in the way of measuring readability, each readability index evaluated the readability of the webpages similarly and yielded equivalent results. Therefore, understanding readability indices and selecting an appropriate readability index while evaluating the readability of medical webpages.

Websites operated by laypersons were not included in the top 100 ranked websites during our research. This may be because hand osteoarthritis, despite its high prevalence, has not received as much attention from the general public compared to more common diseases such as hypertension and diabetes. This could also be attributed to the recent decrease in the public interest in hand osteoarthritis in the aftermath of the COVID-19 pandemic. Thus, it is necessary to educate laypersons that cases of hand osteoarthritis require proper management, similar to other chronic diseases such as diabetes or hypertension. This also can be achieved by making it easier to access web pages related to hand osteoarthritis through internet searches, and by providing adequate content on self-management of hand osteoarthritis.

There were several limitations of this study. First, we only used one search engine and limited our study to English websites only. The available information on the internet changes dynamically and is written in various languages. Therefore, the web pages evaluated in this study do not reflect all of the information on the internet. Second, although reliability was evaluated in the website quality analysis process and sufficient reliability was verified, we cannot exclude the possibility of subjectivity by the observer. Third, the internet browser used in this study may not reflect the mobile users’ preference. Fourth, in this study, the website quality was assessed using each quality evaluation tool separately. A future study would be required to evaluate the website’s quality by combining multiple quality evaluation tools.

Conclusions

The online information about hand osteoarthritis had low quality and was difficult to read for the general public. The prevalence of hand osteoarthritis is increasing alongside an aging society, with more patients experiencing reduced quality of life due to its related complications. Therefore, it is necessary to enhance the quality and readability of web-based information related to hand osteoarthritis. Easy retrieval systems for the patients should be created to help them obtain credible information and encourage evidence-based management.