Readability and Quality of Online Health Information Regarding Parathyroidectomy

Abstract

Objective

Assessment of the readability and quality of online health information regarding parathyroidectomy.

Study Design

Cross-sectional analysis.

Setting

Websites providing patient-oriented health information regarding parathyroidectomy obtained via the Google search engine.

Methods

The top 75 Google search results for “parathyroidectomy,”“parathyroid surgery,” and “parathyroid gland removal” were reviewed. Websites were categorized by website type and country of origin. Readability was assessed by Flesch-Kincaid Grade Level and Simple Measure of Gobbledygook. Website quality was assessed per JAMA benchmark criteria and the DISCERN instrument.

Results

A total of 74 unique websites were evaluated. The mean readability of the assessed websites exceeded the recommended sixth-grade reading level on the Flesch-Kincaid Grade Level and Simple Measure of Gobbledygook (P < .001). Readability did not vary significantly by website type. Websites originating from the United Kingdom were significantly more readable than those from the United States. The majority of assessed websites were of poor quality (n = 42, 56.8%) on assessment based on the DISCERN instrument. Quality varied significantly by website category on the JAMA benchmark criteria (P < .001) and DISCERN score (P = .049) with commercial websites receiving the highest scores. DISCERN score also varied significantly by country of origin (P = .036) with UK sites receiving highest mean DISCERN scores.

Conclusion

Online health information regarding parathyroidectomy is largely of poor quality and is poorly readable for many patients. Institutions utilizing well-defined guidelines for development of patient educational resources may provide online health information of greater quality and readability.

Keywords

parathyroidectomy endocrine surgery health information readability quality online internet

Parathyroidectomy is the only treatment offering a definitive cure for primary hyperparathyroidism, and it may be indicated in cases of secondary and tertiary hyperparathyroidism refractory to medical management.^1-3 Although rare, complications of parathyroidectomy, including recurrent laryngeal nerve injury, permanent hypoparathyroidism, and treatment failure, occur in up to 5% of cases and may cause significant morbidity.^4,5 Health literacy has been shown to affect patient outcomes, and high-quality health information may enhance patient education prior to surgery.^6-8 The internet is an important source of health information and is utilized by the majority of patients across all age groups.^9-11 Indeed, the internet has become the first resource used to obtain health information for most patients, and only information originating from health professionals and government agencies is considered more trustworthy.^10,12

For patient-oriented health information to be useful, it must be readable and of sufficient quality. The readability of written health information is a measure of the ease with which it can be read and understood. Approximately 1 in 5 adults in the United States possesses low literacy skills, and to ensure that health information is readable for patients, guidelines from the American Medical Association and Agency for Healthcare Research and Quality recommend the sixth-grade reading level as the upper limit of readability for patient information.^13-15 Numerous studies have demonstrated, however, that the readability of online health information often exceeds the recommended grade level and is too complex for general public use.^16-19

As the provision of online health information is largely unregulated, its quality may vary widely and is poor in many instances.^20-22 Identifying reliable online sources of health education may be difficult for many patients, and those with poor health literacy are less likely to successfully identify inaccurate or misleading health information.^23,24 Identification of high-quality sources of online education regarding parathyroidectomy is therefore essential to ensure that patients obtain reliable information.

The aims of this study were to assess the readability and quality of online health information related to parathyroidectomy. We hypothesized that the majority of online resources would exceed the recommended sixth-grade reading level. We also postulated that the readability and quality of online information would vary depending on website type and country of origin.

Methods

Search Strategy and Website Classification

A Google search was performed on October 10, 2021, for the terms “parathyroidectomy,”“parathyroid surgery,” and “parathyroid gland removal.” The top 75 results from each search were recorded and duplicates removed. Websites requiring subscription, containing video only, presented in a language other than English, or comprising information irrelevant to parathyroidectomy were excluded. Resources aimed at health care professionals, including journal articles and medical reference texts, were also excluded. Where information on parathyroidectomy was contained on a number of pages from the same resource, all relevant pages were assessed. Per the protocols set out by the University Ethics Committee of University College Cork, institutional review board approval was not required for this study, as it did not involve human subjects. The study was exempt from ethical approval by the Clinical Research Ethics Committee of the Cork Teaching Hospitals.

Websites included for analysis were categorized according to country of origin as originating from the United States, United Kingdom, or other countries. Websites were also categorized by website type based on classifications previously described.^25,26 Academic sites were those associated with an academic institution or medical society. Commercial sites were those funded by advertising, industry sponsorship, or online sales. Physician sites originated from individual physicians or health care organizations not associated with an academic or governmental organization. Nonphysician sites were provided by individuals without declared medical qualifications. Nonprofit sites were those relying solely on governmental funding or charitable donations for funding. News-oriented sites were defined as those containing nonmedical news and included social media websites.²⁶

Readability Assessment

The Flesch-Kincaid Grade Level (FKGL) formula and Simple Measure of Gobbledygook (SMOG) index were selected for assessment of readability of online health information and were completed with an online readability analysis tool (readable.com). The FKGL assesses readability by examination of average sentence length and syllables per word, with results expressed as the US grade level of education required to understand the text.^27,28 The FKGL is a commonly used instrument for assessment of online health information and was selected to allow comparison with previous research.²⁹ The SMOG formula assesses readability through examination of polysyllabic words in 30 sentences selected from a text, with results presented as US grade level of education required to allow complete comprehension.^30,31 The SMOG index is well validated and is recommended for use in assessment of health-related information.²⁹

Quality Assessment

Quality of online health information related to parathyroidectomy was completed with the JAMA benchmark criteria and DISCERN instrument. Assessment was completed by 2 reviewers (J.B. and S.K.) and mean scores calculated for analysis.

The 4 JAMA benchmark criteria are authorship, attribution, disclosure, and currency, which may be utilized for assessment of the quality and credibility of online health care information.³² Websites with higher JAMA scores have been shown to be less likely to contain inaccurate health information.³³ One point was allocated for each criterion met, with a maximum score of 4.

The DISCERN instrument is designed to assess the quality of health care information and consists of 16 individual items. Each item is scored on a 5-point scale, yielding a minimum score of 16 and a maximum score of 80.³⁴ Websites were classified according to their final scores on the DISCERN instrument as very poor (16-28), poor (29-41), fair (42-54), good (55-67), or very good (68-80). The DISCERN has been widely used for assessment of online health information and has demonstrated satisfactory validity and reliability.^35,36

Statistical Analysis

Descriptive statistics were calculated for FKGL, SMOG grade, JAMA score, and DISCERN score. One-sample t test was used to compare mean values for the FKGL and SMOG grade to the recommended sixth-grade reading level. Agreement between reviewers for JAMA and DISCERN scores was assessed with the intraclass correlation coefficient (ICC). One-way analysis of variance was used to determine if mean readability and quality scores varied by website category or country of origin. Post hoc analysis with Tukey’s test was performed to assess between category differences in mean scores. P < .05 was considered significant for all analyses. All statistical analysis was performed with SPSS Statistics (version 27; IBM).

Results

A total of 225 websites were screened, with 74 unique websites included for analysis after exclusions and removal of duplicates. Characteristics of evaluated websites are outlined in Table 1 .

Table 1.

Website Characteristics.

	No.	%
Website category
Physician	30	40.5
Academic	29	39.2
Nonprofit	8	10.8
Commercial	7	9.5
Country of origin
US	57	77.0
UK	12	16.2
Other countries	5	6.8

Flesch-Kincaid Grade Level

The mean ± SD FKGL of assessed websites was 9.895 ± 2.381, exceeding the recommended sixth-grade reading level by 3.895 grade levels (P < .001; 95% CI, 3.343-4.446); 71 websites (95.9%) exceeded the recommended sixth-grade reading level. Mean FKGL did not vary significantly by website category (P = .172; Table 2 ). The greatest between-category difference in mean FKGL occurred between physician and nonprofit websites but did not achieve statistical significance (mean difference, 1.841; 95% CI, −0.617 to 4.299; P = .209; Supplemental Table S1, available online). Mean FKGL varied significantly by country (P = .002) with sites from the United States receiving the highest mean scores. Sites from the United States scored significantly higher than those from the United Kingdom (mean difference, 2.157; 95% CI, 0.472-3.842; P = .009) but not those from other countries (mean difference, 2.407; 95% CI, −0.067 to 4.881; P = .058). The mean FKGL of sites from the United Kingdom and other countries did not significantly differ (mean difference, 0.250; 95% CI, −2.573 to 3.073; P = .976).

Table 2.

Readability Grades and Quality Scores by Website Category and Country of Origin.

	FKGL		SMOG grade		JAMA score ^a		DISCERN score ^a
	Mean	SD	Mean	SD	Mean	SD	Mean	SD
Website category
Academic	9.521	2.706	12.04	2.010	1.190	1.047	35.259	10.086
Physician	10.55	2.056	12.85	1.585	0.717	0.568	32.433	6.115
Commercial	9.971	1.582	12.59	1.443	3.0	1.155	41.857	8.915
Nonprofit	8.713	2.482	11.18	1.417	2.25	1.035	39.125	13.082
Total	9.895	2.381	12.327	1.791	1.284	1.132	35.155	9.268
Country of origin
US	10.407	2.1937	12.684	1.592	1.167	1.115	33.658	7.972
UK	8.250	2.454	11.133	2.215	1.542	1.215	40.458	12.243
Other	8.000	1.790	11.120	1.281	2.000	0.935	39.500	11.219
Total	9.895	2.381	12.327	1.791	1.284	1.132	35.155	9.268

Abbreviations: FKGL, Flesch-Kincaid Grade Level; SMOG, Simple Measure of Gobbledygook.

Values presented as mean scores from 2 independent raters.

SMOG Grade

The mean SMOG grade was 12.327 ± 1.791, exceeding the recommended sixth-grade reading level by 6.327 grade levels (P < .001; 95% CI, 5.912-6.742). All evaluated websites exceeded the sixth-grade reading level. Mean SMOG grade did not vary significantly by website category (P = .079; Table 2 ). The highest between-category difference in mean SMOG grade occurred between physician and nonprofit websites (mean difference, 1.672; 95% CI, −0.153 to 3.497; P = .084), but no significant between-category differences were detected on post hoc analysis (Supplemental Table S2, available online). Mean SMOG grade varied significantly by country of origin (P = .006) with websites from the United States scoring significantly higher than those from the United Kingdom (mean difference, 1.551; 95% CI, 0.267-2.835; P = .014). No significant difference in mean SMOG grade was noted between sites from the United States and other countries (mean difference, 1.564; 95% CI, −0.321 to 3.449; P = .123) or between sites from the United Kingdom and other countries (mean difference, 0.013; 95% CI, −2.138 to 2.165; P > .99).

JAMA Benchmark Criteria

The ICC for JAMA score was 0.879 (95% CI, 0.807-0.924), indicating excellent reliability.³⁷ The overall mean score was 1.284 ± 1.132, with only 1 website achieving the maximum score of 4. JAMA score varied significantly by website category (P < .001; Table 2 ). Commercial sites scored significantly higher than academic and physician sites but not nonprofit sites. Nonprofit sites scored significantly higher than academic and physician sites. No significant difference in score was found between academic and physician sites ( Table 3 ). JAMA score did not vary significantly by country of origin (P = .2) with no between-category differences identified on post hoc analysis.

Table 3.

Post Hoc Comparison of JAMA Score by Website Category and Country of Origin.

Comparison	Mean difference	95% CI	SE	P value
Website category
Academic vs physician	0.473	−0.138 to 1.084	0.232	.184
Academic vs commercial	−1.810	−2.798 to −0.823	0.375	<.001
Academic vs nonprofit	−1.060	−1.997 to −0.124	0.356	.02
Physician vs commercial	−2.283	−3.267 to −1.299	0.374	<.001
Physician vs nonprofit	−1.533	−2.466 to −0.600	0.355	<.001
Commercial vs nonprofit	0.750	−0.463 to 1.963	0.461	.370
Country of origin
US vs UK	−0.375	−1.228 to 0.478	0.356	.547
US vs other	−0.833	−2.086 to 0.420	0.523	.256
UK vs other	−0.458	−1.888 to 0.972	0.597	.724

DISCERN Score

The ICC for DISCERN score was 0.852, indicating excellent agreement.³⁷ The mean DISCERN score for all websites was 35.155 ± 9.268. The majority of sites were classed as poor (n = 42, 56.8%), with only 5 sites (6.8%) classed as good and none classed as very good ( Table 4 ). DISCERN score varied significantly by website category (P = .049; Table 2 ), with commercial sites obtaining the highest mean DISCERN score (41.857 ± 8.915). No statistically significant differences in DISCERN score were identified between individual website categories, with the greatest difference occurring between physician and commercial websites (mean difference, −9.424; 95% CI, −19.31 to 0.464; P = .067; Supplemental Table S3, available online). Mean DISCERN score varied significantly by country of origin (P = .036), with sites from the United Kingdom achieving the highest mean DISCERN score. The greatest difference in mean DISCERN score occurred between sites from the United States and the United Kingdom (−6.80; 95% CI, −13.6 to 0.018; P = .051), but this did not achieve statistical significance.

Table 4.

Website Classification by DISCERN Score.

DISCERN classification	No.	%
Very good	0	0.0
Good	5	6.8
Fair	8	10.8
Poor	42	56.8
Very poor	19	25.7

Discussion

The internet has become the first port of call for those seeking health information, and online sources of health education are trusted by many patients. For online health information to be beneficial for patients, however, it must be easily comprehensible and of high quality. This is particularly important for patients seeking information on surgical procedures, as comprehension of the indications, complications, and possible outcomes of surgical intervention is essential to ensuring truly informed consent. Additionally, provision of adequate patient education may affect outcomes after surgery.⁷ Unfortunately, the results of the current study indicate that online patient information related to parathyroidectomy is generally of poor quality and is difficult to understand for a large proportion of the general public at which it is aimed.

The findings of the current study indicate that the readability of the majority of online health information related to parathyroidectomy exceeds the recommended sixth-grade level and therefore may exceed the reading ability of many patients. This finding is consistent with numerous previous studies examining the readability of online resources regarding surgical procedures.^19,38-41 Additionally, our results concur with previous research examining the readability of online information on parathyroidectomy suggesting that the readability of these resources has not appreciably improved in the intervening period.⁴²

SMOG grade was consistently higher than FKGL across all website categories. This was also previously demonstrated, as SMOG grade assumes complete comprehension of assessed material while FKGL assumes approximately 75% comprehension. Indeed, SMOG grade may be a better tool to assess the readability of health information where greater levels of comprehension are desirable.^29,43

While nonprofit websites were consistently more readable than those from other sources in the current study, readability did not vary significantly by website category, as demonstrated in previous studies.²⁶ Country of origin did appear to influence readability, as resources from the United Kingdom tended to be more readable than those from the United States or other countries. The majority of assessed UK websites originated from sources affiliated with the National Health Service (NHS), and their greater readability may be explained by well-developed NHS guidelines and tools for development of patient-centered digital and education resources.⁴⁴ Development of such institutional guidelines may be beneficial in ensuring accessibility, consistency, and readability of patient educational resources. Additionally, when patients are directed to sources of online health information, recommendation of resources developed with a well-defined framework may enhance patient comprehension of the relevant aspects of their surgical procedure.

In addition to being poorly readable, the results of this study indicate that online information regarding parathyroidectomy is often of poor quality, with the majority of websites obtaining low scores on the DISCERN instrument and JAMA benchmark criteria. This finding is consistent with the results of previous studies demonstrating wide variability and often poor quality of online health information on surgical procedures and surgical disorders, including thyroidectomy, surgery for colorectal cancer, breast cancer, and vascular surgery.^20,33,45,46 Interestingly, academic and physician websites obtained the lowest average quality scores, suggesting that health care providers may be failing to make high-quality online health information available to their patients. The quality of information provided by commercial websites, however, appears higher. The reasons for this are unclear, but websites not affiliated with a particular health care institution may have greater freedom to outline a range of treatment options with their risks and benefits. Sites from health care providers, though, may be more likely to provide information on only the procedures available at their institutions. The quality of online health information also appeared to vary significantly by country of origin, with sites from the United Kingdom obtaining the highest DISCERN scores. Once again, employment of NHS guidelines in the development of the majority of these resources may have contributed to the greater quality of online health information obtained from UK websites.

The current study has a number of limitations. First, search results generated with Google may vary depending on the location of the user. To ensure that analyzed websites provided a representative sample of online health information regarding parathyroidectomy, the top 75 websites for each search term were screened for eligibility. Second, increased syllable counts associated with terms such as “hyperparathyroidism” and “parathyroidectomy” may have resulted in artificially elevated FKGL and SMOG grades without necessarily affecting the readability of this online health information. Additionally, simplification or substitution of complex medical terminology may not be appropriate and may not aid in improving the readability and comprehensibility of health information. While Google searches for “parathyroid gland removal” and “parathyroid surgery” were included for assessment, it is unclear whether websites with these terms had improved readability scores. The FKGL and SMOG were selected, however, due to their widespread use for assessment of health information, thus allowing comparison with previously published research. Additionally, only English-language sources were selected for analysis, as the readability formulas selected for use were developed for assessment of written materials in English.^27,30 Adaptations of these formulas for use in other languages exist, but there appears to be little evidence regarding their comparability to the FKGL and SMOG grade.

A further limitation of our study is the absence of an assessment of which online resources were deemed most valuable by patients. While the DISCERN score is a validated means of assessing the quality of health information, it is possible that resources with poor ratings on it may still have been deemed useful or valuable by patients.

Finally, while websites containing video only were excluded from analysis in the current study, a small number of sites contained video in addition to written material. Information contained in these videos was considered during assessment of website quality, but their effect on readability could not be assessed. Additionally, there is a paucity of well-validated instruments for assessment of health information quality presented in video format. Use of the DISCERN instrument and JAMA benchmark criteria for this purpose has been described, but both instruments lack validity and reliability for the assessment of video content.^47,48

Conclusion

The internet is an important and easily accessible source of health information for patients. Online information on parathyroidectomy, however, appears to be of poor quality and is poorly readable in many instances. Information originating from the United Kingdom appears to combine improved readability and quality. While the reasons for this are unclear, utilization of institutional guidelines for development of online patient resources may contribute to improved comprehensibility and quality of online health information. When patients are directed to online health resources related to parathyroidectomy, identification of sources adhering to such guidelines may be recommended. Online patient resources must be designed with the health literacy and reading abilities of their target population in mind to enhance accessibility and patient comprehension.

Author Contributions

Jim Byrne, study concept and design, acquisition of data, analysis and interpretation of data, drafting of manuscript, critical revision; Samuel Keogh, acquisition of data, analysis of data, drafting of manuscript; Carolyn Cullinane, interpretation of data, drafting of manuscript, critical revision; Zeeshan Razzaq, interpretation of data, critical revision; Henry Paul Redmond, study concept and design, critical revision.

Disclosures

Competing interests: None.

Sponsorships: None.

Funding source: None.

Supplemental Material

sj-docx-1-opn-10.1177_2473974X221133308 – Supplemental material for Readability and Quality of Online Health Information Regarding Parathyroidectomy

Supplemental material, sj-docx-1-opn-10.1177_2473974X221133308 for Readability and Quality of Online Health Information Regarding Parathyroidectomy by Jim Byrne, Samuel Keogh, Carolyn Cullinane, Zeeshan Razzaq and Henry Paul Redmond in OTO Open: The Official Open Access Journal of the American Academy of Otolaryngology-Head and Neck Surgery Foundation

Footnotes

ORCID iD

Jim Byrne

Supplemental Material

Additional supporting information is available at

References

Wilhelm

Wang

Ruan

, et al. The American Association of Endocrine Surgeons guidelines for definitive management of primary hyperparathyroidism. JAMA Surg. 2016;151(10):959-968.

Steinl

Kuo

JH.

Surgical management of secondary hyperparathyroidism. Kidney Int Rep. 2021:6:254-264.

Tang

Friedman

Hwang

, et al. Parathyroidectomy for tertiary hyperparathyroidism: a systematic review. Am J Otolaryngol. 2017;38:630-635.

Singh Ospina

Roeriguez-Gutierrez

Maraka

, et al. Outcomes of parathyroidectomy in patients with primary hyperparathyroidism: a systematic review and meta-analysis. World J Surg. 2016;40(10):2359-2377.

Norman

Lopez

Politz

Abandoning unilateral parathyroidectomy: why we reversed our position after 15,000 operations. J Am Coll Surg. 2012;214(3):260-269.

Burgess

Arundel

Wainwright

TW.

The effect of preoperative education on psychological, clinical and economic outcomes in elective spinal surgery: a systematic review. Healthcare. 2019;7(1):48.

Ramesh

Nayak

Pai

, et al. Effect of preoperative education on postoperative outcomes among patients undergoing cardiac surgery: a systematic review and meta-analysis. J Perianesth Nurs. 2017;32(6):518-529.

Wright

Edwards

Goggins

, et al. Association of health literacy with postoperative outcomes in patients undergoing major abdominal surgery. JAMA Surg. 2018;153(2):137-142.

Estacio

Whittle

Protheroe

The digital divide: examining socio-demographic factors associated with health literacy, access and use of internet to seek health information. J Health Psychol. 2019;24(1):1668-1675.

10.

Finney Rutten

Blake

Greenberg-Worisek

, et al. Online health information seeking among US adults: measuring progress toward a healthy people 2020 objective. Public Health Rep. 2019;134(6):617-625.

11.

Crouch

Gordon

NP.

Prevalence and factors influencing use of internet and electronic health resources by middle-aged and older adults in a US health plan population: cross-sectional survey study. JMIR Aging. 2019;2(1):e11451.

12.

Health Information National Trends Survey. HINTS briefs number 39: trust in health information among American adults. May 2019. Accessed December 15, 2021. https://hints.cancer.gov/docs/Briefs/HINTS_Brief_39.pdf

13.

OECD. Skills matter: additional results from the survey of adult skills. November 2019. Accessed December 20, 2021. https://www.oecd.org/skills/piaac/publications/Skills_Matter_Additonal_Results_from_the_Survey_of_Adult_Skills_ENG.pdf

14.

Weiss

BD.

Health Literacy: A Manual for Clinicians. American Medical Association Foundation and American Medical Association; 2003.

15.

Brega

Barnard

Mabachi

, et al. AHRQ Health Literacy Universal Precautions Toolkit. 2nd ed. Agency for Healthcare Research and Quality; 2015.

16.

Miles

Baird

Choi

, et al. Readability of online educational material related to breast lesions requiring surgery. Radiology. 2019;291(1):112-118.

17.

Szmuda

Ozdemir

Ali

, et al. Readability of online patient education material for the novel coronavirus disease (COVID-19): a cross-sectional health literacy study. Public Health. 2020;185:21-25.

18.

Fefer

Lamb

Shen

, et al. Multilingual analysis of the quality and readability of online health information on the adverse effects of breast cancer treatments. JAMA Surg. 2020;155(8):781-784.

19.

Garfinkle

Wong-Chong

Petrucci

, et al. Assessing the readability, quality and accuracy of online health information for patients with low anterior resection syndrome following surgery for rectal cancer. Colorectal Dis. 2019;21(5):523-531.

20.

Wasserman

Baxter

Rosen

, et al. Systematic review of internet patient information on colorectal cancer surgery. Dis Colon Rectum. 2014;57(1):64-69.

21.

Cassidy

Baker

JF.

Orthopaedic patient information on the world wide web: an essential review. J Bone Joint Surg Am. 2016;98(4):325-338.

22.

Bailey

Coughlin

Sohrabi

, et al. Quality and readability of online patient information for abdominal aortic aneurysms. J Vasc Surg. 2012;56(1):21-26.

23.

Schulz

Pessina

Hartung

, et al. Effects of objective and subjective health literacy on patients’ accurate judgement of health information and decision-making ability: survey study. J Med Internet Res. 2021:23(1):e20457.

24.

Diviani

van den Putte

Giani

, et al. Low health literacy and evaluation of online health information: a systematic review of the literature. J Med Internet Res. 2015;17(5): e112.

25.

Mathur

Shanti

Brkaric

, et al. Surfing for scoliosis: the quality of information available on the internet. Spine. 2005;30(23):2695-2700.

26.

Broderick

McCarthy

Hogan

Osteotomy around the knee: assessment of quality, content and readability of online information. Knee. 2021;28:139-150.

27.

Kincaid

Fishburne

Rogers

, et al. Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for navy enlisted personnel. Institute for Simulation and Training. January 1975. https://stars.library.ucf.edu/istlibrary/56/

28.

Shedlosky-Shoemaker

Sturm

Saleem

, et al. Tools for assessing readability and quality of health-related web sites. J Genet Couns. 2009;18(1):49-59.

29.

Wang

Miller

Schmitt

, et al. Assessing readability formula differences with written health information material: application, results, and recommendation. Res Social Adm Pharm. 2013;9(5):503-516.

30.

McLaughlin

GH.

SMOG grading: a new readability formula. J Read. 1969;22:639-664.

31.

Friedman

Hoffman-Goetz

A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav. 2006;33(3):352-373.

32.

Silberg

Lundberg

Musacchio

RA.

Assessing, controlling, and assuring the quality of medical information on the Internet: caveant lector et viewor—let the reader and viewer beware. JAMA. 1997;277(15):1244-1245.

33.

Meric

Bernstam

Mirza

, et al. Breast cancer on the world wide web: cross sectional survey of quality of information and popularity of websites. BMJ. 2002;324(7337):577-581.

34.

Charnock

The DISCERN Handbook: Quality Criteria for Consumer Health Information on Treatment Choices. Radcliffe Medical Press; 1998.

35.

Rees

Ford

Sheard

CE.

Evaluating the reliability of DISCERN: a tool for assessing the quality of written patient information on treatment choices. Patient Educ Couns. 2002;47(3):273-275.

36.

Ademiluyi

Rees

Sheard

CE.

Evaluating the reliability and validity of three tools to assess the quality of health information on the internet. Patient Educ Couns. 2003;50(2):151-155.

37.

Cicchetti

DV.

Guidelines, criteria, and rule of thumb for evaluating normed and standardised assessment instruments in psychology. Psychol Assess. 1994;6(4):284-290.

38.

Hansberry

Agarwal

Shah

, et al. Analysis of the readability of patient education materials from surgical subspecialties. Laryngoscope. 2014;124(2):405-412.

39.

Prieto

West-Santos

Montgomery

, et al. Patient-oriented online resources in pediatric surgery: are we failing the readability test? J Pediatr Surg. 2020;55(10):2048-2051.

40.

Behmer Hansen

Gold

Lad

, et al. Health literacy among neurosurgery and other surgical subspecialties: readability of online patient material found with Google. Clin Neurol Neurosurg. 2020;197:106141.

41.

Rai

Landsberg

Nguyen

, et al. Online educational materials for appendectomy patients have good quality but poor readability. Am J Surg. 2021;221(6):1203-1210.

42.

Patel

Sanghvi

Cherla

, et al. Readability assessment of internet-based patient education materials related to parathyroid surgery. Ann Otol Rhinol Laryngol. 2015;124(7):523-527.

43.

Meade

Smith

CF.

Readability formulas: cautions and criteria. Patient Educ Couns. 1991;17(2):153-158.

44.

National Health Service. NHS digital service manual: content style guide. Accessed December 10, 2021. https://service-manual.nhs.uk/content

45.

Chen

Harris

Jia

, et al. Search trends and quality of online resources regarding thyroidectomy. Otolaryngol Head Neck Surg. 2021;165(1):50-58.

46.

Grewal

Alagaratnam

The quality and readability of colorectal cancer information on the internet. Int J Surg. 2013;11(5):410-413.

47.

Azer

SA.

Are DISCERN and JAMA suitable instruments for assessing YouTube videos on thyroid cancer? Methodological concerns. J Cancer Educ. 2020;35(6):1267-1277.

48.

Gabarron

Fernandez-Luque

Armayones

, et al. Identifying measures used for assessing quality of YouTube videos with patient health information: a review of current literature. Interact J Med Res. 2013;2(1):e6.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.02 MB

0.00 MB