Abstract
Background:
Historically, there have been many factors that have influenced mumps, measles and rubella (MMR) vaccine uptake, including media bias, social/economic determinants, parental education level, deprivation and concerns over vaccine safety. Readability metrics through online tools are now emerging as a means for healthcare professionals to determine the readability of patient-facing vaccine information. The aim of this study was to examine the readability of patient-facing materials describing MMR vaccination, through employment of nine readability and text parameter metrics, and to compare these with MMR vaccination literature for healthcare professionals and scientific abstracts relating to MMR vaccination.
Materials and methods:
The subscription-based online Readable program (readable.com) was used to determine nine readability indices using various readability formulae: Established readability metrics (n = 5) (Flesch–Kinkaid Grade Level, Gunning Fog Index, SMOG Index, Flesch Reading Ease and New Dale-Chall Score), as well as Text parameters (n = 4) (sentence count, word count, number of words per sentence, number of syllables per word) with 47 MMR vaccination texts [patient-facing literature (n = 22); healthcare professional–focused literature (n = 8); scientific abstracts (n = 17)].
Results:
Patient-facing vaccination literature had a Flesch Reading Ease score of 58.4 and a Flesch–Kincaid Grade Level of 8.1, in comparison with poorer readability scores for healthcare professional literature of 30.7 and 12.6, respectively. MMR scientific abstracts had the poorest readability (24.0 and 14.8, respectively). Sentence structure was also considered, where better readability metrics were correlated with significantly lower number of words per sentence and less syllables per word.
Conclusion:
Use of these readability tools enables the author to ensure their research is more readable to the lay audience. Patient co-production initiatives would help to ensure that not only can the target audience read the literature, but that they understand the content. Increased patient-centric focus groups would give better insights into reasons for MMR-associated vaccine hesitation and vaccine refusal.
Background
There are many factors that can influence mumps, measles and rubella (MMR) vaccine uptake, including media bias, social and economic determinants such as parental education level, deprivation level of the area and concerns over the safety of the vaccine, as described previously. 1 It has been suggested that perception of susceptibility to disease also has an impact on parental decision-making, leading to either non-vaccination or failure to complete the vaccination schedule as evidenced in a report by MacDonald et al., 2 which concluded access to a regular healthcare practitioner, arrangements for childcare and parents who have been transient at some stage during the vaccination programme to be factors.
Health literacy, which may be impacted by parental education level, can be defined as the capacity of the individual to acquire, digest and comprehend basic information and services needed to make suitable decisions regarding their health. 3 In the age of digital public access to many information sources, aimed at both the general public and the healthcare professional, the general public may become overwhelmed with the volume and complexity of the material available. According to Flesch, 4 public information sources are only as good as the individual’s ability to grasp the content. It is imperative, therefore, that these materials are written in a manner that is easily digestible to facilitate understanding, therefore helping guide the decision-maker to an informed conclusion.
Readability scoring is used to quantify the difficulty of written language, giving an estimation of how accessible to the population any given piece of text is. 5 The more general audience the text is aimed at, the more important the readability factor of the literature becomes. 6 Flesch Reading Ease (FRE), Flesch–Kinkaid Grade Level (FKGL), Simple Measure Of Gobbledygook Index (SMOG), the Gunning Fog Index and the New Dale-Chall Score are examples of the formulae used to determine the readability score.
It was therefore the aim of this study to examine the readability of patient-facing materials describing MMR vaccination, through employment of nine readability and text parameter metrics, and to compare these with MMR vaccination literature for healthcare professionals, as well as with scientific abstracts relating to MMR vaccination.
Methods
Retrieval of MMR literature and examination of readability indices using readability tools
Open-access literature (n = 47) related to MMR vaccination aimed at the general public and healthcare professionals in the United Kingdom, Northern Ireland (NI), the Republic of Ireland (ROI), the United States, Australia and Canada, and various public health sources were searched and retrieved. All information was sourced using PubMed, MEDLINE, Google Scholar National Health Service (NHS) websites and governmental websites from the various geographical regions. Keywords used in the search for relevant material included ‘vaccine uptake’, ‘vaccine hesitancy’, ‘vaccine safety information’, ‘vaccine information for parents’, ‘incidence’, ‘correlation’, and ‘measles, mumps and rubella’.
Determination of readability metrics
The subscription-based online Readable program (readable.com) was used to determine nine readability indices using various readability formulae for the following: Established readability metrics (n = 5) (FKGL, Gunning Fog Index, SMOG Index, FRE and New Dale-Chall Score), as well as Text parameters (n = 4) (sentence count, word count, number of words per sentence, number of syllables per word). These readability metrics were selected for investigation as they are usually quoted by most readability papers. Each piece of literature was duly retrieved and converted to a text-only format in a Microsoft Word document, with all images and non-essential text such as headers or hyperlinks removed before being subjected to readability analysis.
Statistical analyses
A Shapiro–Wilk test, performed using SPSS, was applied to check that the data conformed to a normal distribution to ensure all data were normally distributed before performing two-tailed, unpaired t tests. For all tests, p < 0.05 (5%) was considered statistically significant.
Ethical statement
This study did not involve any human subjects or animals and therefore did not require ethical approval. All data presented in this study were taken from information in the public domain.
Results
Analysis of 47 MMR vaccination texts was performed, including (1) patient-facing literature (n = 22), healthcare professional–focused literature (n = 8) and scientific abstracts (n = 17), and scored using various readability parameters, as shown in Tables 1–3. All data sets were found to be normally distributed, and for comparison, two-samples unpaired t tests were performed and showed statistically significant differences for all nine readability metrics (combined) (p = 0.007) when comparing patient-facing and healthcare professional materials, as well as (p = 0.0002) when comparing patient-facing materials with scientific abstracts. When specifically examining patient-facing materials and healthcare professional materials, as well as patient-facing materials and scientific abstracts for both the FKGL and the FRE, all comparisons were statistically different (p < 0.0001). For patient-facing MMR resources, the mean FKGL, FRE, SMOG and Gunning Fog scores were 8.1, 58.4, 10.6 and 9.9, respectively. As sentence structure is linked to readability formulae, sentence structure was also considered, where better readability metrics were correlated with significantly lower number of words per sentence and less syllables per word. For an overview of this relationship between sentence structure and readability, please refer to Table 4.
Readability grading of MMR patient-focused materials (n = 22).
MMR, mumps, measles and rubella; ROI, Republic of Ireland; SMOG, Simple Measure Of Gobbledygook.
Readability grading of healthcare professional–focused materials (n = 8).
MMR, mumps, measles and rubella; ROI, Republic of Ireland; SMOG, Simple Measure Of Gobbledygook; WHO, World Health Organization.
Readability grading of MMR vaccination scientific abstracts (n = 17).
MMR, mumps, measles and rubella; SMOG, Simple Measure Of Gobbledygook.
A selection of four readability scoring systems and their relationship with sentence structure.
Discussion
To examine whether the readability of publicly available MMR vaccination information sources could have a bearing on the decision to vaccinate, an analysis was carried out on a selection of patient-focused literature, healthcare professional–focused literature and scientific abstracts (Tables 1–3).
For patient-facing material, the FKGL score achieved the target grade level of 8.1, Gunning Fog and SMOG indices showed the material to be above the target level by 2 and 3 grades, respectively, and New Dale-Chall scoring placed patient information at 5.9, well below the target grade.
As stated in a report by Nutbeam, 7 an individual’s reading capability is usually dependant on their level of education, and FRE is used to evaluate the level of education needed to read a specific text and is scored usually between 0 and 100, with a lower score being considered more difficult. An acceptable score for public consumption is deemed to be between 60 and 70 to reach the education level of 13- to 15-year-olds. 8 FKGL produces a reading grade level which is based on both word and sentence length, with difficulty increasing in line with grade level, and a score of 8 is considered to be accessible for the majority of the population, as it is suitable for those aged 13–14 years. 9 The Gunning Fog Index produces a grade level between 0 and 20 reflecting the years spent in formal education (from US grade 4 to college level) required to comprehend the text.
Public literature should produce a score of 8 which is suitable for a US grade 8 reading level (age 13–14 years). 10 An acceptable score is 8, which is that of a US grade 8 reading level. 11 The New Dale-Chall scoring index is based on a list of familiar words, and as with the previous readability tests, the score translates to a specific reading grade level, and a New Dale-Chall score of 8 is acceptable as suitable for the general public. 12
FKGL and FRE have been found to underestimate the grade reading level by 2–3 grades, while SMOG is considered best suited for health applications as it is consistent, more robust and assumes 100% comprehension. 13
The above readability formulae were chosen for this study as according to Ley and Florio, 14 they are the most commonly used formulae in healthcare to evaluate the readability of information widely accessible to the general public.
Comparisons drawn between UK and non-UK government patient-facing material indicated UK sources to have significantly lower readability scores. This indicates that although the patient-facing material was found to be above the target grade level, the UK governmental sources have a superior readability score than non-UK governmental sources. In a 2021 study by Clarke et al., 15 UK patient resources used for aiding decision-making in the treatment of latent tuberculosis infection were also found to have significantly lower readability than the non-UK counterparts.
For the majority of readability formulae used to test the material, it was found that on average all patient-facing materials are above the target level; nevertheless, readability formulae have their limitations and have been criticised as they are seen as only quick analytical measures to target information for public consumption, assessing only the reading level, not the patients’ understanding of the text. 8 Readability formulae also do not consider background knowledge of the topic. The medical nature of the materials being evaluated also raises readability scoring due to the complexity of the text and scientific vernacular required. With the exception of the SMOG Index, the majority of the readability formulae have been devised for general reading and are not optimal for use with medical or scientific-focused material. Large and complicated medical terms can push the readability score over the target threshold, leaving room for improvement, and in the future, it may be possible to design more bespoke formulae aimed at medical literature.
Going forward, it would be recommended that all patient-facing materials be analysed using both the FKGL and the FRE indices, with the SMOG Index as the gold standard, with a recommended reading grade level of between US sixth and eighth grades. This target grade level has been arbitrarily set previously by health institutions, including the National Work Group on Cancer and Health, the American Medical Association and the National Institutes of Health, while the Centers for Disease Control and Prevention (CDC) recommends the readability to be lower than eighth-grade level. 16
Study limitations
There are several limitations with using readability formulae to assess how readable narrative text is in patient-facing literature. Grade reading level recommendations are arbitrary and not based on strong evidence. We advocate that the FRGL target should be between sixth and eighth grades to improve readability; however, we appreciate that this may be difficult to accomplish in real-world scenarios, given that readability metrics struggled to accomplish favourable outcomes at eighth grade. Nevertheless, this lack of achieving our targets should not be a deterrent from attempting to improve readability by reducing the grade level and should remain a target, and solutions are sought to help achieve this readability aspiration. Another way to help achieve this would be to focus more on supporting developers to create accessible, health literate information, as well as evaluate or improve the health literacy demands of health information, such as the Patient Education Material Assessment tool. Indeed, some institutions (such as the Agency for Healthcare Research and Quality [AHQR]) already recommend a grade reading level of 4–6 for easy-to-read materials (www.ahrq.gov/talkingquality/resources/writing/tip6.htm). Furthermore, it is unknown whether changes in grade reading levels actually result in any meaningful change in outcomes such as improved knowledge/comprehension.
In conclusion, the readability of vaccine information sources varied across the different types of materials available. Material oriented towards the general public, such as NHS information leaflets or web pages, had an improved readability score than information aimed at the healthcare professionals, which in turn was lower than the information found in scientific abstracts. It can be noted, however, that information sources aimed at the general public, on average, have a readability score above the target level, and this should be addressed by aiming for a desirable reading FKGL of between sixth and eighth grades, to accommodate a wider population. Material available from health facilities could also be produced at different reading age levels, and the local knowledge of the general practitioner or healthcare provider could ensure that the most appropriate material for the individual was available.
