Abstract
Background
Evidence-based nutrition education resources are one way to help registered dietitians (RDs) translate scientific knowledge to consumers.
Aim
To develop a checklist based on suitability assessment of materials (SAM) and to assess its use to refine nutrition education resources.
Methods
RDs were recruited online to assess two nutrition education resources using SAM. Three rounds of surveying and two rounds of resource refinements occurred. A “checklist” was created to refine the resources between rounds. Descriptive statistics and nonparametric tests were performed to explore differences in SAM-scores between rounds.
Results
RDs participated in the first (n = 45), second (n = 37), and third (n = 27) surveys. SAM-scores significantly improved in both resources by the third round. The refined checklist included more explicit instructions and provided examples to help guide resource changes.
Conclusions
Using the checklist improved SAM scores. Future work should include end-users to help with checklist validation.
Introduction
Registered dietitians (RDs) are health professionals who use nutrition education resources, such as fact sheets, to communicate scientific information and support client understanding. To ensure these materials are effective, they must be appropriate for the intended audience. The suitability assessment of materials (SAM), by Doak, Doak, and Root in 1996, is a tool used to evaluate patient-education materials (Doak et al., 1996). It assesses these materials based on several key factors: content clarity and focus, literacy demand aiming for a fifth-grade reading level or lower, quality and relevance of graphics, layout and typography, learning stimulation and motivation, and cultural appropriateness. By considering these aspects, SAM ensures that educational resources are accessible, understandable, and effective for diverse audiences. Research has shown that the SAM tool demonstrates acceptable internal consistency, with a Cronbach’s alpha coefficient of 0.75 (Anderson et al., 2014).
Despite its utility, SAM is often used only as an evaluation tool, with limited guidance on how to revise materials once deficiencies are identified. This creates a gap between assessment and actionable improvement. To our knowledge, no formal, dietitian-developed checklist exists to help systematically revise nutrition education resources based on SAM findings. The objectives of this project were to develop a checklist based on SAM and assess its use to refine nutrition education resources originally developed in 2016 by the Canadian Sugar Institute.
Methods
This is a mixed-methods instrument development and evaluation study. Participants were English-speaking Canadian RDs who were recruited from social media and listservs between August 2022 and October 2023. After consenting, RDs completed an online demographic questionnaire through Qualtrics ® (Provo, UT). They evaluated two nutrition education resources (Supplemental Figure 1) that were originally developed in 2016 on the topic of sugar and health using SAM, which asks about (a) content, (b) literacy demand, (c) numeracy, (d) graphics, (e) layout/typography, and (f) learning stimulation/motivation using a 3-point Likert Scale 0 (not suitable), 1 (adequate), or 2 (superior). The RDs were informed that the resources were intended for use by dietitians. The co-authors did not independently assess the tools using the SAM instrument prior to the study's launch.
Checklist development and tool refinement
After the first round (September to December 2022) (n = 45), data was analyzed and a checklist was developed by two researchers based on the components of SAM, adopted from Smith et al. (Smith et al., 2014) to ensure relevance for health-professional related resources. CAD (a RD not involved in the study) then used the checklist to refine the tools (Supplemental Figure 2). The same RDs were asked to re-score [(January to March 2023) Round 2, n = 37] the refined tools using SAM and the checklist was refined again based on SAM scores to improve its use. Specifically, the checklist was refined to improve organization and user-friendliness. Items were grouped into categories, and checkboxes were added to allow users to track their progress. Notes sections were added to document how each item was addressed or needed improvement. Finally, a section on “Target Audience and Branding” was also introduced, responding to feedback from the dietitians about the importance of understanding the intended audience and ensuring consistent branding throughout.
The results were analyzed again and CAD refined the tools again using the refined checklist (Version 2.0). A third survey [(June to October 2023) n = 27 RDs, 3 of which were “new”] was launched and CAD refined the tools one last time, at which point the checklist was refined and finalized (Figure 1). Specifically, the finalized checklist included the following updates: the addition of logos and branding. The notes sections were removed following this round of testing, as they were deemed unnecessary and made the resource overly lengthy; this adjustment streamlined the checklist to a concise two-page format for ease of use.
Resource Checklist.
At the end of Survey 3, RDs were asked whether they had participated in the previous surveys and whether they believed the refined resources were improvements on the original resources (yes/no).
Statistical analysis
Analyses were conducted in SAS® Studio (version 3.81). Descriptive statistics were used to summarize participant characteristics, and SAM scores. Next, the total SAM score was divided by the maximum total score and converted to a percentage to describe the scores as unsuitable (0–39%), adequate (40–69%), or superior (70–100%) [7]. SAM scores by category were analyzed separately whereby the scores were summed per category, and then divided them by the maximum SAM score and also converted these into percentages. Fisher's exact test of independence was used to determine the association between surveys and the SAM scores and Friedman's nonparametric test was used to assess for significance in the SAM score. Post-hoc analysis using the Dunn–Bonferroni method was conducted after Friedman's test to analyze the pairwise comparisons of the difference in means between any pair of surveys for which the means were significantly different. Finally, a Sensitivity Analysis using the Wilcoxon signed-rank test was conducted to assess for significant differences in results for Survey 3 with and without the n = 3 additional RDs; key findings did not differ therefore our results include those n = 3 RD. The statistical significance level was set at 0.05.
Results
This study included mostly women (n = 44) who have been practicing dietetics for >2 years (Table 1). Eighty-nine percent of the participants in Survey 3 had participated in the previous surveys.
Study participant count and frequency [n (%)].
Participants were given the option to select more than one area of practice. One participant in Survey 3 did not answer the question about time working as a registered dietitian.
The demographic information of the three new participants that were recruited in Survey 3 was not collected.
Table 2 displays the results of the SAM scores for each resource from each round. The total (mean ± standard deviation) SAM score of Resource 1 improved from 66.6 ± 16.0% (“adequate”) in Survey 1 to 78.8 ± 16.7% (“superior”) in Survey 3 (p = 0.007). Significant improvements were seen primarily for Writing Style 1 (p = 0.021), Vocabulary 3 (p = 0.024), Layout (p = 0.039) and Interaction (p < 0.0001). For Resource 2, total SAM scores significantly improved from Survey 1 [70.6 ± 18.9% (“superior”)] to Survey 3 [82.0 ± 16.6% (“superior”)] (p = 0.016), with most significant changes shown for the variables titled Quick Glance (p = 0.003), Types of Illustrations/Images (p = 0.008), Typography (p = 0.004), Interaction (p < 0.0001), and Cultural Images and Examples (p < 0.0001).
SAM ratings [frequency (%) and total SAM scores (mean ± SD) for nutrition education resources from Survey 1 to Survey 3 [n (%)]. a
Note: SAM: suitability assessment of materials; SD: standard deviation. SAM scores are graded as 0, 1, or 2, and total SAM score is reported as a percentage of points earned out of total possible points. *P < .05 considered statistically significant.
Friedman's nonparametric test comparing the mean scores of the three different surveys.
Post-hoc Dunn's test: means significantly different from Survey 1 when P-value is adjusted by the Bonferroni method.
Post-hoc Dunn's test: means significantly different from Survey 2 when P-value is adjusted by the Bonferroni method.
Writing Style 1: Use of conversational style and active voice.
Writing Style 2: Use of simple sentences.
Vocabulary 1: Use of common words.
Vocabulary 2: Use of technical, concept, category, value judgment words (CCVJ).
Vocabulary 3: Use of appropriate imagery words
Overall, 85.2% of the RDs ranked Resource 1 as having improved in comparison to the original version. One hundred percent of participants thought that Resource 2 had improved in comparison to the original version.
Discussion
Through the use of a checklist that was based on assessing the suitability of health resources, this study resulted in improved SAM scores for resources that were created by RDs for RDs to enhance their knowledge on sugars and health. This, along with others, highlights the importance of appropriately engaging the proper end-users or seeking expert opinions when creating resources that are intended for a specific population. The SAM tool was selected because it is a widely used, validated framework for evaluating the suitability of health education materials, offering a structured approach to assess key elements such as content clarity, literacy demand, and visual design.
In this case, this study focused on Canadian RDs. Our methods are supported by others: Williams et al. used SAM to assess and improve patient-education materials in ophthalmology using evidence-based recommendations. Following the refinements, two glaucoma specialists scored the materials independently using SAM (Williams et al., 2016). The readability score improved from a tenth-grade reading level to a sixth-grade reading level and the mean SAM score of the materials improved from 60 ± 7% (adequate) to 88 ± 4% (superior) (Williams et al., 2016). Another study had health education specialists and educational technologists use SAM to evaluate the suitability of adolescent educational material in preventing hookah smoking in adolescents (Sadeghi et al., 2019). After refinement, the modified materials were given to 10 young adolescents for reassessment of their suitability. All materials improved in their SAM scores after being tailored based on the experts’ initial ratings (Sadeghi et al., 2019).
Specific to this study, the improved SAM scores through the evolution of the resources could be attributed to the use of our checklist, which was refined to incorporate questions and practical information in an easy-to-read manner aligned with SAM. For example, Resource 2 showed significant improvements for “Summary/Review” (p = 0.031) likely attributed to the checklist's emphasis on including summaries. Williams et al. reported similar improvements to this variable after the researchers only included crucial content, identified action steps, and only used words common to individuals without medical training (Williams et al., 2016). In this study, both resources improved in Survey 3 because the content focused more on dietetic practice, as opposed to general nutrition information.
Another example worth highlighting is the “Quick Glance” variable within Graphic Material, which significantly improved (p = 0.009) for Resource 2. Similarly, Types of Illustrations/Images significantly improved from Survey 1 to 3. The checklist's four recommendations on the Graphic Material (e.g., to incorporate a high contrast between paper and type, to use bright and fun colours, and to ensure that graphics appear professional) appear to have led to this rise in the SAM score.
Similarly, Cultural Images & Examples showed a major improvement in Resource 2 from Survey 1 to Survey 3 (p < 0.0001), which can be attributed to the checklist ensuring images are culturally appropriate and “match” the focus of the resource. Williams et al. achieved similar improvements in this variable by implementing the recommendation to “make certain content is appropriate for the age and culture of the target audience” (Williams et al., 2016).
It is acknowledged that the variable “Interaction” remained relatively low in their SAM scores compared to other variables. The SAM criteria for the variable of Interaction suggests the inclusion of such problems or quizzes, which were not included in our Resources. Finally, the proportion of “superior” ratings for Reading Grade Level in both resources started high, decreased in Survey 2, and increased again in Survey 3. These changes may be a result of participants being uncertain of the target audience of the education resources.
This study's limitation includes the fact that only English-speaking RDs were surveyed, which lacked the perspective of other-language-speaking individuals. The time between surveys was also long, therefore RDs may not have remembered the previous versions as clearly, which may have impacted the ratings. We did not ask RDs if they had previous experiences with SAM, which could have biased our results. Dietitians were surveyed on their views of the resources related to sugars and health, therefore the interpretation of our findings is limited to the specific topic area. Expanding the scope of application to a wide range of nutrition topics is warranted. Our sample size was too small to conduct internal consistency or reliability testing. Finally, while SAM is a widely used tool for evaluating health education materials, it has limitations, including the subjective nature of its scoring. It was also developed for print-based resources and may not fully reflect how people access health information today, particularly online. Additionally, while this study focused on dietitians, future work should include testing with end-users, as their understanding and needs may differ significantly from those of health professionals.
This study reinforces the importance of engaging with experts or end-users when developing educational resources. Through the use of our checklist, our study highlights the importance of choosing appropriate colours and graphics when developing or refining tools as these small details can improve overall suitability scores. Using the SAM checklist, RDs were able to revise the patient education materials to align with SAM-specific criteria, ensuring the nutrition education resources are suitable for their intended audience. Future studies should use our checklist to assess if other nutrition or health education resources improve SAM scores after their use.
Supplemental Material
sj-pdf-1-nah-10.1177_02601060251365357 - Supplemental material for Enhancing nutrition education resources through the development and refinement of a checklist using the suitability assessment of materials (SAM)
Supplemental material, sj-pdf-1-nah-10.1177_02601060251365357 for Enhancing nutrition education resources through the development and refinement of a checklist using the suitability assessment of materials (SAM) by Oliver Sage, Flora Wang, Chiara DiAngelo, Sandra Marsden, Claudia Faustini, Shannan Grant and Tamara R Cohen in Nutrition and Health
Supplemental Material
sj-pdf-2-nah-10.1177_02601060251365357 - Supplemental material for Enhancing nutrition education resources through the development and refinement of a checklist using the suitability assessment of materials (SAM)
Supplemental material, sj-pdf-2-nah-10.1177_02601060251365357 for Enhancing nutrition education resources through the development and refinement of a checklist using the suitability assessment of materials (SAM) by Oliver Sage, Flora Wang, Chiara DiAngelo, Sandra Marsden, Claudia Faustini, Shannan Grant and Tamara R Cohen in Nutrition and Health
Supplemental Material
sj-docx-3-nah-10.1177_02601060251365357 - Supplemental material for Enhancing nutrition education resources through the development and refinement of a checklist using the suitability assessment of materials (SAM)
Supplemental material, sj-docx-3-nah-10.1177_02601060251365357 for Enhancing nutrition education resources through the development and refinement of a checklist using the suitability assessment of materials (SAM) by Oliver Sage, Flora Wang, Chiara DiAngelo, Sandra Marsden, Claudia Faustini, Shannan Grant and Tamara R Cohen in Nutrition and Health
Footnotes
Acknowledgments
We would like to thank all the registered dietitians who took time out of their day to participate in this study.
Authors contributions
Oliver Sage contributed to data curation, formal analysis, investigation, and writing‒original draft. Flora Wang contributed to conceptualization, funding acquisition, methodology, resources, supervision, and writing‒review and editing. Chiara DiAngelo contributed to conceptualization, funding acquisition, methodology, resources, supervision, and writing‒review and editing. Sandra Marsden contributed to funding acquisition and writing‒review and editing. Claudia Faustini contributed to writing and formal analysis‒review and editing. Shannan Grant contributed to methodology and writing‒review and editing. Tamara Cohen contributed to conceptualization, funding acquisition, methodology, project administration, supervision, and writing‒review and editing.
Availability of data and materials
All data and related metadata underlying the findings reported are included in the submitted article and its supplementary information files. Other data is available on request
Consent for publication
No identifying information is being published.
Declaration of conflicting of interests
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Wang, DiAngelo and Marsden (co-authors) are employees of the Canadian Sugar Institute which co-funded this project. The Canadian Sugar Institute is a nonprofit industry association that supports evidence-based nutrition research and communications. They were consulted in the development of the survey questionnaire and reviewed the manuscript. However, they are not involved in the data collection or analysis.
Ethical statement
This study was conducted according to the guidelines laid down in the Declaration of Helsinki and all procedures involving research study participants were approved by the University of British Columbia's Behavioural Research Ethics Review Board. Written informed consent was obtained from all participants.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the Mitacs Accelerate Program with the Canadian Sugar Institute (grant number: F22-00300), as the industry partner.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
