Abstract
Background
Text messaging campaigns by healthcare providers seem to increase patient engagement and compliance. As an emergent strategy, text messaging presents bidirectional challenges related to health literacy, culture, and linguistic fluency. A numeric rubric was developed to measure text messages’ alignment to federally mandated Culturally and Linguistically Appropriate Services (CLAS) and the National Institutes of Health (NIH) Clear Communication standards.
Objective
Use a CLAS rubric evaluate health-related text messages prior to or after distribution from healthcare providers to patients. Using standardized outcome ratings, offer feedback to improve text messages and subsequently increase reported positive patient outcomes.
Methods
Using the novel rubric, two bilingual researchers separately scored each message and conducted auditing to agree on message scores. The rubric considered cultural relevance, accessibility, linguistic accessibility, literacy level appropriateness, and clear action-oriented messaging. Each element was awarded a score out of 4 points, with 20 as a perfect score.
Results
Sixty-one outbound text messages were evaluated as part of a comprehensive evaluation for an urban health department health literacy campaign. Text message evaluation was conducted twice during the two-year campaign, with recommendations integrated during the project's second year. Nearly a third of text messages (31.5%) during the project's second year scored a 3.5 out of 4 or higher.
Conclusions
The campaign was generally consistent with NIH best practices and CLAS standards. Application of the rubric and scores before dissemination was most impactful for Federally Qualified Health Centers to improve the quality of health-related text messages.
Background
Arizona experienced one of the highest COVID-19 infection rates in the United States. The state reported over two million cases and more than 30,000 deaths when the described health literacy intervention was launched; at the time Hispanic, American Indian, Black, Asian, and Pacific Islander Arizonans accounted for 60.0% of COVID-19 deaths.
1
In Maricopa County, with minorities comprising 46.7% of the population,
2
positive infection rates were the highest among poor, minority communities and among those who had limited English proficiency, curtailing access to culturally and linguistically appropriate health information. To address this access gap, the Maricopa County Department of Public Health Literacy Project established the following goals:
Improve adherence to COVID-19 public health practices among socially vulnerable Maricopa County populations through equitable distribution of health information. Improve vulnerable communities’ access to and ability to use essential health information and services, which are impacted by concurrent social determinants of health. Enable equitable distribution of tailored health information to optimize adherence to health guidance, thus addressing health literacy and social determinants of health.
A text messaging campaign was implemented through a large urban health department in collaboration with Federally Qualified Health Centers to increase the uptake of COVID-19 vaccines and several health-related behaviors among individuals unvaccinated for COVID-19. Community Health Workers (CHW) played a crucial role in this campaign, leveraging their trusted status within the community to enhance communication, provide culturally and linguistically appropriate information, and support individuals in making informed health decisions.
Using a numeric rating scale, university-based researchers evaluated that text messages prior to or after distribution from healthcare providers to various patient groups. Using the numeric ratings, feedback was given for the purpose of improving text messages to subsequently increase reported patient action or compliance with several health-promoting behaviors.
Text messaging has emerged as a powerful tool in the healthcare industry, demonstrating significant potential in enhancing patient engagement and adherence to health-promoting behaviors. Research has shown that healthcare providers’ use of text messaging can effectively increase patient compliance and foster better health outcomes. For instance, a text messaging campaign facilitated higher COVID-19 vaccination rates among older adults by enabling easy scheduling of vaccination appointments. 3 The text message campaign offered older adults an opportunity to schedule a vaccine appointment via telephone or email, and 9.5% of over 30,000 text message recipients expressed interest in receiving a vaccine. Similarly, targeted text messages have been instrumental in boosting seasonal influenza vaccine uptake. 4 Utilizing over 47,000 text messages, the vaccination rate improved by an average 5%. Researchers suggested the text messages promoting influenza vaccines were most effective when paired with routine healthcare provider communications, rather than isolated or seemingly random communications.
In another study, a text messaging program successfully aided in the control of patients’ systolic blood pressure over a six-month period. 5 This study sent regular text messages to promote healthy habits over the long term; studies exceeding six months were not observably more effective than the six-month programs. Text messaging interventions have similarly improved health-related behaviors among diabetic patients, with particularly strong effects observed among Spanish-speaking individuals. 6 The most substantial improvement among enrolled patients was pertinent to medication adherence, in addition to potential (but not significant) improvements in HbA1C levels.
These successful examples underscore the possibility for health-promotion text messaging campaigns to positively influence the health of recipients. Existing literature offers promising approaches to augment ongoing healthcare activities with automated text messages. Text messaging campaigns can be especially impactful when thoughtfully designed to consider cultural factors, access issues, and relevant theoretical frameworks.
As an emerging health-promotion strategy, text messaging brings bidirectional challenges related to health literacy, culture, and linguistic fluency. Health literacy is concerned with recipients’ ability to receive, understand, and use health-related information and services to inform health-related decisions, and the senders’ ability to effectively transmit information given the recipients’ global context. 7 Cultural considerations involve understanding and respecting the diverse beliefs, values, and practices that influence how individuals perceive and engage with health information, while linguistic fluency encompasses the ability to communicate effectively in the recipient's preferred language. Consequently, health-promotion text messages should be accessible and understandable to recipients allowing them to learn and develop their health literacy, if desired. Therefore, healthcare providers communicating via text message (or any means) must consider the dynamic roles of all participants within a health information exchange.
Research also suggests a multilayered approach to communications including integration of health behavior theory, use of plain language, and delivery via trusted health messengers increases effectiveness. 8 For this reason, the text message campaign and associated evaluation developed for this project integrated a culturally reflexive approach applying universal federal standards, and delivered by live, accessible, bilingual CHWs. Use of CHWs has been shown to be effective in the prevention and mitigation of COVID-19, especially in areas with a large Hispanic community. 9 Community Health Workers’ embeddedness as community participants allows them to act as transformative and trusted agents of health information.
For this study, the outgoing text messages were evaluated for patient engagement in health-related conversations, motivation to schedule healthcare appointments, and acceptance of the COVID-19 vaccine. For example, patients due for their annual well-woman examination were sent a reminder message specific to scheduling an annual exam; patients reaching milestones in age were sent text messages about take-home colorectal screening tests. All messages were designed to be sent in the recipients’ preferred language, English or Spanish.
Outbound text messages were monitored for recipient response, and CHWs used a decision tree of responses to schedule appointments and answer general questions in English or Spanish when patients replied. Recipients who did not opt-out and engaged in messaging eventually received a message specific to COVID-19 vaccination, after all immediate healthcare needs were met. Patients could then schedule a vaccine or ask questions to receive evidence-based information.
Methods
Evaluators developed a numeric rubric to measure outgoing text messages’ alignment to federally mandated Culturally and Linguistically Appropriate Services (CLAS) 10 and the National Institutes of Health (NIH) Clear Communication standards. 11 Existing tools such as the Patient Education Materials Assessment Tool and the Suitability of Materials assessment were considered, but ultimately these validated instruments do not emphasize cultural or linguistic variability to the extent needed for this project and patient population. Evaluation researchers worked closely with health department staff and clinic patient advocates to create a rubric that represented the full spectrum of community needs.
The custom rubric for this project consisted of five elements including (1) culture and (2) accessibility under social dynamics, (3) linguistic accessibility, (4) literacy level under readability, and (5) clear, action-orientation under NIH standards. These elements were specifically chosen to cover the breadth of concerns addressed by CLAS and NIH standards. Elements were measured on a scale of 1 to 4: 1 = insufficient, 2 = needs work, 3 = adequate, and 4 = ideal (see Table 1).
Text message evaluation rubric.
Social dynamics
The element of culture considered how well the message was tailored to the recipient community, and to what extent the cultural preferences of each community were considered. Cultural elements were related to the target audience (i.e., persons lacking cancer screenings, older adults, Spanish speakers, or unvaccinated patient groups). Accessibility of all audiences to equitably benefit from the message was included in the rubric. This element screened for measures that made the text messages more accessible for action, such as the ability to reply and get a live person or integration of clickable hyperlinks to websites and phone numbers.
Readability
Linguistic accessibility considers the number of locally spoken languages in which the messages were transmitted, as well as the quality of their interpretation and dialect. Generic translations were considered inadequate; effective translations were customized and considered for the recipient community. The other element of readability was literacy level. Researchers looked for a fifth-grade reading level or below, and that all information not commonly understood was simply defined.
National Institutes of Health standards
Based on the NIH recommendation that all health messaging be simple, clear, and direct, 11 researchers looked for messages free of distraction and superfluous information, and that were action oriented. Messages were also evaluated for being free of erroneous symbols, numbers, or abbreviated script, including text message shorthand. Furthermore, information that did not provide the reader with a next step was less desirable. Evaluators looked for text messages including an actionable directive the reader could immediately implement to potentially improve or protect their health.
The numeric rubric presented was one component of comprehensive evaluation conducted to measure success of the text messaging campaign. The purpose of the rubric was to quickly apply a numeric scale to evaluate the quality of outgoing, community-specific, mass text messages prior to or after distribution in a manner which is consistent, and based on federal standards. Outgoing text messages were analyzed by two evaluation researchers who independently scored each text message in English and Spanish according to the rubric. Each CLAS or NIH element (cultural relevance, accessibility, linguistic accessibility, literacy level appropriateness, and clear action-oriented messaging) was awarded a score out of 4 points, with 20 as a perfect message score.
After independent scoring, the evaluators met to share and audit their score on each outbound text message with discussion focused on areas of divergence in text message scores. If a message diverged by one point or less, an average score was used. If the scores differed by more than one point, researchers each provided rationale and context for their specific score. Typically, the discussion would lead to one researcher understanding an application of the rubric that was previously missed, and one researcher adjusted their score. In each case agreement was reached, and the two researchers were able to rescore the element within one point of one another to yield an appropriate average score. In many cases, the scores were similar and did not require any additional discussion.
To ensure consistent, context-specific interpretation of the scoring rubric, this approach leveraged a collaborative, qualitative method between two bilingual researchers rather than quantitative methods such as statistical analysis or significance testing. As a qualitative study, presenting descriptive statistics, confidence intervals, and significance testing was not possible or appropriate due to the small sample size of messages and the limited number of ratings assigned to each text message. Rather, the evaluation project engaged two experienced qualitative researchers. Researcher one carried an MPH degree, and 15 years’ experience in qualitative methods and evaluation, in addition to teaching about limiting bias and subjectivity in research approaches at the graduate level. Researcher two had a PhD and five years’ experience in evaluation and qualitative research and was a native Spanish speaker. The evaluation approach was approved and monitored by a PhD qualitative researcher with over 40 years’ experience.
Researcher inter-rater reliability was informally assessed through independent scoring followed by in-depth verbal discussions. Researchers systematically reviewed and deliberated on each statement, ensuring interpretation aligned with the scoring, based on a scale of 1 to 4. This consensus-driven data auditing approach allowed for a comprehensive understanding of the criteria and ensured the evaluation of each domain was grounded in shared reasoning and mutual agreement within the local context.
Results
Sixty-one outbound text messages were evaluated as part of a comprehensive community health literacy campaign at an urban health department. The messages were selected independently by each clinic from a list generated by the health department. In total, 4722 patients engaged via text messaging by responding to a message. Responses to text messages were monitored and addressed by CHWs. Patients were able to make medical appointments and ask questions live with CHWs during regular business hours; off-hours messages were responded to on the next business day.
Twenty-three text messages were distributed during year 1 of the project and 38 during year 2. Outgoing text message evaluation was conducted twice during the two-year campaign, with evaluative recommendations integrated prior to the project's second year. Data shown are for year 2. Percentages are used for comparison, even though the sample size was too small to measure statistical significance.
Year 2 text messages were scored using the rubric and cross analyzed by health literacy domain and clinic. English and Spanish messages earned the same scores and were integrated for presentation. Clinic names are withheld, and the narrative will reference Clinics 1, 2, and 3 for the participating Federally Qualified Health Centers.
The average score for all year 2 messages was 3.33, representing an increase from the year 1 average score of 3.25 for 23 text messages sent. Nearly a third of text messages (31.5%) during the project's second year scored a 3.5 out of 4 or higher. Eight messages (21.1%) scored a 3.4 and seven text messages (18.4%) scored a 3.3, accounting for nearly 40% of all text messages distributed earning placement in the 3.3–3.4 range. Lastly, four messages received a 3.2, two a 3.1, two a 3, two a 2.9, and one message earned a 2.8, with over a quarter of messages (28.9%) scoring in the poorest range of 2.8 to 3.2 (see Figure 1).
Percentage of year 2 messages earning each score (n = 38).
Culture
When analyzed by rubric element, the text messages distributed during year 2 scored highest in the element of culture. Overall, the text messages earned an average 3.9 in culture accessibility and reflexivity. This score was an increase from year 1 (3.9). Researchers observed that messages used a casual, friendly approach, similar to a text message from a friend, fostering a warm reception from patients, and meeting the criteria in the cultural element. The majority of the 38 outgoing year 2 text messages (84.2%, n = 32) scored a 4, and the remainder a 3.5.
Accessibility
The year 2 text messages distributed earned an average score of 3.3 in the accessibility element, an increase from the year 1 score (3.1). Most messages scored a 3 because they only had one accessibility measure, the built-in communication capability with a CHW. Additions of clickable links earned some messages a higher score.
Linguistic accessibility
Linguistic accessibility was the lowest scoring area for the year 2 outbound text messages, with an average score of 2.5 for all messages. This score was similar to year 1 (2.6). Two messages were not translated into Spanish. To ensure the 2.5 score was not a false low score brought down by the lack of two messages not being translated, the other 36 messages (translated messages) were independently analyzed. The mean score remained a 2.6 with the untranslated messages omitted.
Compared to year 1, aspects of the year 2 text messages improved, with fewer grammatical errors, increasing use of concordant Spanish language masculine and feminine subject agreement, and correct use of articles. Nevertheless, incorrect verb conjugation agreement remained where messages switched back and forth from the formal and informal tenses, which can be confusing for the reader.
Most of the poorly scoring messages were generically translated from Spanish. These messages seemed to be machine-generated, rather than colloquially and locally translated, or at least vetted by a member of the Sonoran native Spanish-speaking community. Some messages used century-old dialects, or dialects commonly used in Spain. Many local Spanish speakers (who are often not completely literate) do not understand these terms. While best practice dictates that health communications are translated by a local, native speaker, researchers did not create the messages and therefore were not aware whether the messages were machine translated or translated according to best practice.
All messages appropriately translated into Spanish earned a 3. There were no messages translated into additional languages at the end of year 2; therefore, there were no messages that earned a perfect score of 4 out 4 in linguistic accessibility.
Literacy level
The literacy level was quite appropriate for most of the outbound text messages, with an average score of 3.4 overall. This represents an increase from year 1 (3.3). Eleven messages scored a perfect 4, and 14 messages scored a 3.5. Twelve messages scored a 3, and one message each scored 2.5 and 2. Messages were awarded a lower score for use of unnecessarily complicated words, undefined medical terms, proposition of complex situations, or use of shorthand abbreviations that may be confusing for lower literacy readers.
National Institutes of Health standards
Most messages met the NIH Standards for simple and clear language, and the messages collectively scored a 3.6 in this element, demonstrating an increase from year 1 (3.5). Fifteen messages scored a 4. Sixteen messages were awarded a 3.5. There were three messages scoring 3 and four messages that scored a 2.5. A few messages earned a decreased score because they were not actionable. Other messages did not offer specific information and were too general in nature.
Overall, the outgoing text message scores improved in all categories from year 1, except for linguistic accessibility, which yielded a similar score to year 1. In Table 2, year 2 messages are bolded for easy reference.
Average health literacy score for all years outgoing text messages by element.
Federally qualified healthcare clinics
During year 1, only one Federally Qualified Health Center was able to participate in the outbound text messaging project. The year 1 evaluation was used to improve text messages for year 2 distribution, based on the scores. During year 2, three clinics were able to participate in the text messaging campaign. The messages distributed by each clinic were evaluated to see if any patterns in scoring emerged.
Clinics were compared with no clear takeaways, other than the iterative feedback provided to each site for improvement of future messaging campaigns. Clinic 1 excelled in the domain of social dynamics, consisting of cultural and accessibility elements. Clinic 1 also did an outstanding job of meeting NIH Standards. However, Clinic 3 rose to the top in terms of the readability, which includes the elements of linguistic accessibility and literacy level.
Discussion
The campaign was generally consistent with NIH best practices and CLAS standards. To demonstrate application and utility of the rubric, shared below are the final rating scores for select year 2 messages.
Limitations
This project took place within an urban, Southwester United States setting with multiple agencies’ federally qualified health clinics. The languages evaluated by the proposed approach included English and Spanish. The CLAS rubric has not been applied to contexts outside of this community setting, and may not be generalizable to all communities.
A final consideration evaluating outbound text messages is honoring the difficulty and complexity necessary to craft health-promoting messages that score high in all CLAS and NIH elements within the limits 150 characters. Each of the three clinics demonstrated alignment to varying elements of health literacy. By leveraging the strengths of all organizations and working together with localized translators, high-quality text messages can ignite impactful conversations between patients and CHW, ultimately yielding increased healthcare access and increased health-promoting behavior among patients.
Conclusions
The rubric was successful in determining a standardized process for evaluating proposed health-related text messages and aligning them with CLAS and NIH standards. Application of the rubric and scores before dissemination was most impactful for Federally Qualified Health Centers. Healthcare practices and networks utilizing a greater number of outgoing messages and longitudinal data may benefit from application of further statistical analysis of message scores.
Evaluators recommend feedback and rubric scores be considered and changes integrated to improve the quality of health-promoting text messages prior to distribution. The final recommendation of the rubric's utility is not necessarily for post-campaign evaluation as it was used in this case, but applied prior to dissemination. Summative feedback was provided to clinics at the end of the project period, and the clinic-based teams were appreciative. Clinics intended to integrate the findings for future campaigns. In addition to application of a standardized rubric, user testing, CHW feedback, and iterative design approaches will improve the utility and acceptability of health-promotion text messaging campaigns. Studies with a larger number of text messages will allow for statistical test and measuring significance to further refine and quantitatively validate the tool.
High scoring message.
Upper middle quadrant scoring message.
Lower middle quadrant scoring message.
Low scoring message.
Footnotes
Acknowledgments
The authors would like to acknowledge the volunteers and staff from the health clinics who shared their expertise for the purposes of this evaluation. The authors would also like to express our mutual gratitude for the multiagency partnership between Arizona State University, federally qualified health centers, and the health department that made this project possible. Gratitude is offered to the Office of Health Disparities and Lilliana Cardenas at the Maricopa County Department of Public Health for collaboration on this project. Special thanks to Lisa Blue from ProviderTech in leading the efforts for the program coding of the text messages and data extractions.
Ethical considerations
This evaluation study was approved by the Institutional Review Board of Arizona State University (IRB#STUDY00015269) on February 23, 2022. Patient consent and participant enrollment were facilitated by individual health clinics, and de-identified data provided retroactively by each clinic.
Consent to participate
Consent was not applicable to this project. Consent to receive text messages was handled independently by each clinic. This manuscript evaluates outgoing messages only, and no patient data were included in data analysis for this manuscript.
Author contributions
Mara de Luca Funke: Conceptualization, data curation, formal analysis, investigation, methodology, validation, visualization, and writing—original draft. Natalia Rodriguez González: Data curation, formal analysis, investigation, project administration, validation, and writing—review and editing. Maria Aguilar-Amaya: Funding acquisition, project administration, resources, supervision, and writing—review and editing. Wendy Wolfersteig: Funding acquisition, resources, supervision, and writing—review and editing.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This text messaging evaluation project was funded by a Cooperative Agreement from the Department of Health and Human Services, Office of Minority Health, to support the Maricopa County Department of Public Health – Advancing Health Literacy to Enhance Equitable Community Responses to COVID-19 (PH NOI 21021) project with an inter-governmental agreement to Arizona State University (AWD00037169) for project evaluation. The purpose of the federal funding was to improve health literacy among vulnerable populations, specific to COVID-19 prevention and mitigation. Arizona State University's role was to evaluate the effectiveness of activities designed by the health department to meet the federal goals.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data availability
Data are available upon reasonable request, subject to approval by partnering entities and the Institutional Review Board.