Abstract
Objectives:
Sexually transmitted infections (STIs) are an ongoing public health issue in adolescents. the Centers for Disease Control and Prevention and American Academy of Pediatrics have and continue to recommend STI screening in at-risk adolescents, however screening and testing continues to lag behind the need. We previously designed and implemented an electronic risk assessment tool to support STI testing in our pediatric emergency department. Pediatric primary care clinics may be better positioned for STI risk assessments, as they can offer greater privacy and confidentiality, a lower stress environment, and greater opportunity for longitudinal care. STI risk assessment and testing continues to be a challenge in this setting. The goal of this work was to evaluate the usability of our electronic tool to support adaptation and implementation in pediatric primary care practices.
Methods:
We conducted qualitative interviews of pediatricians, clinic staff, and adolescents from 4 pediatric practices as part of a study whose goal is to ultimately implement STI screening in pediatric primary care. The goal of the interviews were (1) to understand contextual factors related to STI screening in primary care, which we have reported previously, and (2) to obtain feedback on our electronic platform, the questionnaire content, and their perspective on implementing it in primary care settings, which we report here. We received quantitative feedback using the System Usability Scale (SUS). The SUS is a validated, reliable tool to measure the usability of hardware, software, websites, and applications. SUS scores range from a score of 0 to 100, with a score of 68 or higher indicating above average usability. We additionally obtained qualitative feedback via interviews and used inductive analysis to identify common themes.
Results:
We recruited 14 physicians, 9 clinic staff, and 12 adolescents. Participants rated the tool highly using the SUS, with a median score of 92.5 (threshold for average usability is 68) and an interquartile range of 82.5 to 100. Thematically, all participants perceived a need for such a screening program and indicated the format would encourage more honest responses on the topic adolescents. We used these results to modify the questionnaire prior to implementing it into participating practices.
Conclusion:
We demonstrated that our electronic STI risk assessment tool had a high level of usability and could be adapted for use in pediatric primary care.
Introduction
Adolescents continue to bear a high burden of the sexually transmitted infections (STIs) gonorrhea, chlamydia, and human immunodeficiency virus (HIV), with almost half of new STIs in youth ages 15 to 24 years old. 1 Because of the significant impact of STIs on patients in this age group, the Centers for Disease Control and Prevention and American Academy of Pediatrics (AAP) have and continue to recommend STI screening in at-risk adolescents.1,2 However screening and testing continues to lag behind the need.1-3
Numerous barriers to screening have been identified in the existing literature. Córdova et al 4 interviewed adolescents and found adolescents widely perceived clinicians as uncomfortable and judgmental around the subject matter of STI testing. They also highlighted that maintaining confidentiality was a key aspect to foster communication around this issue, and that better communication would facilitate greater disclosure. Goyal et al surveyed primary care pediatricians on their STI screening habits 5 and 71% reported difficulty screening discretely when the adolescent is accompanied by parents or guardians.
We previously piloted and implemented an STI risk assessment program in our pediatric emergency department (ED).6-9 This platform utilized a novel electronic questionnaire and software platform that integrated into the electronic health record (EHR). 9 A robust literature has already shown that adolescents prefer disclosing sensitive information such as sexual histories via digital means instead of face-to-face interviews.10-12 We demonstrated similar findings when implementing this program in our ED, and it led to increased STI diagnoses. 6
The St. Louis region has among the highest incidences of STIs in the United States and has a critical need for improved STI screening among adolescents. 1 Pediatric primary care clinics may be even better positioned for STI risk assessments, as they often offer greater privacy and confidentiality, a less stressful patient environment, and greater opportunity for longitudinal care. STI risk assessment and testing continues to be a challenge in this setting, however, due to infrequent sexual history taking and STI testing in many pediatric primary care practices.5,13-17
The overall goal of our work is to adapt our ED-based STI screening program for use in pediatric primary care to improve STI screening in this setting and is guided by the implementation science framework the Consolidated Framework for Implementation Research. 18 We previously reported on interviews we conducted with primary care pediatricians, clinic staff, as well as adolescents, and their parents who receive care in practices in our region to understand facilitators and barriers to STI screening to inform this effort. 19 We asked these same interviewees to evaluate the existing ED-based electronic STI risk assessment tool to assess its usability and provide qualitative feedback to inform modifications prior to implementing in primary care, and report on those findings here.
Methods
This study is being carried out in 2 stages: (1) qualitative interviews with clinics providing feedback to adapt the tool and (2) implementation of the tool for a minimum of 12 months. Change in outcomes for STI screening will be evaluated in a future report.
We interviewed pediatricians, pediatric clinical staff from pediatric practices, and adolescents who receive care in these practices. Practices were recruited in collaboration with the Washington University Pediatric & Adolescent Ambulatory Research Consortium (WU PAARC), a network of over 30 pediatric practices in our community that participate in research studies. We obtained qualitative and quantitative feedback on the electronic platform, the questionnaire content, and their perspective on using it in the primary care setting. A significant aspect was evaluating the tool’s usability, 20 a crucial element for digital health interventions. The Human Research Protection Office at Washington University in St. Louis approved this study.
The Screening Tool and Workflow Integration
The current iteration of the program utilizes the Epic EHR to offer an STI risk assessment questionnaire for adolescents receiving care in our ED, regardless of the reason for their visit. While the EHR has changed from our original platform, the questionnaire content for ED patients is unchanged from what we previously published, 9 with the only change being that the current EHR platform no longer provides an audio-component. In the ED, 15 to 21 years old patients and their families are given a brochure giving a non-specific overview of the program and discussing the need for privacy and confidentiality when the patient answers the questionnaire.6,9
Patients using the questionnaire read a brief introduction explaining the purpose of the questionnaire—to identify if they should be offered STI testing during the visit. They were then asked a series of questions to obtain a sexual history; an integrated decision rule then provides recommendations as to whether the patient should be offered testing for gonorrhea/chlamydia and/or HIV during the visit. Recommendations are given on-screen to the patient, who can electronically “opt-in” to testing. Questionnaire responses and STI test recommendations are integrated in real-time into the EHR for the physicians and nurses to review.
Recruitment and Participation
We interviewed all physicians at 4 participating practices, up to 5 clinic staff at each practice, and a convenience sample of 5 adolescents aged 15 to 21 years and one of their parents from each practice. Data reported here were obtained during interviews that were also used to examine facilitators and barriers to STI care in each practice; those data have been reported separately. 19 We did not show the electronic tool to parents as it is only intended for use by adolescents and healthcare providers, thus no parental data is in this report.
Healthcare workers were recruited via email, phone, or in-person. We identified potential eligible adolescents through the EHR by reviewing scheduled appointments for upcoming yearly preventive visits. Families were contacted by phone in advance of their appointment to screen for eligibility and to gauge interest. Written informed consent for the interviews was obtained from all participants 18 years and older, and written assent obtained for all minor participants less than 18 years of age. All participants received a $50 gift card as remuneration.
Interviews and Data Collection
Interviews were conducted from April 2020 through May 2021. In the first half of each interview we conducted semi-structured interviews to understand their beliefs around STI screening and testing; these data have been reported previously. 19 All data reported here were obtained in-person during the second half of participant interviews, during which the tool was being evaluated for use in the primary care setting.
After completing the semi-structured interviews, participants were provided an overview of the existing electronic STI risk assessment tool used in our ED, given a demonstration of the platform on a tablet computer, and then allowed to trial it. Figures 1 and 2 provide tablet screenshots representative of the questions shown to patients completing the questionnaire on the tablet.
Representative screenshot of adolescent survey asking sexual history.
Representative screenshot of adolescent survey asking patient to “opt-in” to HIV testing when recommended by program.
After using the questionnaire on the tablet, participants were asked to complete the System Usability Scale (SUS).21,22 The SUS is a validated, reliable instrument to measure the usability of many items, including hardware, software, websites, and applications. The SUS asks participants 10 questions using a five-point Likert-scale with responses ranging from strongly agree to strongly disagree. 23 Questions include items such as “I thought the system was easy to use” and “I found the various functions in this system were well integrated.” Participants completed the SUS by using REDCap. 24
After completing the SUS, all participants were then asked to provide open ended, qualitative feedback that could be used to refine the electronic tool. All interviews were conducted by study team member VD. All interviews were audio-recorded and transcribed verbatim by a transcription service and verified for accuracy by study team member VD.
Analysis
Quantitative survey data from the SUS was managed in REDCap. 24 These data were analyzed using descriptive statistics using SAS 9.4 (SAS Institute Inc., Cary, NC, USA). SUS scores range from a score of 0 (lowest usability) to 100 (highest usability); a SUS score of 68 points or higher indicates above average usability.25-27
Open-ended responses were analyzed qualitatively. We developed a codebook using thematic coding techniques to describe users’ evaluation of the electronic screening tool and perspectives on its use in pediatric primary care. Authors CM and PC reviewed each transcript and conducted inter-rater reliability checks on all data to ensure consistent interpretations; any differences were adjudicated by author VM. These data were analyzed using NVivo (version 20).
Results
We recruited 47 participants (14 physicians, 9 clinic staff, and 12 adolescents) across 4 pediatric primary care practices. All 14 physicians participated in interviews, however, one physician did not complete the SUS. We have previously reported demographics of this group 19 Briefly, 15/23 (65.2%) clinical staff were between the ages of 40 to 60, 19/23 (82.6%) were white, and 20/23 (87.0%) were female. Of the participating adolescents, the median age was 17 years, 7/12 (58.3%) were female, 9/12 (75.0%) were white, and 3/12 (25.0%) black.
System Usability Scoring
Participants rated the tool highly using the SUS, with a median score of 92.5 and an interquartile range of 82.5 to 100 across adolescents, physicians, and clinic staff. Supplemental Appendix 1 provides a summary of responses for each SUS item across adolescents, physicians, and clinic staff. Adolescents rated the tool highly. All 12 adolescent participants indicated they would like to use the system and that it was easy to use. Only 1 of the 12 adolescents indicated they would need assistance using the tool, but all 12 indicated they could learn to use it easily.
Physicians and clinic staff rated the tool highly as well, though their responses had slightly more variation than the adolescents. The 13 physicians scored the tool highly, however, 2 users indicated the system was not easy to use for them, and 3 indicated they found it complex. The 7 participating clinic staff members rated the tool higher than the physicians, with scores closer to that of the adolescents, and near universal agreement the system was easy to use.
Electronic Tool Qualitative Feedback
Overall, the participants in our study gave our tool positive feedback. We identified 3 overarching themes: (1) workflow; (2) perceived need; and (3) honest responses. We placed additional relevant quotes with disparate themes into a general category (Table 1).
Codebook With Themes and Definitions.
Physicians and clinic staff indicated the tool would fit well into their workflow and adolescents indicated it was easy to operate and use (Table 2, quotes 2.1-2.3). Having patients complete the questionnaire early in the visit was identified as the most important workflow criterion. Multiple participating clinics ask adolescents to complete the PHQ-9 depression questionnaire on paper during well-child checks, and the potential to integrate that or other questionnaires into this platform was of significant interest.
Example Quotes From Fitting into the Workflow, Perceived Need, and Promoting Honest Responses Themes.
The potential for our electronic tool to support screening for STIs was of particular importance (Table 2, quotes 2.4-2.7). Multiple clinic staff commented on the high prevalence of STIs in our region and the need for improved ability to screen adolescents. One physician commented they do not routinely offer STI screening and using this tool would facilitate obtaining information they need from patients to more readily identify patients in need of STI testing.
Participants also indicated the confidential and private nature of the tool would facilitate honest disclosure of sensitive information (Table 2, quotes 2.8-2.11). They believe adolescents’ hesitance to discuss sensitive information directly with their physician, especially when a parent may be present, would be greatly ameliorated through use of this tool.
Participants provided general feedback on the tool as well; representative quotes are provided in Table 3. Clinic staff indicated the ability to obtain a patient phone number for direct notification of test results was helpful. Adolescents commented it was comfortable to use the tool, especially when compared to speaking directly to a physician. While a physician commented that the tool was easy to use, a different clinic staff member expressed concern the tool may take too long to complete.
Example Quotes From General Feedback.
Questionnaire and Process Adaptations
We used the initial feedback from the participants to make modifications to the program prior to implementation in pediatric practices. We changed the content of the questionnaire to reflect the change of context from the ED to the primary care setting (Supplemental Appendix 2). This primarily involved modifying the introduction and the questions related to test recommendations. We made very few changes to individual questions, however we did remove several “introductory” questions that the pediatricians indicated were not needed in their environment. This included removing questions related to patient’s self-reported grade level, access to healthcare, race, and ethnicity. The program was implemented in our first participating practice in November 2021. After the first few months of use, physicians requested we review the questionnaire to identify if we could further reduce its length. After an in-person review, we further shortened the introduction.
Integration of the questionnaire into primary care required process adaptations as well. Participating pediatricians currently offer the questionnaire to 15 to 21 years old patients presenting for their yearly well-child checks. We created best practice advisories (BPAs), alerts in the EHR for clinicians, and staff, to notify office staff when a patient is eligible for the questionnaire. Patients and families are given a brochure, modeled off our ED brochure, describing the general purpose of the electronic screening without disclosing the topic. Adolescents are given privacy in a patient room to complete the questionnaire. After completion, a BPA alerts the clinician. The BPA provides a link to review questionnaire responses and STI testing recommendations, and prompts clinicians to order testing when recommended. Questionnaire responses do not appear in the online patient portal, preserving patient confidentiality even should a parent have proxy access to review their child’s information.
Discussion
STI screening in for adolescents continues to lag behind the need and recommendations. Providing this care for adolescents faces barriers compared to adults due to challenges obtaining this care independently and confidentially. We demonstrated that an STI screening tool developed for use in a pediatric ED has a high degree of acceptability by primary care pediatricians and adolescents.
Based on results of the SUS, adolescents, pediatricians, and clinic staff indicated our electronic platform was exceptionally easy to use, scoring well above the “average” usability score of 68, even before adaptations were made for their environment. This is reflective of the significant effort we spent originally developing the program, which included interviews of adolescents to review the questionnaire, and the qualitative feedback provided by adolescents in our ED using it. 7 A consistent theme emerged among all interviewees that this method would encourage honest responses from adolescents and facilitate this difficult conversation. This also mirrors our findings from the ED that adolescents prefer to disclose such sensitive information via electronic questionnaires instead of face-to-face interviews.
While some physicians expressed concern over the length of the ED version of the questionnaire and how to best integrate it into their workflow, they were positive about its potential use in their practices and collaborative in making modifications to the questionnaire for use in their setting. The original ED version of the questionnaire was completed in a median of 8.3 min. 7 With a shortened introduction and fewer questions, and completion in the calmer and more controlled primary care environment, we anticipate a shorter completion time.
Our initial work exploring facilitators and barriers to STI screening in this setting, 19 as well the work of others,4,5 highlights the need for improved ways for adolescents and pediatricians to communicate STI care. Adolescents and clinicians emphasized the importance of maintaining confidentiality when discussing STIs, and our tool offers 1 such opportunity. While confidential delivery of STI care to adolescent minors is still legally protected in all states, 3 providing this care presents many challenges. Our tool provides an efficient and confidential way to discretely screen adolescents, and we found adolescents as well as clinicians were supportive of this approach.
This method is not the only electronic/web-based method to obtain this information from adolescents. Stalgaitis and Glick performed a systematic review of web-based diaries used to identified risky sexual behaviors. 12 The studies they identified included only adults, and were implemented for a limited time. While a real-time diary would reduce the potential for recall bias, it is more time-intensive and could raise privacy concerns with information being stored and transmitted online. These concerns are mitigated by our system, which is integrated into the EHR. Additionally, our patient responses are not viewable in the EHR patient portal, which is important for confidentiality as many parents can access adolescents’ information in the portal. Karas et al described implementation of a clinical decision support tool for a pediatric primary care network to increase STI testing during well-child checks, similar to the goal of our program. 28 Their effort differed from ours as it was only for adolescent females, and utilized documentation and test results in the EHR to alert clinicians instead of patient questionnaires to generate STI testing alerts. They did see a doubling of chlamydia testing in their target population, demonstrating an alternative way to leverage EHR alerts for similar goals. Wood et al similarly pursued increased chlamydia screening for adolescent females in a pediatric primary setting through EHR alerts, with universal obtaining of urine specimens but combined it with universal collection of urine specimens and observed an increase in chlamydia testing. 29 Wayal et al used a web-based survey to obtain STI risk information from adolescent and adults; their program was implemented at sexual health clinics in England, a higher risk setting and population than our effort, however did find high acceptability of their intervention and potential to identify STI risk. 30 A 2016 systematic review of STI screening in clinic settings identified 42 interventions to increase STI screening, however none appeared to use EHR-integrated questionnaires and alerts, and few included men. 31 Our program appears to be unique with our focus on pediatric primary care practices, including all female and male patients, and the use of an EHR integrated questionnaire, however elements such as universal urine collection and enhanced use of existing data from the EHR could improve our process.
We have implemented our tool with the updated questionnaire in a second practice, and will soon be implementing in a third practice. We are monitoring STI testing practices and continue to obtain feedback from physicians to continue to improve the tool. Our integration into their existing workflows and the overall use of the tool will be described in a future report describing our implementation outcomes.
Limitations of our work including recruiting a convenience sample of patients from a subset of patients in our region. As such participating adolescents may not have been reflective of others in the region or nationally. Additionally, while participants rated our tool highly using the SUS, this evaluative mechanism has limitations. Broekhuis et al 20 compared the usability of the SUS compared to other benchmarking instruments and found that think-aloud protocols were a more effective tool and recommended against using the SUS as the sole evaluative method. While we did not have a formal think-aloud protocol, participants were able to provide qualitative feedback on the tool and their comments were reflective of their overall positive view of our tool.
Conclusion
We demonstrated that our electronic STI risk assessment tool, originally developed for use in a pediatric ED, had high usability and appeal in the pediatric primary care setting. With minor modifications, we were able to adapt the tool for use in primary care settings and subsequently implemented an STI screening program in pursuit of evaluating its clinical effectiveness in this new patient environment.
Supplemental Material
sj-doc-2-jpc-10.1177_21501319231172900 – Supplemental material for Adapting an Electronic STI Risk Assessment Program for Use in Pediatric Primary Care
Supplemental material, sj-doc-2-jpc-10.1177_21501319231172900 for Adapting an Electronic STI Risk Assessment Program for Use in Pediatric Primary Care by Fahd A. Ahmad, Pamela Chan, Collin McGovern, Viani Dickey, Randi Foraker and Virginia McKay in Journal of Primary Care & Community Health
Supplemental Material
sj-docx-1-jpc-10.1177_21501319231172900 – Supplemental material for Adapting an Electronic STI Risk Assessment Program for Use in Pediatric Primary Care
Supplemental material, sj-docx-1-jpc-10.1177_21501319231172900 for Adapting an Electronic STI Risk Assessment Program for Use in Pediatric Primary Care by Fahd A. Ahmad, Pamela Chan, Collin McGovern, Viani Dickey, Randi Foraker and Virginia McKay in Journal of Primary Care & Community Health
Footnotes
Acknowledgements
We wish to thank the adolescent patients and their parents, as well as physicians and clinic staff members who participated in this study.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Agency for Healthcare Research and Quality (R18 HS026704)
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
