Abstract
Hospital-based violence intervention programs (HVIPs) are evidence-informed strategies to promote recovery among victims of violence. Limited tools exist to capture client-reported perspectives of program relevance, responsiveness, acceptability, and impact. We conducted a quality improvement project to develop an HVIP-specific tool that can be used to collect information regarding client satisfaction with services to inform ongoing and future program improvement efforts. Four former adolescent clients and 5 caregivers who received services participated in cognitive interviews to share interpretation of questionnaire items, improvements to response options, comfort responding, and missing topics. Participants demonstrated understanding of survey items and offered alternate wording to improve clarity, reduce redundancy, and clarify response options. Revisions based on participant and staff feedback resulted in a 12-item HVIP Client Satisfaction Questionnaire. Our tool provides an opportunity to collect client-reported perspectives regarding satisfaction and perceived short-term program impact to inform continued improvement activities. This tool could be utilized by HVIPs and other violence prevention programs to engage clients in quality improvement efforts in support of program values to be trauma-informed and client centered.
Keywords
Introduction
Hospital-based violence intervention programs (HVIPs) support recovery after violence-related injury, providing trauma-informed case management services to victims of violence. 1 Programs serve as a bridge to community-based services, promoting physical and psychosocial quality of life and reducing adverse outcomes. 2 Frequently, HVIP evaluations measure process metrics (eg, number of clients served, needs addressed) or longer-term/rare outcomes (eg, reinjury, death, incarceration).3-5 Systematic collection of client perspectives on care remains limited.6,7 Empowering clients to share their experiences by centering their voices in program improvement is vital to aligning with the trauma-informed nature of HVIPs and advancing equitable care.2,8
Prior evaluations of satisfaction with and quality of victim service programs focus on specific types of victimization or singular services.9,10 These efforts incompletely reflect HVIPs’ focus on psychosocial recovery, well-being, and engagement in cross-sector services. Within our HVIP, we recognized clients could voice their needs throughout program participation (ie, voluntary enrollment, self-identified recovery goals), but did not have opportunities to provide feedback to improve service quality. Thus, the objective of this quality improvement project was to develop a tool to collect feedback from HVIP clients and caregivers. This tool addresses a critical need to promote continuous program improvement efforts within HVIPs as they support recovery among violently injured individuals.
Methods
Housed at a large urban children's hospital in the mid-Atlantic United States, our pediatric HVIP provides voluntary case management services to young people ages 8 to 18 and their families following treatment in the emergency department or trauma unit for violence-related injuries. To support inclusion of client voices in quality improvement efforts, we implemented a multiphase process to create a satisfaction measure. Due to the nature of this work as quality improvement, it was not subject to local Institutional Review Board review. Risk to participants was minimal given they provided insight into questionnaire development and not personal experiences. To mitigate potential discomfort or confusion, team members who did not provide direct services described the project to participants and that their feedback was confidential.
Initial Questionnaire Modifications
We adapted the Client Satisfaction Questionnaire (CSQ-8), an 8-item scale used in health and human service systems. 11 The CSQ-8 assesses perceived service quality and quantity, extent to which services met needs, and if respondent would return and recommend services. We retained all items and response scales, making minor wording changes for the HVIP setting (eg, changed “If you were to seek help again” to “If you were to experience violence again”). We added 3 open-ended questions to identify most and least helpful program components and improvement recommendations.
Cognitive Interviewing
We conducted cognitive interviews with a purposive sample of former clients and caregivers to ensure questions and response options were clear and interpreted as intended. Former adolescent HVIP clients who had recently completed services and their adult caregivers were recruited via e-mail and phone calls. Clients and caregivers independently completed the pilot version of the HVIP CSQ prior to their cognitive interview. Paired staff members conducted interviews via telephone, with one serving as notetaker. Participants received $20 gift cards as remuneration.
We developed a theoretically grounded cognitive interview guide informed by the model of survey response to inquire about: (1) comprehension of survey items and terminology, (2) judgment about how to respond, and (3) response options. 12 Interview prompts asked participants to describe (1) interpretation of items; (2) understanding of key terminology; (3) ease of using response scales; (4) comfort answering honestly; and (5) recommendations for modifications/additional questions.
Measure Refinement
We conducted interviews until we reached saturation. HVIP team members who supported program direction, evaluation, or research, but no longer provided services, participated in meetings to iteratively review, discuss, and make recommendations for integrating participant feedback. Additionally, we identified missing constructs relevant to HVIP service quality and adapted existing client satisfaction questions.13,14
Results
We attempted outreach to 12 former clients (age ≥ 14 years) and 12 caregivers from 13 families (one client was 18 years old and invited without a caregiver; one client was 10 years old and only their caregiver was invited). Four adolescents and 5 caregivers, including 3 dyads, participated (Table 1).
Cognitive Interview Participant Demographic Characteristics.
HVIP, hospital-based violence intervention programs; CSQ, Client Satisfaction Questionnaire; CAHPS, Consumer Assessment of Healthcare Providers and Systems.
Item source reflects the original source of survey items and whether the item was adapted from existing measures or created based on client and staff feedback.
Clients and caregivers understood the purpose of the CSQ was to “ask if these were the types of services and help [client was] looking for” and “asking how [client] felt during the whole program overall.” Participants noted opportunities for streamlining responses, flagging items with indistinct or complex options (eg, scale including “almost all of my needs were met” and “most of my needs were met”). They also identified words with negative connotations (eg, “problems”) that adversely affected survey tone.
Revisions included simplifying vocabulary (eg, replacing “satisfied” with “happy” and removing words like “quality”) and response scales (eg, replacing “indifferent/mildly satisfied” with “neither happy nor unhappy”). In response to adolescent feedback, we added an item to rate overall program experience on a 1 to 10 scale and included a single open-ended item to elicit comments and recommendations. After identifying a consistent theme in interview feedback that the program was both immediately helpful and fostered continued well-being, we added 2 items to measure to the extent to which the program enhanced (1) knowledge of and (2) self-efficacy to access services. We replaced “problem” to use more positive framing (“Did the program help you/your child to feel better?”) and removed an item asking if respondents would recommend the program to others to eliminate confusion regarding eligibility.
Caregivers questioned whether the subject of items (“you”) referred to their experience or their child's, with one recommending the questionnaire “should be more about the clients and not so much the parents since (adolescents) have experience with (HVIP).” Thus, we created separate versions of the questionnaire—an adolescent/young adult version that uses “you”/“your” and a caregiver version referencing “you and your child.” Based on staff review, we added 2 items to assess whether respondents perceived staff to be (1) culturally competent and (2) careful listeners, in line with our program and broader HVIP values. These modifications resulted in a 12-item HVIP CSQ with parallel wording for an adolescent/young adult and caregiver version (Table 2).
Final HVIP CSQ Items.
1Item source reflects the original source of survey items and whether the item was adapted from existing measures or created based on client and staff feedback.
Discussion
Our HVIP CSQ tool enables our HVIP to collect client and caregiver feedback to inform quality improvement efforts and ensure service responsiveness. Presently, we are aware of no such instruments tailored to the HVIP setting. In line with our values of being client-centered and family-led, we engaged adolescents and caregivers, resulting in item wording and response options relevant to their experiences and preferences. We believe this tool could be utilized by other HVIPs and similar violence prevention programs to identify implementation successes and improvement opportunities.
Obtaining client input regarding satisfaction and acceptability of services is foundational to understanding program impact. Historically, many client- or patient-reported measurement tools have been developed in populations predominately identifying as white and of higher socioeconomic status, reflecting respondent demographics dissimilar from most HVIPs, whose clients tend to be younger and identify as members of racial and ethnic minority groups. 15 By engaging clients and caregivers in measure adaptation, we enhanced the survey's face validity, overcoming potential barriers in alignment of wording with their perspectives and how they spoke about services.16,17 Given the limited sample size and single site, future efforts to establish content validity represent a critical next step.18,19
Our HVIP-relevant measure of client satisfaction fills a gap in the growing area of victim service programming and, in creating intentional space for client voice, represents a critical extension of our HVIP's trauma-informed values. While client experiences and satisfaction with services are important short-term outcomes, healthcare data indicate more satisfied patients have improved longer-term outcomes. 20 Thus, satisfaction is an important component of prospective HVIP evaluation with the potential to provide insight regarding program engagement, long-term wellbeing, and quality of life.
Limitations
First, we conducted this research within a single pediatric HVIP. While our clients are children and adolescents, our measure allows for self- and proxy-report, making it adaptable to other HVIPs. Programs serving exclusively adults could use only the self-report measure. Second, all of our participants spoke English and identified as members of racial minority groups, who may have different perspectives than clients who do not speak English or are of majority racial backgrounds. However, our participants reflect the demographic characteristics of many HVIP participants. Finally, although respondents were invited from a list of recently graduated clients, we recognize the importance of eliciting feedback from individuals who were lost to follow-up or prematurely ended services to fully understand participant experiences.
Conclusion
The 12-item HVIP CSQ, developed with client, caregiver, and staff voice, is optimized to capture client and caregiver perspectives of HVIP quality and improvement opportunities. Continued efforts are needed to establish validity of the measure, particularly through cross-site implementation in a range of HVIPs serving geographically and demographically diverse populations. Importantly, this instrument allows individuals who have been historically marginalized to share their experiences to inform program improvement. The HVIP CSQ provides a foundation for other programs who might adapt this tool for their own unique client populations and services using the systematic process outlined in this work.
Footnotes
Acknowledgments
We gratefully acknowledge the VIP clients and caregivers who generously agreed to participate in this quality improvement project and share their experiences and expertise with our team. We recognize the assistance of Yansy Salmeron and Brittany Dorsonne who helped conduct interviews. We graciously acknowledge Jacqueline Kendrick, MSW, LCSW for her assistance with recruitment and outreach and Carolena Muno, BA and Mariam Jaffer, MBBS, MPH for their assistance in conducting literature reviews and preparing the manuscript for submission.
Author Contributions
Each of the authors contributed sufficiently to be considered for authorship. The following contributions were made to this manuscript: conception or design of the work: Rachel Myers, Hillary Kapa, and Stephanie Garcia; data collection: Hillary Kapa and Stephanie Garcia; data analysis and interpretation: Rachel Myers, Hillary Kapa, Stephanie Garcia, Laura Vega, and Joel Fein; drafting the article: Rachel Myers and Hillary Kapa; critical revision of the article: Rachel Myers, Hillary Kapa, Stephanie Garcia, Laura Vega, and Joel Fein; and final approval of the version to be published: Rachel Myers, Hillary Kapa, Stephanie Garcia, Laura Vega, and Joel Fein.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethics Statement
This work met the criteria as a quality improvement activity by the Children's Hospital of Philadelphia IRB and did not require IRB approval or determination. Our institution does not require IRB review for activities meeting criteria as quality improvement projects.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Statement of Consent
Verbal consent to participate in these quality improvement activities was obtained from adult participants. For participants lesser than 18 years of age, legal guardians provided verbal consent and adolescent participants provided verbal assent prior to participation.
