Abstract
This study demonstrates an ethnographically critically informed process evaluation of piloting an HIV intervention program targeting injection drug users. The authors used systematic ethnographic methods to identify and evaluate the factors that facilitated or inhibited message diffusion. Findings indicated that communication patterns and strategies advocated in training sessions did not translate immediately into the outreach encounter in the drug use communities. The intervention program was refined and developed based on findings from the ethnographic process evaluation. The authors argue that ethnographic methods provided a dynamic, flexible, and iterative process for evaluating the intervention's development and ensuring its cultural relevance.
Keywords
Introduction
In preventing HIV/AIDS domestically and internationally, there is a growing concern about the necessity to develop culturally appropriate intervention programs. Among the numerous intervention programs published, most report program outcomes, but very few document how these programs were developed to ensure their cultural relevance. Process evaluations of these interventions have focused primarily on the “final product” of the program delivery; process evaluations of the piloting stage of the interventions have been largely missing from the program evaluation literature.
In the United States, African Americans have been disproportionately affected by HIV infection. Injection drug use and unprotected sex continue to be the leading risk factors among African Americans (Centers for Disease Control and Prevention [CDC], 2001a; Strathdee et al., 2001). Recent HIV prevention efforts using network-oriented peer outreach interventions with harm reduction strategies have demonstrated efficacy in reducing HIV risk behaviors among injection drug users (IDUs) (Latkin, Sherman, & Knowlton, 2003; Montgomery et al., 2002; Stopka, Singer, Santelices, & Eiserman, 2003). Based on a social influence framework, with theories of social diffusion, social norms, and social identity, these interventions use peers to promote risk reduction within their social networks. For instance, in an HIV prevention intervention in Baltimore, Maryland, intervention participants were three to four times more likely to report the reduction of risky injection behaviors (e.g., cooker and needle sharing) than were control group participants (Latkin et al., 2003). Moreover, results indicated that the intervention led to risk reduction among the current drug using participants' (social) network members, and this effect was equally effective for HIV seropositives and seronegatives (Latkin et al., 2003). Despite these optimal outcomes, however, studies on process evaluation of these intervention programs, especially the piloting stage of these programs, remained largely unpublished.
Piloting and process evaluations of HIV prevention interventions
Piloting is an indispensable component of developing intervention programs. It encompasses the process of translating the program's objectives into actual program modules based on the target population and the available resources (Pick, Poortinga, & Givaudan, 2003). Process evaluations can be used during the piloting stage to monitor and assess the effectiveness and appropriateness of the program.
Traditionally, process evaluations attempt to answer the following questions: What does the intervention program consist of? Was the program actually carried out as planned? and What factors facilitate or inhibit the implementation and the outcome (Steckler & Linnan, 2002)? Process evaluations is a “spiral” process involving first the “monitoring,” which documents and confirms the structural elements of the program, and second the “evaluation,” which assesses whether the specific elements are being established to meet program objectives (Windsor, Baranowski, Clark, & Cutter, 1984).
The development of interventions is an iterative process, with testing and refining of numerous components. Although program piloting is common in social science research, seldom is the process of piloting systematically observed and evaluated, and even less frequently is it reported. The detailed analysis of the development of a behavioral intervention might allow investigators to identify key more easily concepts and approaches that require tailoring to specific contexts. Moreover, this approach can be vital for theory development (Steckler & Linnan, 2002). By identifying essential aspects of social dynamics involved in behavior change, we can develop theoretical models based on these observations. Although there is a growing emphasis on process evaluation in social and behavioral interventions, there remains a lack of systematic methods with theoretical underpinning for conducting process evaluations (Steckler & Linnan, 2002).
Ethnographic process evaluation
Ethnographic research is highly compatible with the goals of process evaluation, with a key advantage of its holistic approach to understand behavior in context (Agar, 1996; Britan, 1981; Bourgois, 1998; Bourgois & Bruneau, 2000; Cecares, 1993; Latkin, 1998; Singer, Himmelgreen, Weeks, Radda, & Martinez, 1997; Singer, Zhongke, Schensul, Weeks, & Page, 1992). The application of ethnographic methods in HIV prevention interventions can be traced back to the emergence of the HIV epidemic in early 1980s (Agar, 2002; Bolton & Singer, 1992; Feldman, 1990). It has been employed in all aspects of HIV prevention intervention projects (Agar, 1996; Dickson-Gomez, Knowlton, & Latkin, 2003; Hopson, Peterson, & Lucas, 2001; Stopka, Singer, Santelices, & Eiserman, 2003) but is particularly suited for peer education intervention evaluations because of the difficulty of obtaining data concerning the peer interactions. As Svenson (1998) has noted, “Peer education activities can involve a wide range of informal and formal influences that are difficult to capture using only questionnaires. Moreover, it is difficult to translate people's subjective thinking and feelings into quantitative variables that can be measured” (p. 44).
Despite the common acceptance of ethnographic methods in formative, outcome, and impact evaluation of HIV prevention programs, to date little research has been conducted concerning ethnography's role in the process evaluation of developing intervention programs. To address this void, we conducted an ethnographically informed process evaluation during the piloting phase of a social network oriented peer outreach HIV prevention program. The program targeted a high-risk, IDU population in Baltimore, Maryland. Ethnographic evaluation efforts focused primarily on the evaluation component and sought to better understand how the program's diffusion-oriented harm reduction approach could be optimized for the program participants, as well as for their social network members.
Method
The STEP into Action intervention program
Based on social influence theory and a previously successful HIV prevention intervention (Latkin et al., 2003), the STEP Project is a network-oriented intervention that trains IDUs to promote HIV prevention within their drug and sexual networks by using a harm reduction and social influence framework. STEP is the acronym for a communication tool that the participants are trained to use when talking to their networks about harm reduction. S stands for “stand up and be positive,” T stands for “talk with respect,” E stands for “evaluate the situation,” and P stands for “put a plan into action.”
The study design is a controlled Phase II efficacy trial. Participants (known as “index members”) are randomly assigned to either an experimental condition, where they receive the STEP intervention, or an equal-attention control condition. STEP training consists of six sessions, including four group sessions that focus on peer mentoring and communication skills, one individual session, and one dyad session (in which the index participant brings his or her primary drug or sex partner). Booster sessions are provided after the conclusion of the main intervention phase. STEP was tested and revised through five cohorts (waves) of pilots beginning in July 2003 and ending in January 2004. Although each pilot cohort consisted of the same six-session components described above, significant changes were made to the intervention manual as a result of the ongoing process evaluation and the feedback occurring at the time. All the piloting procedures were approved by the IRB in The Johns Hopkins University School of Public Health.
Research site and participants
This program took place in the city of Baltimore, Maryland, which has one of the highest rates of injection drug use, HIV, and sexually transmitted infections in the United States (CDC, 2001b). Inclusion criteria for index participants for this program are (a) at least 18 years old, (b) recruitment of at least one drug or sex risk network member, and (c) self-reported IDU within prior 3 months. Participants were recruited using word-of-mouth, community outreach, advertising in community newspaper, and flyers on the streets. The number of participants in the pilot sessions varied from 5 to 11 in different cohorts. Most of the participants were African American (83%), and 69% were male.
Ethnographic process evaluation and data collection
Systematic ethnographic methods were used in the process evaluation. Data were collected through unobtrusive observation, participant observation, and interviews. One of the goals of the ethnographic evaluation was to observe the intervention process from the development of session materials, to the facilitators' presentation of the materials, to the participants' reactions and interactions with the session materials, and, finally, to the participants' ability to communicate the program's HIV prevention information to their networks and demonstrate risk reduction behaviors.
After obtaining consent from the participants, at least one ethnographer observed the group sessions. During the session break, the ethnographers would talk to participants to elicit feedback about the session and build rapport. After several sessions, the ethnographers would ask participants if they were willing to demonstrate skills during outreach activities in the community. Ethnographers then accompanied the participants into the community to observe how they practiced the skills they learned with their risk partner or other community members. Some of these activities and conversations were audiotaped. Detailed field notes were also taken. For individual and dyad session, where the presence of observers was deemed to be too intrusive, sessions were taped and analyzed at a later date.
During their observations and fieldwork, ethnographers identified opinion leaders and key informants (i.e., those who had extensive community networks). Individual informal interviews were set up with these informants. These informal interviews usually took place in the street settings. Ethnographers took field notes on return from the street.
Data analysis and translating findings into action
The dataset consisted of tapes and notes of training sessions, fieldwork and interviews in the communities, and in-depth interviews. These notes, along with other field notes, were uploaded to an ethnographic database in Microsoft Access. Ethnographers initially analyzed the field notes using the themes that paralleled the domains and/or content areas addressed in the session activities and objectives, with an allowance for new themes emerging through the analytical process (LeCompte & Schensul, 1999; Silverman, 1993).
When a session was complete (i.e., both the training session and street outreach were completed), the ethnographic team organized all available data and met with the implementation team to discuss how to translate the evaluation findings into possible actions. Through group discussions, decisions were made to refine the training protocol or to reframe the implementation. For each problem we identified in the ethnographic process evaluation, there was a correspondent action to “fix” the problem. Because the focus of this article is on process evaluation, particularly how ethnographic methods inform piloting an intervention program, we will present exemplary findings of those aspects of the intervention that did not work out very well in training sessions or the community settings, and how they were altered.
Findings
“Getting clean” versus “harm reduction”
The STEP into Action intervention program is based on the philosophy of harm reduction, which acknowledges participants' current risky behaviors and encourages them to reduce the risks step by step. However, from the first pilot, ethnographers observed a tension between the beliefs about “getting clean” and “harm reduction.” Drug users in the group often expressed that getting clean was the ultimate solution—once they were clean of drugs, then all other problems in their lives would diminish—whereas the facilitators often brushed past the notion of getting clean and talked instead about reducing risks in ways other than abstinence. As one ethnographer observed in his session notes, “some of the participants were internalizing their participation in the group session as treatment or intervention for cessation of their drug use.” Several ethnographers made similar observations while listening to the session tapes. Another noted, “I have noticed that participants talked about a sense of remorse when they knowingly practice high risk behaviors. This feeling is compounded by a sense of powerlessness to do anything about it” (emphasis in original).
However, to achieve the goal of getting clean, individuals often go through many steps of harm reduction. At this point, getting clean and harm reduction are convergent rather than conflicting. Based on this understanding, the intervention language was modified so that the facilitators first acknowledged that getting clean was a positive objective and then emphasized that harm reduction was one process for achieving their ultimate goal of abstinence while reducing their HIV risk. Throughout the program, we communicated to the participants that getting clean and harm reduction were on a continuum rather than dichotomous categories, and we emphasized that the skills to achieve both beliefs were similar. We changed the training script to show how harm reduction can help them get clean and that getting clean was often an overgeneralization for all their problems; but the skills they learned in the program such as social and communication skills would allow them to address important interpersonal issues and establish a foundation for controlling and ceasing drug use. We also pointed out that there were major life goals that could be enhanced while they were trying to get clean, foremost among them improving relationships with family members and partners. Participants, therefore, were encouraged to discuss their experiences of getting clean and how they integrated the harm reduction philosophy into achieving their getting clean goal. They acknowledged that every small positive change they made while trying to get clean is safer for themselves and others.
Partner versus network member
The objective in the first session of the peer mentor training is to establish the identity of “health educator.” The index members are expected to bring in risk network members, which include sexual or drug-using partners, to the program and disseminate the HIV prevention information to these partners. The intervention-specific meaning of partners, however, was not very clear to the participants during the initial sessions. In the first cohort of the pilot phase, the participants were told to identify “a person who they wanted to help.” However, “for several participants the priority for their efforts ‘to help’ did not reveal skills directly involving the Health Educator skills,” an ethnographer noted in his observation notes of group session, “A link between their ‘helping others’ and Health Educator needs to explored or established.”
Several examples we observed in the training sessions and the street outreach indicated an unclear notion of partner or network member for the participants. In the first cohort session, for instance, one participant stated that he wanted to assist his 3-year-old daughter, because the child's mother had recently died and he was taking on the responsibility of raising his daughter. In contrast, an ethnographer observed on the street that a female participant practiced outreach by initiating a tense conversation of condom usage with a stranger passing her house, whereas her conversation with her significant other regarding harm reduction was minimal. In some informal interviews, several participants indicated that when talking about a partner, they thought of only a sexual partner, not a drug user partner, and that they did not tend to use the word partner in their daily conversations.
These observations indicated that the notion of peer mentor was either interpreted so broadly as to include everyone they knew as a network member or understood so narrowly as to include only their sex partners. Therefore, the training protocol was revised so that it would be less confusing and more flexible for participants to disseminate the harm reduction message to their network members. First, we stopped using the word partner in the group session and encouraged participants to identify their network members that they listed in a network interview. We also invited several “harm reduction educators,” who had graduated from previous intervention programs, to share their experiences of learning and disseminating harm reduction skills to their sexual partners, drug use partners, and other network members. In addition, in the “homework debriefing,” when participants returned for their next session, they were encouraged to share their experiences of disseminating harm reduction messages in the communities and especially among their network members, so that the facilitators could help them stay on track and the session participants could learn from each other.
“Stay positive” versus “Sometimes you gotta go hard”
Originally, the S in STEP stood for “start positive and stay positive,” which encouraged the health educators to always look at the positive side of behaviors for themselves, their partners, and other community members. In a group session when participants discussed their previous homework assignment, one participant commented that “sometimes you gotta go hard,” because many times the situations in the streets or communities did not allow him merely to “stay positive.” He provided an example of trying to disseminate the safe drug-splitting procedure to his drug use partner, who ignored his advice and continued preparing drugs in a shared cooker. Finally he “went hard” and insisted on his safe procedure, and his partner gave in and began to listen to his suggestion. Other participants, however, shared their positive experiences of “stay positive.” One suggested that he told his friends that “the life you save is your own.” Another expressed a sense of accomplishment from working as a health educator in the community: “Helping somebody creates a domino effect, when I help someone else, who helps someone else.”
These observations of negative and positive experiences of working as a health educator in the community suggested that that some participants had interpreted being positive narrowly as being nice to everyone all the time. Feedback also suggested that staying positive alone was not an adequate tool for participants who encountered many hard situations in the street. Through informal interviews with some key informants and observations in the streets, ethnographers noted that in the community of drug users, whose ideologies differed from “mainstream” values, identifying or promoting oneself as a positive model without acknowledging the realities in the communities might stimulate negative “sniff” (disregard, contempt). Therefore it is critical to be respectful while being positive.
Based on these observations, we changed the letter S in STEP to “stand up and be positive,” with an emphasis on having a positive attitude toward life. Staying positive became more of a sense of respect and not telling others how to run their lives, and believing in oneself and others. We encouraged participants to initiate the conversations about safe drug use or sexual practices with their network members by first acknowledging positive steps that their network members were taking already. Rather than saying, “It's stupid to share needles/cookers,” the conversation should start with how the network felt and their own concern. Group problem solving activities were designed to allow participants to focus on the skill of “standing up and being positive.” Participants were read a scenario and asked to identify the positive aspects of the individual's behavior. Eventually, participants began describe conversations with their networks in terms of first acknowledging the positive aspects of their networks' behavior.
Communication skills taught in training sessions versus communication in the streets
Through some individual interviews with the participants, we learned that “communication” was the most cited barrier for health educators. For example, several participants mentioned that they had difficulty bringing up the subject of condoms and safe sex with their partners, because they thought their partner would get suspicious of their sudden interest in using protection. Some mentioned that their partners did not want to discuss the program, because they felt that if the participants got clean they would end up leaving their drug-using partners, so they avoided discussing any issue regarding harm reduction. Others exchanged the ideas of “what made it easy to go out and practice.” One said being “straight up” helped, because people often found it difficult to initiate such conversation; but when they were in the conversation, they were much more comfortable.
During the first few pilot sessions, the communication skills taught focused on active listening. Many participants responded that these skills were not very effective when they had conversations with their peers. Some participants mentioned being nervous, because they did not want to “get into other people's business,” or they didn't want to take the risk of “getting cursed out.” During a community outreach, two participants told an ethnographer that in real life, they rarely picked up the topic of HIV, especially in interactions involving injection. They also suggested that “if there is something ‘in it’ for someone they will tend to listen.”
Based on the ethnographers' observations and participants' suggestions, we changed the communication skills from focusing on active listening to focusing on talking with respect and evaluating the situation through asking open-ended questions and incorporating the notion of “stand up and be positive.” Conversations could start with respect and concern of the network's current situation, and then gradually moved to a discussion on how to reduce risks in their injection or sexual behaviors so as to have a safer lifestyle.
We also created “conversation starters” to help initiate non-confrontational conversations. Some were different facts about HIV or HIV prevention; for example, “Did you know that condoms were first used by Egyptians more than one thousand years ago?” and “Did you know cold water works better than hot water in cleaning needles?” Some conversation starters were novel condoms, such as polyurethane male condoms and a condom marketed specifically to an urban clientele named the Jimmy Hatz. All of these conversation starters were topics that would be interesting and seem natural to the setting. Later, in the street outreach, we found that participants could initiate conversations more easily by using these conversation starters.
Good metaphor versus useful tools
In the session on injection-related HIV risk behaviors, we used an “injection risk ladder” to help participants understand the different levels of risk involving drug use. For example, splitting drugs dry or using new needles each time you inject are considered as safe drug use practices and are placed at the bottom of the injection risk ladder. In contrary, using someone else's needle without rinsing it is an unsafe practice and, therefore, placed on top of the ladder. During the session, participants found this risk ladder very interesting and actively identified the levels of risk using the ladder. The injection risk ladder served as an effective teaching aid for the facilitators as well as a useful metaphor for the participants to describe methods of reducing risks of HIV acquisition and transmission.
However, in the informal conversations with the participants after the session, some participants expressed the concern as to whether risk was always the factor in decision making. They asked how they could constantly remind themselves about the levels of risks in situations where consequences were not the first thing in their mind, especially when they were holding the drugs in their hands. Ethnographic observations in some shooting galleries suggested that how to split drugs equally was a big concern for many participants who shared drugs and “shot up” together. Drug users understood that sharing needles was risky, but many still shared cookers or used dirty needles to prepare and split drugs. The two most popular methods of drug sharing are “wet” and “dry.” Dry-split was rarely used because of the difficulty of measuring dry drugs and the loss of drugs due to paper absorption. Wet splitting (splitting the drug in solution) was commonly used because of the perceived ability to split drugs equally.
Based on these observations, we changed the injection risk ladder from numerous items on the ladder to only a few items and incorporated “getting clean” on the bottom of the ladder. We also added more emphasis on practices of safer splitting. We made a video of different safe splitting methods and ask participants to practice some of the methods in the sessions. We also provided tools and supplies that they would be need for each method. For dry splitting we gave participants laminated cards with square grids so that drugs could be divided easily without any loss due to paper absorption. Because this method could easily split drugs equally, participants named this kind of paper the Even Steven.
The second method is called the “norm-ject.” One clean syringe without a needle (i.e., a norm-ject) with cc markers was used only to draw up the water and to measure the drugs; drugs were cooked in one fixed clean cooker; and then the norm-ject was used to draw up the drugs and squirt equally to other cookers. Participants were initially not very enthusiastic about this splitting method, because “it is too complicated and takes up too much time.” Ethnographers introduced this method to some opinion leaders in the shooting galleries and found it could be potentially successful. Video demonstrations and practices in the training sessions further reduced the anxiety of applying this method. We distributed norm-jects to participants and their network members with a card explaining its use. Participants reported increased acceptance of this method, and even gave it a new name, the New Jack.
Discussion
Feasibility and efficacy of applying ethnographic methods in process evaluation
Our experiences have demonstrated the feasibility and efficacy of applying ethnographically informed process evaluation for piloting the STEP program. In this network-oriented peer outreach intervention, program success and sustainability depend on the dissemination of the harm reduction messages through social networks. Therefore, it was crucial to develop a culturally and linguistically appropriate program that could motivate the participants to begin and continue harm-reduction conversations with their network members. The best way to assure that the intervention materials were relevant and could be diffused was to observe their social interactions within the session and among their network members in natural settings. Ethnographic methods complimented the nature of such intervention development.
Ethnographic research is often labor intensive; however, we can minimize this disadvantage if we design our evaluation strategy with clear objectives, build a data collection strategy to document the problems and the context systematically, upload data into a database in a timely fashion, and keep an interactive communication between the ethnographic findings and intervention development and implementation. Meanwhile, our strategy allowed us to display the advantages of ethnographic methods. We gathered information that could not be captured easily by surveys, particularly the participants' thinking and feelings, and the changing social environments that are difficult to measure by quantitative methods. The ethnographic process evaluation helped describe how the program was being operationalized in the field, identified factors affecting the intervention's delivery, and contextualized these factors (Booth & Koester, 1996; Latkin, 1998). It also served as a vessel to allow different perspectives in piloting and refining the program while taking into consideration the perspectives of all the major stakeholders, including project staff, participants, and the community members.
Strengths and limitations
This ethnographic process evaluation effort possessed a number of important strengths that could serve as a model for developing other process evaluations for piloting and implementing community-based interventions. First, the whole project team made a commitment to develop an ethnographic process evaluation effort for this study. We built the ethnographic process evaluation strategy into the program during the design phase of the project and revised and developed it along with the piloting project. Second, the ethnographic team developed a systematic strategy for this evaluation effort. We set up an ethnodatabase so that all data could be uploaded and available to the whole project team. We also created a grid of data collection, whereby data were collected vertically (along the chronicle of program sequences) and horizontally (across training session, street outreach, and individual interviews), so that they could be compared more easily across sessions and across piloting “waves.” Third, the evaluation team and implementation team collaborated closely to translate the findings into concrete actions. We established a channel of active dialogue between process evaluation and implementation, and between documented findings and the final intervention program.
Despite these strengths, several limitations should be mentioned. First, the sheer volume of data collected in the process evaluation served to limit this study. Because of the time-sensitive nature of the piloting phase, very little analysis time was permitted between pilot waves, thus limiting our ability to analyze the data thoroughly. Second, it is usually more difficult to collect data in natural settings, and this was the case for our project: More data were collected in the training session than in the street outreach and individual interviews. Third, we did not collect data on comparison groups because of the limited time and resources. It would have been desirable to collect data on communication dynamism in the comparison group, which was an equal attention group.
Nonetheless, it is worth noting that intervention development is both an art and a science, and not all of the changes were based on empirical data as we describe in this study. Many of the major changes and minor refinements were based on practical knowledge and experience with interventions and target populations.
Conclusions
This study illustrates an ethnographically informed process evaluation that was carried out to pilot a network oriented peer outreach HIV prevention intervention program. We presented how to use a systematic ethnographic data collection strategy to collect and manage data and modify the intervention. We specifically listed some exemplary findings and explained the spiral process of translating findings into actions (i.e., to refine the program and implementation). This study has highlighted the efficacy and strengths of ethnographic methods in the process evaluation of piloting an intervention. In traditional intervention development, participation and the interventions do not take place in the natural environments, therefore participants tend to react to demand characteristics of the settings and there is often a gap between what occurs in the intervention sessions and what occurs on the street. Ethnographic methods can help bridge this gap by observing the differences of both sides and adjusting the intervention to the local context. Its interactive features also allow its application in process evaluation to develop culturally appropriate, sustainable and effective interventions.