Abstract
This article addresses a lack of attention in the implementation science literature regarding how to overcome recruitment and retention challenges in longitudinal studies involving large samples of service agencies and health service providers (“providers”). Herein, we provide a case-illustration of procedures that improved recruitment and retention in a longitudinal, mixed-method study—Project Interprofessional Collaboration Implementation—funded by the US National Institute of Mental Health. Project Interprofessional Collaboration Implementation included counselors, program workers, educators, and supervisors. We present a research-engagement model to overcome barriers that included developing a low-burden study, social gatherings to engage stakeholders, protocols to recruit agencies and providers, comprehensive record-keeping, research procedures as incentives to participation, a plan to retain hard-to-reach participants, and strategies for modifying incentives over time. Using our model, we retained 36 agencies over the life of the project. Between baseline (N = 379) and 12-month follow-up (N = 285), we retained 75% of the sample and between the 12- (N = 285) and 24-month follow-ups (N = 256), we retained 90%. For qualitative interviews (between baseline and 12-month follow-up and between 12- and 24-month follow-ups), we retained 100% of the sample (N = 20). We provide a summary of frequency of contacts required to initiate data collection and time required for data collection. The model responded to environmental changes in policy and priorities that would not have been achievable without the expertise of community partners. To recruit and retain large samples longitudinally, researchers must strategically engage community partners. The strategies imbedded in our model can be performed with moderate levels of effort and human resources. Creating opportunities for research partners to participate in all phases of the research cycle is recommended, which can help build research capacity for future research.
Keywords
To advance the implementation of evidence-based practices, we must engage and retain large cohorts of community-based service agencies (“agencies”) and public health services providers (“providers”) in research studies. Nonetheless, providers and the agencies for which they work are often overburdened with large caseloads and lack of time and resources. Therefore, “implementation research is beset by a ‘small n’ problem”; creative methods are required (Proctor et al., 2009). To be relevant and to succeed, researchers must strategically and fundamentally engage community partners. Principles and the best practices of community-engaged research have been employed worldwide to guide the work of researchers, providers, and community members as equal partners in each phase of the research cycle (Belone et al., 2016; Cashman et al., 2008). Researchers have typically been involved in the procedural aspects of research such as advising on recruitment strategies, gaining entry into hard-to-reach populations, and communicating community risks and benefits to their partners. They have also been involved in substantive aspects of research such as the specification of aims, data collection, analysis and interpretation, and dissemination (Spector and Pinto, 2017). A thorough assessment of community-engaged research outcomes has shown improved overall research quality (Viswanathan et al., 2004); however, publications seldom detail methodological practices for recruiting and retaining large cohorts of providers over long periods of time.
To improve practitioners’ use of evidence-based practices, it is essential to document and understand the long-term course of partnerships between researchers and practitioners. The use of checklists of key concern in designing and conducting longitudinal studies, and the employment of well-defined procedures for data collection, have both been suggested to help overcome barriers to conducting longitudinal studies in practice settings such as recruitment and retention of large cohorts (Ployhart and Vandenberg, 2010; Van Weel et al., 2006). When followed, these recommendations can improve researchers’ capacity to collect organizational- and provider-level data over time. For decades, strategies with the potential to improve recruitment and retention in health and public health research have been described in the literature: developing promotional materials that speak of the cultural beliefs of potential participants, offering incentives to participate, and providing childcare and transportation whenever needed (Swanson and Ward, 1995).
More specifically, flexible scheduling of interviews and multiple reminder phone calls have been recommended to sustain communication among research teams and both agencies and individual research participants (Witte et al., 2004). But missing in the literature are strategies for overcoming challenges in recruiting and retaining health and public health providers, particularly for longitudinal studies that require repeated measures and thus demand more time for sustained participation. Potential strategies for overcoming challenges regarding recruitment and retention of study sites (e.g. service agencies) and service providers include involving agency staff as research collaborators to design and develop research methods and procedures (Witte et al., 2004). Isolated strategies to recruit and retain participants over time in longitudinal studies are not in themselves great innovations; however, the engagement of community partners in performing a systematized combination of research tasks and procedures known to have a positive effect on recruitment and retention is innovative (Faridi et al., 2007; Viswanathan et al., 2004). This will be demonstrated by a case illustration below.
Missing data are common in any research effort, but this is particularly true in more complex studies, such as longitudinal and mixed-methods designs, where attrition (loss of follow-up) of study participants leads to decreased sample sizes over time and may cause bias in data analysis and interpretation (Ibrahim and Molenberghs, 2009; Little and Rubin, 1987). The combination of qualitative and quantitative approaches within longitudinal research is feasible and encouraged, but it also introduces further biases that may arise from the absence of quantitative or qualitative data or both (Clark et al., 2015). Studies that call for the collection of longitudinal qualitative and survey data measures may end up with distorted results when there is loss of both types of data. Imputation techniques may be used to correct quantitative analyses (Asendorpf et al., 2014; Biering et al., 2015), but these methods rely on untestable or random assumptions about the lost participants and may thus lead to biased results and some distortion of the findings. Qualitative data can help assuage distortions by describing, explaining, and contextualizing human behavior and social phenomena.
Loss of data may jeopardize data analysis and interpretation, but studies that employ rigorous methods for recruitment and retention have the potential to generate the comprehensive datasets needed for innovative data analysis. For example, multilevel data (from providers nested within agencies) and longitudinal survey data (from repeated surveys across time within the same providers) allow for multilevel structural equation modeling (SEM) to study temporal and interactive relationships at the provider and agency levels simultaneously (Rabe-Hesketh et al., 2004). Moreover, comprehensive datasets, including qualitative data (e.g. archival data or in-depth interviews), allow for rich descriptions of organizational changes, analysis of different levels of influence, and predictive models, including myriad influences on organizational and provider changes over time. These innovations require researchers to secure participation and subsequent recruitment and retention of large samples of agencies, of the individuals who manage these agencies, and of providers who offer services to clients/patients.
Although the literature provides recommendations for best research practices, specific examples or explorations of the steps researchers may take to improve retention and avoid attrition are hard to find. Research on service practices requires a comprehensive examination of both organizational data (e.g. capacity, size, number, and types of services provided) and provider-related data (e.g. demographics, including years of experience, attitudes about evidence-based practices, and frequency of provision of services). Models of organizational evolution have been described in the literature, characterizing organizations facing constraints (the inertial model), those that are highly influenced by environmental conditions (the external-control model), and those in which senior managers determine patterns of change (the strategic-management model; Romanelli and Tushman, 1986). In choosing a research design involving large samples of service agencies, it is thus important to consider and collect data on the influence of managers, service providers, and the environment on organizational changes.
This article addresses key gaps in the literature—a lack of details about how previous studies overcame recruitment and retention challenges in longitudinal studies involving large samples of service agencies and providers, and a lack of specific examples to improve retention and avoid attrition. We have used a case-illustration approach to narrow these gaps. In using a specific research project to illustrate the methodological issues we seek to uncover and address, this article describes tested procedures that improved actual recruitment and retention in a longitudinal study involving 36 agencies and 379 providers. Our case illustration focuses on Project Interprofessional Collaboration Implementation (ICI), a 5-year study (2012–17) funded by the US National Institute of Mental Health (NIMH; R01MH095676). Project ICI is the first longitudinal, mixed-method study designed to examine the impact of collaboration among providers in service agencies (“interprofessional collaboration”) as they implement evidence-based HIV-prevention services (“HIV-prevention services”) in New York City. Providers include counselors, program workers, educators, and supervisors.
With an eye toward narrowing the gaps identified above, herein we demonstrate how we reframed the original study aims to respond to environmental changes (federal and local policy changes) that hindered service agencies’ and providers’ capacity to participate in a longitudinal study. We also provide a model for engaging, recruiting, and retaining large samples of service agencies and providers. The case illustration offers detailed descriptions of barriers to recruitment and retention and how to address them, and examples of two innovations (criteria sampling and SEM) made possible because of the completeness of the datasets produced by Project ICI.
Case illustration
Project ICI is a 5-year, mixed-methods, and longitudinal study whose design includes the following:
Recruitment and retention of 250 providers in 36 nonprofit agencies in New York City, an epicenter of the HIV pandemic, over a baseline and 12- and 24-month follow-ups. Baseline and follow-up surveys were conducted via face-to-face interviews.
In-depth qualitative interviews with a randomly selected sub-sample of 20 providers. Qualitative interviews took place between the baseline and the 12-month follow-up, and between the 12- and 24-month follow-ups. Participants were selected using criteria sampling and provided face-to-face interviews.
Training providers in collaborative practices to improve implementation of evidence-based HIV-prevention services. Project ICI training took place between the baseline and the 12-month follow-up and it was meant to spark interprofessional collaboration among the providers in the study.
Table 1 provides a summary of Project ICI’s design and methods. Our sequential mixed-method approach is grounded in diffusion of innovations, institutional, and behavioral theories. Diffusion of innovations and institutional theories suggests that implementation involves providers who routinely facilitate evidence-based interventions in agencies of local diffusion systems (Ajzen, 1991; Ajzen and Fishbein, 1980; Gotham, 2004; Lehman et al., 2002; Rogers, 1995). Reasoned action and planned behavior define the key factors influencing implementation: providers’ attitudes, self-efficacy, and subjective norms. Project ICI examines the role of interprofessional collaboration in resolving day-to-day challenges providers face as they implement HIV-prevention services. The project is grounded in the evidence that interprofessional collaboration—mutual referral making, information sharing, service evaluation, and outcome dissemination—may help improve agencies’ and providers’ ability to deliver HIV-prevention services.
Summary of Project ICI design and methods.
ICI: Interprofessional Collaboration Implementation; NIMH: National Institute of Mental Health; CDC: Centers for Disease Control and Prevention.
It has been well documented that single-cohort, long-term longitudinal survey design has key advantages over cross-sectional survey design, including the opportunity to collect baseline information about demographics and cognitive aspects of participants, and to track these variables over time. Longitudinal data provide insight into how natural changes occur and thus allow for behavioral prediction. Nonetheless, long-term longitudinal survey research can delay results and suffer from cumulative attrition. Therefore, use of multiple single cohorts, each one starting at a different age, has been recommended for achieving the benefits of longitudinal methods while reducing attrition (Galbraith et al., 2017). Since our research participants were service providers, their ages were important in that the literature shows that age is related to provider use of research findings in practice (Chagnon et al., 2010; Owczarzak and Dickson-Gomez, 2011; Pinto et al., 2013; Spector and Pinto, 2017). Moreover, regardless of the age at which a provider starts his or her career, providers learn more about all issues related to practice as time goes on. Therefore, Project ICI involved service providers of different ages at different points in their careers.
Study context
The literature suggests that in choosing a research design involving large samples of service agencies, researchers ought to consider the influence of the socioeconomic environment on organizational changes (Romanelli and Tushman, 1986). Project ICI’s procedures have been influenced by environmental changes that will likely not be unique to HIV-prevention studies, given global economic and public health policy shifts. We present these changes and then discuss their implications for future studies. Initially, Project ICI aimed to study exclusively the Centers for Disease Control and Prevention (CDC) Diffusion of Effective Behavioral Interventions (DEBI) program, which took off in 1999 (CDC, 2010). Dissemination of evidence-based HIV prevention in the United States has taken place largely through the DEBI program, in which agencies received funding and technical assistance (TA) to deliver “packaged” behaviorally based programs that had been selected as “best evidence” programs (CDC, 2013; Collins et al., 2006). This national approach is based on best evidence from Kelly et al. (2000a, 2000b); it offered paper-based manuals, training workshops, and follow-up TA.
Between 2012 and 2017, despite the need for its benefits, DEBI implementation remained limited (Durlak and DuPre, 2008; National Institutes of Health, 2006; Norton et al., 2009; Tinkle et al., 2013; Witte et al., 2015). The HIV-prevention field was experiencing shifts in scientific priorities based on changing global health policies and economics. These shifts included movement from behavioral to biomedical prevention approaches; promotion of behavioral couple-based models to overcome barriers in traditional individually based models; and the recognition that efficacious multisession programs need to be briefer for sustainability in service settings and that programming should target individuals living with HIV and their intimate partners as opposed to HIV-negative individuals from the general public or from specific high-transmission risk groups (Mermin and Fenton, 2012; Rotheram-Borus et al., 2012). The United States was also slowly rebounding from an economic recession. This context, in light of the traditional barriers to implementation efforts in real-world HIV-services agencies, made it even more challenging for agencies to participate in research efforts.
As a part of these priority shifts, in 2011, the CDC began to deemphasize the DEBI program and launched the High-Impact HIV Prevention approach, emphasizing only the interventions that were most cost-effective and whose key behavioral changes related to HIV testing and primary care access (CDC, 2011, 2015). The high-impact approach requires providers to offer individuals of unknown HIV status access to HIV testing, primary care, biobehavioral interventions, support services, and structural interventions such as syringe-exchange programs.
This consolidation of programming was taking place just after the economic recession that began in 2008, which had a devastating effect on HIV-services agencies in New York State; some lost as much as 30% of their funding within 3–6 months. It was already a time of low or dwindling resources and of considerable organizational disruption in the provision of services due to the recession and related cuts. Together, these events resulted in many agencies losing funds and resources, thus hindering their ability to focus on anything but recovering from their losses. Many agencies had to dismantle entire HIV-prevention departments and terminate scores of providers, then hire and train less-experienced staff to fill the gaps. Service providers, a workforce characterized by high burnout and turnover, became more difficult to recruit and retain. Although these challenges appeared unique to this project, changes in the fiscal and geopolitical environment are relevant to any research project. Therefore, how we addressed and adapted to a new context in response to external changes may help researchers facing different challenges in other geographic contexts. Below, we describe how best to engage and retain large numbers of service agencies and providers over long periods of time in longitudinal studies. Specifically, we provide a model of community engagement and set forth the specific adaptations we made to our research procedures in order to respond to policies affecting the agencies and providers involved in Project ICI.
Research-engagement model
Globally, community-engaged research is the leading paradigm with a theoretical and empirical basis for understanding and advancing implementation readiness, for evaluating collaborations between researchers and service providers, and for developing strategies to boost the acceptability of health interventions (Blevins et al., 2010; James et al., 2013; Maar et al., 2015). Project ICI adheres to principles and practices of community-engaged research, encouraging researchers to involve community partners (e.g. providers) in each phase of the research cycle—from defining study aims to protocol development to dissemination of findings (Israel et al., 1998).
The basis for a research-engagement model
Using qualitative and quantitative data, our research team has conducted several studies to examine provider willingness to become involved in research. Our work shows that providers’ perceptions of researchers’ availability, the benefits of research, and agency preparedness are associated with their willingness to engage with researchers to pursue scientific research (Pinto, 2013). Our evaluation of mediated and nonlinear relationships between personal and agency factors and provider willingness to become involved in research has generated a predictive model of practitioner involvement: the Provider-Researcher Partnership Model (Pinto et al., 2014), showing that providers favor “balanced partnerships” in which researchers and providers conduct research tasks and procedures in ways that reflect their unique knowledge and skill sets. Our work also reveals that research conducted in partnership with community collaborative boards, involving agency representatives, providers, and individuals who receive services in these agencies, improves recruitment and retention in large studies. Based on these findings, and on the most rigorous review of the community-engaged research literature presented by Viswanathan et al. (2004), we developed and conducted Project ICI, including its design and methods (e.g. recruitment and retention procedures). Clear evidence indicates that all the strategies we describe below are effective in improving recruitment and retention. It also suggests that combining different tasks and procedures may increase the impact of each individual strategy.
Implementation Community Collaboration Board processes and dynamics
Project ICI’s methods were developed and conducted by the Implementation Community Collaborative Board (ICCB), composed of 8–10 university researchers, providers, managers, and consumers whose unique contributions included research, practice experience, delivery of behavioral interventions, and intervention facilitation. The ICCB was developed using six published steps meant to engage members in procedural and substantive research roles: engaging members, developing relationships, exchanging information, negotiating and making decisions, retaining members, and studying dynamic processes (Pinto et al., 2011, 2015). ICCB members are trained in community-engaged principles and practices, protection of human participants, cultural competence, survey development, recruitment scripts, and data collection and analysis. The ICCB generally meets monthly (sometimes every 2 weeks or every 2 months, based on the methodological needs of the project). Meetings are usually 2 hours long. Participants receive appropriate compensation for attending meetings and for work outside meetings (e.g. drafting surveys and qualitative protocols, and collecting and interpreting data).
During meetings, we use dialectic processes, mutual support, problem-solving, and procedural triangulation to facilitate the implementation of research methods and to respond to environmental challenges that may hinder the recruitment and retention of research participants. For example, in order to develop the Project ICI survey instrument, ICCB members worked individually to create questions in specific areas of inquiry that matched their HIV skills and knowledge. These questions were then examined by the group during scheduled meetings. Discussions allowed for the exchange of knowledge and experiences (dialectic processes) that resulted in survey questions that could be answered by providers with diverse levels of HIV knowledge and skills. ICCB members expressed their individual ideas while supporting other members’ ideas despite differences of opinion (social support), and combined all the ideas into one project. In working out differences of opinion (problem-solving) about the best language to use in the survey questions, the sequence of questions, and the length of the survey, ICCB members collectively created a survey that was not only valid but readily accepted by the providers who participated in the study.
The ICCB reframed Project ICI’s aims to respond to new policies and changing contexts. The original study aims and corresponding methods and procedures had focused exclusively on issues regarding dissemination of the effective behavioral interventions from the CDC. The launching of the high-impact approach, in 2011, gave us the opportunity also to study myriad HIV-prevention services encompassed by the new program, such as expanded HIV testing and links to primary care (National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention, 2015). Our modified aims reflected the mandate requiring providers to engage in the HIV Continuum of Care and in the Test and Treat Approach. This approach integrates myriad services (e.g. HIV-, STI-, and hepatitis-C testing; primary care; and mental health services) to maximize the public health impact and calls for complex longitudinal designs, large samples of providers retained over time, and mixed methods to capture providers’ implementation behaviors over time (Palinkas and Soydan, 2011).
The ICCB has been instrumental in developing strategies for conducting research in response to the consolidation of programming at the federal level and its local consequences, such as the loss of funding and resources, dismantling of HIV-prevention departments, and termination of scores of providers, to strengthen methods for future research efforts. Specifically, the ICCB broadened the scope of the study and developed new recruitment and retention strategies, as described below.
Recruiting and retaining service agencies and providers longitudinally
Strategies for obtaining responses to surveys have been described in the literature. One strategy commonly used is Dillman et al.’s (2014) Total Design Survey Method (TDSM), which offers a sequence of specific steps researchers need to take in order to secure participation in survey research. Grounded in social-exchange theory, the TDSM helps researchers identify ways to increase the number of and enhance the quality of survey responses. The TDSM has shown to yield 75%–80% return rates for mail and telephone surveys. It encourages researchers to systematize data-collection tasks and procedures, to convey to survey respondents the importance of the research, and to help respondents weigh the costs and benefits of participation. The method was developed for using both mail and telephone surveys, but its basic tenets have been updated over the years and have been helpful in implementing face-to-face, Internet, and mixed-mode surveys. Dillman et al. recommend that researchers follow steps such as sending cover letters describing the purpose of the research and why respondents’ opinions are needed, and follow-up letters to nonresponders formatted in a specified manner.
Designs calling for in-person survey and qualitative-data collection require techniques beyond those provided by Dillman et al. As other researchers have done in the past, we followed the fundamental ideas contained in the TDSM to guide the specific tasks and procedures for Project ICI, but added several other strategies that improved recruitment and retention in Project ICI, which may do the same for studies with more complex designs (Hoddinott and Bass, 1986). The tasks and procedures developed for Project ICI are innovations that went far beyond those recommended for cross-sectional, in-person, mail, and Internet surveys. These included developing a low-burden study; social gatherings to help engage stakeholders; protocols to identify a point person at each service agency who would help recruit service providers; comprehensive record-keeping; research procedures that served as incentives to participation; a plan to retain hard-to-reach participants; and strategies for modifying incentives to participation over time. These are described below.
Social gatherings
In order to get buy-in from service agencies, we went through a pre-study phase in which we held recruitment brunches—community-focused events to engage agencies. Our goals for these events were to make the research team available for one-on-one conversations with agency representatives and providers, to communicate to them the opportunities offered by the study, to understand their concerns about the time commitments for their participation, and to create a safe space for discussing research aims and methods. ICCB members were instrumental in this phase of the project; they came to the social gatherings and spoke about Project ICI from their own experience. Managers and decision-makers heard well-informed presentations about Project ICI from providers who had been directly involved in the development of the grant that funded the project. Stakeholders from many agencies revealed that they made the decision to participate in the study because of the providers’ endorsement and direct knowledge of the project. Momentum from the brunches carried us into our first phase of data collection. We began at agencies where we had the closest partnerships because of past collaborations or the presence of ICCB members on the agency staff. We held an additional event 2 months later with more agencies. Eight of these agencies joined the study several months after we had held the first recruitment brunch, demonstrating what may be the long-term influence of community-based gatherings.
Barriers to recruitment and how to address them
In our attempts to recruit service agencies, we experienced pushback. Some agencies did not wish their providers to make even the minimal time commitments; others felt their internal systems were too overwhelmed for them to add promotion and explanation of the study to their internal conversations.
Developing low-burden studies
In order to facilitate recruitment of providers, we emphasized to agency representatives that participation in Project ICI would involve a low burden. Members of ICCB provided incentives and traveled to each agency for data collection so that providers did not need to travel. We scheduled our data collection at times that were convenient to both agencies and providers; we surveyed up to four participants at a time, with a time commitment of only about 1 hour per person.
Identifying a point person
Each participating agency designated a “point person” to coordinate interview schedules and obtain the approvals necessary to confirm participation. Point people were often administrators with multiple duties, and we struggled because these individuals sometimes neglected to return emails and telephone calls. In response to this challenge, we used e-blasts and personalized phone calls from the principal investigator or the Project ICI director. We also faced challenges in recruiting and scheduling individual providers. Providers were often called away to see clients just when they were scheduled to take our survey. Client needs and program crises extended beyond the providers’ available time. Often, if providers were called away to address a program issue, they expressed a commitment to the interview and made attempts to resume or reschedule the interview for the same day, but this process frequently ended in rescheduling. Sometimes, providers were ill or occupied with family or other personal issues.
Developing comprehensive record-keeping
We developed and established a system of record-keeping, including a Google calendar for appointments and a detailed and standardized checklist for data collectors. All research staff were trained in how to implement and document each item on the checklist, including the informed-consent script; signed consent forms from each participant; agency and personal contact information for all participants; notes on data collection for each agency and for providers; and signed receipts for incentives. We found at every stage that good record-keeping improved our ability to have a fluid workflow and retention of participants. This also allowed for easy replacement of support staff—an important consideration because most longitudinal projects face research-staff turnover.
Giving back to the community as a facilitator of retention
Community-engaged research calls for opportunities to give back to those who make time in their lives to be involved in research and who willingly give myriad types of data that are used by researchers to advance science and inform both practice and policy (e.g. in the areas of social work, medicine, and public health). Here, we discuss three strategies from our community-engaged model, grounded in the literature discussed above, which have helped us with recruitment and retention in several studies, including Project ICI.
Appealing to altruism and stressing Project ICI benefits
In community-engaged research, the study goal is rooted in community members’ interests and intentions. Our providers, including agency administrators, work in HIV-prevention services; this is a service community with a long history of loyalty generated from personal connection with the communities served and activism to improve access to care. During all phases of the study, staff were trained and supervised to remind participants of the potential benefits study participation and success might offer to the local community. Providers shared feelings of strong commitment to their clients and the larger community. A great many providers expressed interest in the dissemination process—the use of findings as they related to the betterment of HIV-prevention services. Personal benefits experienced by participants were amplified and these supported the motivation to be “part of the solution” by increasing interprofessional collaboration across agencies.
Offering training as incentives for research participation
After the baseline was established, providers were offered ICI training (Table 1), a full day of programming delivered in line with CDC’s vision of improved collaboration to enhance HIV/AIDS prevention (CDC, 2009). ICI training was developed by the ICCB; it involved conceptualizing goals, specific modules used to teach participants the benefits of evidence-based services, subjective norms and barriers to interprofessional collaboration, self-efficacy techniques, and specific steps participants could use to engage in collaboration. All members of the ICCB participated in the pilot training, and several taught different modules of the training when it was offered in the full-day workshop. ICI training was provided between the baseline and the 12-month follow-up, and it served as an opportunity for us to solidify our partnerships with participating agencies and providers. Interactions prior to and during the training helped participants develop a sense of belonging to a cohort and thus being committed to one another’s retention for the 24-month follow-up. Some 142 study participants from 34 agencies attended the trainings. Providers’ schedules, workloads, and personal and family issues were identified as obstacles to attendance. Agencies found it challenging to afford staff a day for training due to work responsibilities and program needs (e.g. limited staff, client crises, and mandatory program events). There were instances in which providers scheduled to attend the training had to miss it because of personal obligations. Following the advice of the ICCB, the research team responded by scripting and shooting a 37-minute video version of the training that could be shown to our participants at their agencies and at their convenience. Several ICCB members participated in writing and performing in the video. The video followed the same arc as the live training, and it discussed all the same elements of collaboration. Although interactive group activities could not be included in the video, stopping places were built into the video (whose total training time was 60 minutes), so that those watching could stop to discuss critical points and engage in question-and-answer sessions. An additional 84 providers were trained using the video approach. The ICI training, in both forms, served as an engagement tool, providing a needed service for the community of agencies and providers in the study.
Sharing research findings with the community: Project ICI dissemination symposium
Community-engaged research calls for the dissemination of findings through different media; specifically, it calls for researchers to make certain research findings available to service providers, managers, stakeholders, and policy makers so that the findings can be used to guide practice and policy. Research findings are commonly shared through publications in journals, but such results often do not extend past the academic community. Community providers are often engaged in different stages of the research cycle, but are seldom reengaged after the research has been completed. Reengagement after data collection, training, and other research participation is needed to help researchers contextualize findings and empower members of the community to use those findings in their lives—in this case, to empower providers, managers, and policy makers to use findings to guide practice and policy.
In order to decrease the gap between research and practice, Project ICI, at the end of its final year, hosted a dissemination symposium to bring together community members, providers, administrators, and researchers (Ahead Magazine: Social work research at University of Michigan, 2017).
All Project ICI participants were invited to attend this event. An important incentive for providers who participated was continuing education units (CEUs) for training. Social workers who attended were eligible for 6 hours of CEUs, so the incentive was equivalent to an opportunity for professional development. Symposium participants represented a cross-section of HIV-prevention and treatment professionals in New York City. The event offered an opportunity for researchers to tackle difficult questions with providers, explore their experiences, and discern areas of impact for future research; participants had their opinions and experiences validated by peers.
The 6-hour day included social activities such as breakfast, lunch, and afternoon coffee. These activities echoed how Project ICI started with social gatherings to offer opportunities for providers to network and compare and contrast their experiences with the research project. Keynote addresses—by Dr Chris Gordon, chief of the HIV Treatment and Translational Science Branch at the Division of AIDS Research at the NIMH, and Dr Julie Myers, the director for HIV prevention at the New York City Department of Health and Mental Hygiene—focused on the importance of community engagement in research on combating HIV. Most important, nearly 60 participants heard from and engaged with a panel of administrators, providers, and researchers who discussed and analyzed the Project ICI findings. After the panel, ICCB members led discussions with small groups of participants about the issues revealed by the findings. This afforded us an opportunity to ask providers to help researchers interpret the findings; this represents a large-scale example of “member-checking,” a technique used in qualitative research that involves including research participants in the analysis and interpretation of findings (Padgett et al., 2012). An issue of Ahead, the research magazine published by the University of Michigan School of Social Work, highlighted the dissemination symposium and was distributed to all Project ICI participants and to administration at participating agencies.
Barriers to cohort retention and how to address them
When we began to collect the 12-month follow-up surveys, we realized that provider attrition rates were high due to the environmental challenges discussed above. In some cases, the point person himself or herself had left the agency. The project staff was typically notified of such situations during contact attempts. This information was relayed in various ways: by a new employee who had filled the position, by another participating provider, or even by a receptionist who would then refer us to an alternate representative within the agency. Not until we started reaching out to this “hard-to-reach” group did we come to understand the myriad challenges we would have in getting in contact with each provider.
Developing a plan to retain hard-to-reach participants
We found that the information we had for many providers—home addresses, personal email addresses, and personal phone numbers (usually for mobile phones)—was no longer valid. We took note of those who did not respond to our outreach efforts—those whose postal mailings were returned to us, or whose email accounts, personal or professional, bounced our messages back. Returned postal mail, bounced emails, and a lack of responses to phone calls were the only indications that certain individuals’ contact information had changed. In retrospect, we would like to have had some indication about how many participants might become hard to reach. We did not completely understand why there was such an attrition rate. We knew the roles policy and priority changes had played in recruitment and retention, but could attrition also be due to job dissatisfaction, or to economic or demographic trends at participants’ workplaces? Were people taking positions at new agencies that would offer better benefits or working conditions? As we began to interview hard-to-reach participants, we learned that all these issues were present. Agencies experiencing downsizing tended not to have the capacity to absorb providers into other roles within the agencies. Providers who had left their agencies of their own accord typically expressed a desire for better supervision or felt undervalued and deserving of more compensation for the quality of work expected in the role.
We pursued hard-to-reach individuals by mailing a letter with an ICI Project fact sheet to all for whom we had mailing addresses. We then followed up by calling all those from whom we had not heard. We called each individual twice within the space of 2–3 days, most often leaving messages that reacquainted participants with Project ICI, informed them of our need to follow-up, and offered to come to their new workplaces to survey them. If we received no response from a participant, we made a third attempt, usually via text message. Provider recall of and reengagement in the study were reinforced by the ability of project staffers to provide details from the previous interview process. The project director supported this process through supervision and information-sharing with project staff. Details could include the location and time of year of the interview, idiosyncrasies involving the scheduling of interviews or shared life events, and reminders of names of the project staff members who had been involved in the scheduling process.
Working with Institutional Review Boards to modify incentives
We based our decisions about how much to use incentives to recruit and retain participants based on the well-known concept of undue influence, which occurs when compensation or incentives are used to induce potential research participants who otherwise would not participate in a study. Grounding its thoughts in community-engaged research best practices, the ICCB discussed this issue several times and consulted with the Institutional Review Board at our institution. Our main concern was to avoid inducing providers, particularly those with lower salaries, to make decisions that were not in their best interests (Williams and Walter, 2015). We resolved the retention issues by requesting approval from our Institutional Review Board to double the incentive we were offering each point person for facilitating recruitment for the 24-month follow-up. Furthermore, we also nearly doubled the incentive for providers to participate in the 12- and 24-month follow-up interviews. Although the cost of conducting the study grew higher, we know that the datasets we have developed, because of their completeness, have allowed us to use innovative sampling strategies for a major component of the mixed-method design (in-depth interviews) and for quantitative data analyses, both described below.
Lessons learned
With an eye toward narrowing identified gaps in the literature, we sought to demonstrate how researchers can respond to environmental changes (such as federal and local policy changes) that hinder participation in research, particularly in longitudinal studies characterized by large samples of service agencies and providers. We provided a model of research engagement shown to yield rates of recruitment and retention higher than those for previously described strategies for cross-sectional studies. Project ICI is a complex study whose design required engagement of a large number of service agencies in order to yield a dataset containing survey data for 250 individual participants at the baseline and 12- and 24-month follow-ups, and another dataset containing in-depth interview data for 20 providers. Moreover, the study provided a full-day (N = 142) or video (N = 84) training session on interprofessional collaboration to a total of 226 providers.
We learned that our research-engagement model yielded high rates of retention in its collection of survey data at the baseline and the 12- and 24-month follow-ups; of qualitative data between the baseline and the 12-month follow-up and between the 12- and 24-month follow-ups; and of interprofessional collaboration training between the baseline and the 12-month follow-up. By developing a comprehensive series of tasks and procedures, we were able to recruit 36 service agencies in New York City and a total of 379 providers at the baseline. We retained all 36 agencies over the 5-year study, even though two pairs of agencies merged between the baseline and the 12-month follow-up, resulting in a final sample of 34 agencies. Using the community-engaged model, between the baseline (N = 379) and the 12-month follow-up (N = 285) we retained 75% of the sample, and between the 12-month follow-up (N = 285) and the 24-month follow-up (N = 256) we retained 90% of the sample. For the qualitative interviews, we retained 100% of the sample. This degree of retention appears to surpass return rates for studies with simpler designs described in the literature (Dillman et al., 2014; Hoddinott and Bass, 1986).
By using an innovative approach—a systematic combination of specific tasks and procedures over time—we achieved excellent recruitment and retention rates. Because we retained a large number of agencies and providers over time, we have datasets with crucial data that have allowed us to engage in innovative sampling and data analyses that would not be possible without complete datasets. For example, in selecting providers for in-depth interviews, we used criterion sampling to focus on a predefined group of participants (Patton, 2002; Ragin and Becker, 1992). We selected 20 providers out of 250 participants who provided survey data at the baseline and the 12- and 24-month follow-ups. We based our selection on two variables from the survey: the frequency with which the providers made links to HIV services and interprofessional collaboration experience. This allowed us to select providers with different patterns of link-making (“low” and “high” link-makers) and collaboration (“low” and “high” collaborators). We randomly assigned five providers to each of the four groups (N = 20), and interviewed them between the baseline and the 12-month follow-up, and then between the 12- and 24-month follow-ups. Another innovation achieved only because of the completeness of the dataset was the combination of multilevel survey data (from providers nested within agencies) and longitudinal survey data, which allowed us to conduct hierarchical or “multilevel” SEM, which allows temporal regression-type relationships to be examined simultaneously. Multilevel SEM allows provider- and agency-level predictors and outcomes to be treated (when appropriate) as latent variables, and allows for joint modeling of longitudinal trajectories.
The speed and innovation with which we responded to environmental and historical changes in policy and priorities (without compromising rigor) would not have been achievable without the expertise and commitment of our community partners. US national directives and funding opportunities continue to urge researchers to engage community partners, yet few studies fully describe community-engaged principles and practices (Institute of Medicine, 2000; National Institutes of Health, 2014; Proctor et al., 2009; United States Department of Health and Human Services, 2006). This article helps fill this gap by providing a detailed account of how our model for community engagement helped us improve Project ICI’s aims and procedures. Our model offered us the immediate capacity to develop sustainable solutions to changes in policy and priorities that might have hindered our study design and methods. Therefore, it has potential application for other studies.
Implementation-system policies are fluid, and science must keep pace, yet we seldom publish our efforts or relate how we address research challenges. Project ICI was developed with a collaborative board whose members have firsthand experience with the practices—and the policy shifts that alter practices—at community-based agencies. The involvement of community experts has helped us and will continue to help us respond to environmental shifts and improve recruitment and retention. Box 1 shows a summary of the time that was required for data collection, the frequency of contacts required to initiate data collection, and the time required for training 226 providers. These data are not sufficient to conduct a cost–benefit analysis; however, our team will keep collecting this type of data in future studies so as to be able to describe and weigh the costs and benefits of using a research-engagement model. Since we work under the paradigm of community-engaged research, we do not possess any data to show differences between our model and other models of engagement, recruitment, and retention. Nonetheless, our efforts to recruit and retain agencies and providers adds evidence to a large body of research showing that community engagement can improve overall recruitment and retention (Viswanathan et al., 2004). Our work further validates how community-engaged research puts researchers at an advantage when responding to environmental changes that otherwise might hinder research design, methods, and procedures. The strategies provided here have implications for research using different designs and methods in myriad other contexts.
Recruitment and retention of 36 agencies and 250 participants.
Project ICI has public health implications because it promotes the implementation of evidence-based HIV-prevention services. Knowledge of how interprofessional collaboration influences service implementation informs the diffusion of various services across multiple HIV-prevention delivery systems. This project’s design will enhance our capacity to interpret longitudinal patterns, especially those patterns influenced by such factors as scarcity of resources and provider turnover faced by participating agencies during the study. An advantage of conducting the longitudinal study in New York City is the diversity of providers and agencies that will facilitate the ability to generalize our findings.
These lessons learned may also be applicable in low- and middle-income countries. Our model of community-engaged research has been used to guide health research in Brazil, Mongolia, and Kazakhstan and has been published elsewhere (Pinto et al., 2014). We believe that the strategies we used here can be performed with moderate levels of effort and human resources (see Box 1). This is particularly true in the initial phases of partnership-building and formative low-budget research, when partners may have only seed funding. Creating opportunities for research partners to participate in all phases of the research cycle is recommended; this can help build research capacity usually absent in low- and middle-income countries. Research partners can be involved in data collection—particularly through qualitative interviews, which can generate findings more accessible to lay individuals than are quantitative ones.
Conclusion
We wish to encourage researchers and their community partners to watch for policy and priority shifts that might present an opportunity to expand the scope of their research aims to respond to new trends. Researchers will need to maintain partnerships with policy makers, who are the primary sources of information about new practices and the policies guiding them. We hope that researchers, guided by community-engaged values and practices, will disseminate any and all methodological innovations that might enhance recruitment and retention of research participants. Our work suggests that engaging and retaining community partners over time may help build structural capacity—for example, through ongoing collaborative research boards. Community experts on these boards who are involved in the conceptualization of research aims will, from one research project to the next, improve their capacity to formulate solutions to the problem of attrition in research. Capacity can also be built by establishing a thorough and detailed system of record-keeping, by training staff and community partners in how to use that system, and by evaluating the system and making changes to address environmental challenges.
Footnotes
Acknowledgements
We wish to thank all members of the Implementation Community Collaborative Board who generously provided inspiration and ideas for this paper and for the grant proposal that funded this study: Angela Ghesquiere; Charles Sanky; Jeannette Ruffins; Rosa Bramble; Karen Baird; and Wendy Whitman.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research presented here, the preparation for this manuscript, and publication of this article were supported by NIH grant R01MH095676 (Principal Investigator: R. M. Pinto).
