Implementation of Best Practices Regarding Treatment Fidelity in the Family Colorectal Cancer Awareness and Risk Education Randomized Controlled Trial

Abstract

Treatment fidelity is associated with improvement in research outcomes and increased confidence in significant findings. However, few studies report on recommended areas of treatment fidelity (i.e., study design, training, treatment delivery, treatment receipt, and treatment enactment), leaving a dearth of information about implementation components that contributed to a study’s success. Without such information, it is difficult for researchers to correctly assess previous findings and for practitioners to correctly implement findings into practice. Thus, it is crucial that studies assess both treatment fidelity and applicability of treatment fidelity findings. We report measures of treatment fidelity in a randomized controlled trial of an intervention promoting colonoscopy in at-risk relatives of colorectal cancer (CRC) patients. We describe assessments related to both treatment delivery and treatment receipt. We conducted separate ANCOVAs to model the change in each of the treatment receipt variables, comparing the two intervention arms. Compared with the control group, the intervention group had significantly greater improvements in CRC knowledge (f = 17.46, p < .0001), perceptions about susceptibility (f = 15.08, p = .0002), response efficacy (f = 7.46, p = .0076), self-efficacy (f = 8.16, p = .0053), and reduced decisional uncertainty (f = 19.59, p < .0001) from baseline to 1-month follow-up. Overall, our study adhered to most of the best-practice guidelines for behavioral intervention fidelity. This demonstrates that our intervention was delivered as intended and positively affected the cognitive processes that are purported to be predictive of adherent behavioral outcomes.

Keywords

treatment fidelity cancer screening colorectal cancer cancer risk behavioral intervention

Introduction

Treatment fidelity is defined as “the ongoing assessment, monitoring, and enhancement of the reliability and internal validity of a study” (Borrelli, 2011, p.1). Adherence to and reporting of treatment fidelity may seem like obvious tasks in any study, yet they are rarely published in relation to specific behavioral intervention trials (Resnick et al., 2005). The lack of assurance that a study has been conducted according to protocol opens up the door to error when new studies build upon previous outcomes. Monitoring and reporting treatment fidelity assure scientific transparency in a way that facilitates reproducibility, as it both reports on and appraises study process methods. In addition, treatment fidelity measures can be used to facilitate theoretical testing, to provide an outline for research dissemination in clinical practice, and to reduce random and unintended variability (Bellg et al., 2004). The 2004 National Institutes of Health (NIH) Behavior Change Consortium recommends five areas of treatment fidelity best practices: study design, training, treatment delivery, treatment receipt, and treatment enactment (Bellg et al., 2004).

In an analysis of 342 behavior change studies, Resnick et al. (2005) found that only 15.5% of studies met the NIH Behavior Change Consortium’s standards for high treatment fidelity reporting. Such low numbers indicate a dearth of important information about the processes by which behavioral science research is being conducted. More recently, the 2004 treatment fidelity recommendations have been expanded, providing specific criteria for treatment fidelity assessment in each of the five fidelity categories (Borrelli, 2011). In general, studies with higher treatment fidelity (according to the checklist of the 25 treatment fidelity attributes) were more likely to demonstrate significant results than studies with low treatment fidelity (Resnick et al., 2005). These findings highlight important considerations for those interested in optimizing intervention research.

Yet, the importance of treatment fidelity to future research has been contested by some researchers who argue that strong adherence to fidelity measures presents potentially important limitations both to the study implementing these procedures and to future research dissemination. In an article responding to Bellg et al. (2004), Leventhal and Friedman (2004) presented concerns that treatment fidelity may impede personalized medicine; if providers must adhere to rigorous guidelines, it may inhibit their ability to provide individualized care where a situation requires it.

Leventhal and Friedman (2004) also questioned the validity of treatment receipt, asserting that true comprehension of a concept may be a more complex process than the ability to regurgitate knowledge. Furthermore, they argue that understanding may not be necessary in facilitating behavior change. Finally, they expressed concerns that the idea of treatment enactment may be indistinguishable from treatment adherence (or study outcome). Leventhal and Friedman (2004) contested that this concept of enactment is highly dependent on the kind of study being conducted, and may not be measurable under all circumstances. If an intervention does not involve participants learning a set of measurable skills, it may not be practical to attempt to assess for enactment.

In addition, Leventhal and Friedman (2004) and others (Cohen et al., 2008; Hawe, Shiell, & Riley, 2008) have argued that treatment fidelity, while potentially useful in providing confidence in outcome efficacy, may actually impede flexibility when translated into clinical practice or when implemented in future studies where conditions may differ in both the populations being targeted and the resources available. They question the value of treatment fidelity against the reality of adaptation in real-world settings. Hawe et al. (2008) echoed this concern in their response to Spillane et al.’s (2007) report of treatment fidelity in a randomized controlled trial on heart disease prevention, noting that interventions that are tailored to context may be more effective than studies intent on direct translation according to fidelity measures. Cohen et al. (2008) found that changes to interventions when translated into practice settings are both common and, often, unanticipated.

These critiques present important considerations to examine for those who wish to implement treatment fidelity best practices in their study interventions. Critical assessment and evaluation of treatment fidelity measures increase confidence that Type I or Type II errors did not occur, thereby providing assurance that unplanned effects of protocol deviations do not lead to acceptance or rejection of study hypotheses. However, research that identifies treatment fidelity must also be willing to discuss how to incorporate elements of successful studies into practical, real-world scenarios, to allow for tailored implementation.

This article aims to address both the identified benefits and the potential limitations to treatment fidelity reporting by examining treatment fidelity in the context of the Family Colorectal Cancer Awareness and Risk Education (Family CARE) Project, a randomized controlled trial of an innovative telehealth intervention that aimed to improve colonoscopy uptake in individuals considered to be at intermediate risk of familial colorectal cancer (CRC). Specifically, we examined and evaluated treatment fidelity as defined according to the five best-practice areas (Bellg et al., 2004, Borrelli et al., 2005). We also identified and assessed potential limitations to existing treatment fidelity measures, and discuss the translatability of our intervention to future research and clinical practice.

Method

Study Design and Participants

Family CARE was a randomized controlled trial investigating whether a personalized, remote, theoretically based risk-assessment, and counseling intervention was more effective than a mailed, low-intensity targeted print message delivered to individuals who are considered at intermediate familial risk for CRC. Research staff conducted a three-tiered recruitment process: (a) identifying CRC patients through population-based cancer registries, (b) contacting these patients to request information about their at-risk relatives, and (c) contacting identified relatives (Simmons et al., 2013). At-risk family members who enrolled in the study were randomized by family unit to either the tailored intervention group (TeleCARE) or the minimal intervention group (PRINT). A detailed description of the study methods and outcome is provided elsewhere (Kinney et al., 2014; Pengchit et al., 2011; Simmons et al., 2013). In all, 481 eligible intermediate-risk family members of CRC patients, representing 337 families, were enrolled and randomized between September 2009 and August 2011. The Institutional Review Boards of participating institutions approved all study procedures. Figure 1 provides a CONSORT diagram from initial assessment through 1-month post-intervention evaluation.

Figure 1.

Consort diagram.

We assessed treatment fidelity using the checklist created by the NIH Behavior Change Consortium (Bellg et al., 2004) and expanded by Borrelli (2011). Figure 2 outlines the absence or presence of best practices in the Family CARE study. Items that could not be implemented, due to study design, are noted. After outlining these best practices and their implementation, we evaluated fidelity measures of treatment delivery and treatment receipt.

Figure 2.

Adherence to treatment fidelity best practices^a: Family CARE trial.

Treatment Fidelity Adherence Step 1: Study Design

Theoretical models

We primarily used the Extended Parallel Process Model (EPPM) to guide the TeleCARE intervention (Pengchit et al., 2011; Witte, 1992). The EPPM postulates that effective risk communication is optimal if individuals believe that the threat of a disease or health condition is personally relevant (perceived susceptibility) and serious (perceived severity), that the recommended action will help mitigate their risk (response efficacy), and that they have the tools necessary to execute the risk-mitigating action (self-efficacy; Witte & Allen, 2000).

We used a motivational interviewing (MI) style of counseling for the TeleCARE telephone counseling facet of the intervention. MI is a directive counseling technique traditionally used when patient ambivalence is a barrier to positive action (Miller & Rollnick, 2009). Counselors use questions and statements to strategically build rapport, elicit talk in support of change, and collaborate with the patient on a plan. Detailed explanation of the theoretical components as well as the steps of the TeleCARE intervention is provided elsewhere (Pengchit et al., 2011).

Pilot testing

We conducted pilot testing prior to implementing the Family CARE study. Pilot testing took place at the Huntsman Cancer Institute in January and February 2009, with 35 CRC patients and their adult family members. Piloted intervention materials included all recruitment letters and brochures, the educational pamphlet intended for the PRINT arm, and the baseline questionnaire and cover letter. We revised study materials according to pilot participant suggestions.

Study contact

The PRINT group (ⁿ = 249) received only the mailed brochure. The 218 TeleCARE sessions lasted for an average of 39 min (SD = 12.2). We examined the primary outcome of the study (colonoscopy uptake by 9 months) in the 9-month follow-up questionnaire. We also examined intermediate study endpoints, including participant experiences and knowledge, sooner than at the 9-month follow-up. Thus, our study protocol included contacting participants from both arms at baseline, 1 month, 9 months, and 15 months post-intervention.

Protocol monitoring

The TeleCARE sessions were audiotaped to assure fidelity and provide feedback to counselors. Prior to the intervention, counselors obtained permission from participants to record the session. During the session, each genetic counselor used a protocol checklist (referred to hereafter as the Family CARE checklist), to assure that all steps of the intervention were completed. Two study staff members randomly selected intervention audiotapes for review and individually completed the Family CARE checklist while they listened to the audio recordings. After reviewing each tape, the two staff members compared checklists and outcomes, and they discussed protocol deviations in weekly review meetings with the counselors.

Plan for implementation setbacks

We kept training minutes, materials, and the standardized counseling protocol and pedigrees, in the event that new counselors needed to be trained.

Treatment Fidelity Adherence Step 2: Training Interventionists

Selection of interventionists

The study utilized genetic counselors, rather than other health care providers, to deliver the intervention. Genetic counselors have specific expertise in explaining genetic information to lay populations, and as part of this study received theoretical training in behavioral change mechanisms, and extensive experience tailoring cancer screening messages based on known risk and psychosocial factors.

Standardized training

Prior to the intervention start date, each counselor received the Family CARE intervention manual, which outlined specific aspects of EPPM and MI and included a step-by-step overview of each component of the telephone intervention.

In addition, all counselors received 16 hr of in-person MI training from an experienced MI trainer (S.W.), who trained them as a group to ensure that they all received the same information. All counselors were provided with a manual that explained the theory and mechanisms of MI, as well as examples of MI practice. MI training incorporated lecture, video, demonstration, and practice, to effectively accommodate all learning styles.

Practice and feedback

The counselors practiced the intervention on four analogue participants before conducting an intervention on actual study participants. Practice participants included other genetic counselors and individuals who agreed to provide feedback. The MI trainer reviewed the practice sessions, using the evaluation procedures provided below, to assure counselor proficiency prior to delivering the actual intervention.

Intervention fidelity

All counselors participated in review sessions, which assessed compliance with both the theoretical models and the study protocol. During the review sessions, the MI trainer reviewed a taped TeleCARE session to assess for MI skill proficiency and address potential problems. Each session was coded prior to the review, using the Family CARE checklist, to assess for adherence to the theoretical tenets and key clinical aspects of the intervention. After the MI section of the review, adherence to these items was discussed and any potential problems concerning adherence or the protocol were addressed.

To assure that counselors were proficient in performing the study and to address any problems that arose in the initial intervention implementation, review sessions took place weekly for the first 3 months of the intervention. Subsequently, review sessions were held biweekly for approximately 6 months. Toward the end of the intervention, when counselor proficiency was high, biweekly meetings were conducted to assess adherence to the study protocol, and monthly meetings were conducted to assess MI performance.

Enhance interventionist involvement

Three of the five counselors participated in the study design. This helped ensure that the clinician interventionists both understood and endorsed the purpose and methods of the study prior to implementation.

Treatment Fidelity Adherence Step 3: Treatment Delivery

Create collaborative relationships with interventionists

As noted previously, genetic counselors played an important role in study design and implementation. The weekly review sessions provided the opportunity for them to discuss challenging situations. These discussions assured early resolution of specific concerns or problems with the intervention.

Use a scripted curriculum or treatment manual

As Borrelli (2011) noted, the use of scripted treatment manuals for intervention delivery is controversial. A scripted manual helps reduce differences between interventionists, but may also constrain a tailored intervention. Furthermore, the use of a treatment script may be ill suited to MI, which relies heavily on the interventionist/patient discourse to accomplish behavioral change (Miller & Rollnick, 2009). As a scripted intervention did not match the tailoring and MI-style aims of our study, we opted instead for the Family CARE manual and a checklist as guidelines. The checklist assured that interventionists covered all required theoretical areas, but were allowed some flexibility in delivery.

Assess nonspecific effects on an ongoing basis

At the 1-month follow-up survey, participants were asked whether they remembered the counseling session and/or printed materials. They also reported on whether they found the information useful, interesting, valuable, understandable, true, personalized, and acceptable. Finally, participants were asked about their emotional reactions to the intervention. The response format for these items was on a 7-point Likert-type scale where 1 = strongly disagree and 7 = strongly agree.

Minimize differences within treatments and maximize differences between treatments

Independent coders assessed counselor adherence to the Family CARE checklist and provided regularly scheduled feedback sessions.

Ensure adherence to the protocol

We used the Family CARE checklist and two measures of MI—the Motivational Interviewing Treatment Integrity assessment (MITI; Moyers, Martin, Manuel, Hendrickson, & Miller, 2005) and the Behavior Change Counseling Index (BECCI; Lane, 2002)—to evaluate adherence to the theoretical protocol. Two researchers (W.B. and R.S.) were trained to use the MITI, a behavioral coding system designed to be used as a treatment integrity measure for MI intervention trials. The MITI evaluates global characteristics of the interview (e.g., evocation, collaboration, autonomy-supportiveness), as well as specific counselor behaviors (e.g., giving information, asking questions, reflecting, confronting) that are consistent/inconsistent with MI. The advantage of the MITI over simple clinical impression is that the MITI provides objective feedback and specific benchmarks on key aspects of MI. Several intervention trials have used a similar system for quality control purposes (Bennett, Roberts, Vaughan, Gibbins, & Rouse, 2007; Carroll et al., 2006; Feldstein & Forcehimes, 2007; Moyers, Martin, Manuel, Miller, & Ernst, 2010; Pierson et al., 2007; Thyrian et al., 2007). Studies tend to find a substantial agreement between patient and observer ratings of MI skill, as well as prediction of improved participant outcomes based on certain aspects of MITI-related performance. Coders also received training on the BECCI, a short measure of brief MI meant to be used in clinical health care settings. The BECCI consists of 11 items which are scored in a Likert-type style and which provide a total score ranging from 0 to 44 (Lane, 2002). We used both the MITI and the BECCI to assess whether the two coding methods provided similar information about counselor proficiency.

We assessed inter-rater reliability between coders for both the Family CARE checklist and the BECCI using Shrout–Fleiss calculations of inter-class correlation coefficients (ICCs). This calculation is most appropriate when the same raters are assumed to comprise the entire rater population, and thus rate all subjects (Shrout & Fleiss, 1979). For the BECCI, we established internal consistency reliability estimates by calculating Cronbach’s alpha. Variables used to assess proficiency in the MITI included: MI Spirit score (average of Evocation, Collaboration, and Autonomy/Support), ratio of complex to simple reflections, ratio of open to closed questions, and ratio of reflections to questions. Our assessment of inter-rater agreement for these MITI items used guidelines create categories of agreement, including (a) not reaching proficiency, (b) beginning to reach proficiency, and (c) meeting competency.

Establish minimum competency levels, below which interventionists are given remedial training

If coders noticed consistent problems with an area on the Family CARE checklist, they brought up the item for discussion during intervention review meetings. Reasons for the inclusion of the item (or use of the item in order) were reviewed and strategies for improving adherence were discussed. In general, protocol deviations were rare.

To establish MI fidelity, we used competency thresholds as presented in the MITI manual (Moyers et al., 2010). Counselors were expected to meet minimum competency ratings on the MITI global indicators and percent MI adherent statements. This ensured that the overall tone of the intervention was consistent with MI. Because of the structured nature of the intervention and the inclusion of a substantial amount of history-taking questions, counselors did not always meet the threshold for behavior counts. In these cases, the MI trainer determined counselor proficiency.

Treatment Fidelity Adherence Step 4: Treatment Receipt

Ask participants whether they received the intervention materials and telephone counseling for those randomized to TeleCARE

Participants were asked whether they remembered receiving the counseling session and/or printed materials (yes vs. no). They were also asked whether they read the printed materials they received (yes vs. some/no).

Assess pre-and post-test client knowledge

To determine the extent to which participants were familiar with information about CRC, colonoscopy, and screening recommendations, we examined six knowledge-assessment questions in both baseline and follow-up questionnaires. The genetic counselors specifically covered all six questions during the TeleCARE intervention. These items included the following: (a) information regarding the severity of CRC in the United States, (b) how family history increases risk of CRC, (c) information about the effectiveness of early detection of CRC in saving lives (two questions), (d) information about the effectiveness of colonoscopy, and (e) information about different screening recommendations based on family history. Cronbach’s alpha was not calculated for knowledge, which is generally reported as an index score rather than as a scale.

Administer pre-and post-tests of participant decisional uncertainty

One aim of MI is to reduce participant ambivalence about change. Congruent with this aim, we were interested in determining whether the intervention reduced participant decisional uncertainty regarding colonoscopy screening. Decisional uncertainty measures personal perceptions of (a) uncertainty in choosing options; (b) modifiable factors contributing to uncertainty, such as feeling uninformed, unclear about personal values, or unsupported in decision-making; and (c) effective decision-making such as feeling that the choice is informed, values-based, likely to be implemented, and expressing satisfaction with personal choice (O’Connor). We assessed this measure at the time of both baseline and follow-up questionnaires. Cronbach’s alpha was .77.

Administer pre-and post-tests of the Risk Behavior Diagnosis (RBD) Scale

The RBD scale measures the components of the EPPM model (i.e., perceived susceptibility to threat, perceived severity of threat, perceived self-efficacy, and perceived response efficacy) to determine whether an individual is ready for change (i.e., high threat and high efficacy; Witte, Cameron, McKeon, & Berkowitz, 1996). We were interested in determining whether the intervention increased both threat and efficacy perceptions. Each subscale was comprised of four items. Cronbach’s alphas ranged from .83 to .90.

Ensure that materials have appropriate health literacy and cultural relevance

To assure appropriate health literacy in our written materials, we submitted all brochures, pamphlets, and intervention documents to the Huntsman Cancer Institute’s Cancer Learning Center for readability testing using the Flesch–Kincaid scale (Paasche-Orlow, Taylor, & Brancati, 2003). The readability for all documents was set at a seventh-grade level overall, with sections of each document set at no higher than an eighth-grade level. We presented numerical information, so that it would be easy for study participants to comprehend. Our study also attempted to address cultural relevance by incorporating photos of rural individuals (as our study area had large rural populations), and those with different ages, ethnicities, and racial backgrounds. Use of MI as a counseling technique helped assure respectful consideration of cultural attitudes and beliefs.

Use multiple formats for information delivery (verbal, pictures, written)

The PRINT group received an illustrated brochure that outlined the increased risk for family members of CRC patients, provided age recommendations for colonoscopy, gave information on how to obtain a colonoscopy, and provided prevention guidelines. The TeleCARE group received a packet of printed visual aids that were used in tandem with the telephonic counseling session. These materials included the following: (a) a picture of the participant’s family tree (diagramming the history of CRC), (b) a graph showing the difference between an average person’s risk and the risk of someone with a similar family history of CRC, (c) a table outlining the stages of CRC diagnoses and their survival rates, and (d) a step-by-step list of how to obtain a colonoscopy. During the counseling session, the genetic counselor referred to each of these items and explained them in the context of the intervention.

Assure that clients understood the material covered in intervention

After reviewing each printed document in the intervention packet, genetic counselors ascertained whether the participant understood the material covered. To ensure client participation and clarity, the counselors used MI techniques, which allowed them to integrate open questions such as “What questions do you have about this information?” into each session. Asking whether the participant had any remaining questions was a final step on the Family CARE checklist.

Plan for action

The fourth and final document of the intervention packet requested that participants identify what date they would begin the first step of obtaining a colonoscopy. In accordance with the Family CARE checklist, counselors helped participants to brainstorm about the first step(s) to obtain a colonoscopy and (if appropriate) to set a specific date for that first step (e.g., “within the next month” or “after my daughter’s wedding in July”).

Assess participant confidence in applying the intervention

During the telephone session, counselors asked participants to rate their confidence about being able to obtain a colonoscopy on a scale of 1 to 10 (1 being no confidence, 10 being absolute certainty). After the participant provided a rating, counselors elicited the participant’s reasons for the selected confidence score (e.g., “Ok, so you rated it as a 2 out of 10. What are some reasons that you rated it as a 2 rather than a 1?”). This concept aligned with the MI principle of promoting change talk.

Contact for follow-up

Immediately after the telephone session, participants were mailed a tailored letter that included a summary of the discussion and reiterated the participant’s risk of CRC, benefits of colonoscopy, and personalized action plan, including the planned start date. This letter also provided the study’s toll-free number, in the event that the participant needed more information on how to enact the steps to obtaining a colonoscopy.

Statistical Analysis

All analyses were conducted using SAS 9.2 (SAS, Cary, North Carolina). To adjust for the randomized cluster design of the study, random family effect was included in each analytic model. Two-sided p values ≤.05 were considered significant.

Change in knowledge scores

We were interested in whether participant knowledge improved in the intervention arm from baseline to the 1-month follow-up questionnaire. We examined the effect of the intervention on knowledge using ANCOVA, adjusting for the participant’s knowledge at baseline.

Change in decisional uncertainty

We were interested in whether levels of decisional uncertainty were reduced from the baseline to the 1-month follow-up questionnaire, and whether there was a greater improvement in the TeleCARE group, compared with the low-intensity PRINT group. We estimated the effect of the intervention on decisional uncertainty with the same methods as we used to assess change in knowledge.

Change in RBD scale

We examined change in the four RBD measures from baseline to 1-month follow-up between the two intervention conditions. We conducted separate ANCOVAs for each RBD measure to model the change in subscale scores.

Results

As shown in Table 1, the two intervention groups did not differ by baseline sociodemographic or clinical characteristics.

Table 1.

Comparison of Baseline Characteristics, by Intervention Arm, in the Family CARE Study.

Characteristic	Overall n = 481	TeleCARE n = 232 (48.2%)	Comparison group n = 249 (51.7%)	p value^a
M age in years (SD)	50.3 (9.0)	49.9 (9.0)	50.8 (9.0)	.24
Sex				.15
Female	276 (57.4)	91 (39.2)	114 (45.8)
Male	205 (42.6)	141 (60.8)	135 (54.2)
Race/ethnicity				.12
Non-Latino White	454 (94.4)	215 (92.2)	239 (96.0)
Latino	17 (3.5)	9 (3.9)	8 (3.2)
Other/unreported	10 (2.1)	8 (3.5)	2 (0.8)
Marital status				.77
Single	21 (4.4)	10 (4.3)	11 (4.4)
Married	363 (75.5)	172 (74.1)	191 (76.7)
Separated, widowed, divorced	97 (20.2)	50 (21.6)	47 (18.9)
Educational level				.44
High school or less	93 (19.3)	51 (22.0)	42 (16.9)
Post-high school	206 (42.8)	100 (43.1)	106 (42.6)
Bachelor’s	114 (23.7)	50 (21.6)	64 (25.7)
Post-graduate	68 (14.1)	31 (13.4)	37 (14.9)
Rural/urban residence^b				.16
Urban	372 (77.3)	173 (74.6)	199 (79.9)
Rural	109 (22.7)	59 (25.4)	50 (20.1)
household income (yearly)				.48
<US$15,000	39 (8.1)	24 (10.3)	15 (6.0)
US$15-US$29,999	49 (10.2)	22 (9.5)	27 (10.8)
US$30-US$49,999	91 (18.9)	42 (18.1)	49 (19.7)
US$50-US$69,999	73 (15.2)	38 (16.4)	35 (14.1)
≥US$70,000	180 (37.4)	81 (34.9)	99 (39.8)
Missing, refused	49 (10.2)	25 (10.8)	24 (9.6)
Employment status^c				.55
Employed	338 (70.3)	166 (71.6)	172 (69.1)
Non employed	143 (19.7)	66 (28.5)	77 (30.9)
Health insurance^d				.27
Private	339 (70.5)	164 (70.7)	175 (70.3)
Public	54 (11.2)	31 (13.4)	23 (9.2)
No coverage	87 (18.1)	37 (16.0)	50 (20.1)
Refused	1 (0.2)	0 (0.0)	1 (0.4)
Has a personal health care provider				.98
Yes	313 (65.1)	152 (65.5)	161 (64.7)
No	143 (29.7)	68 (29.3)	75 (30.1)
Missing	25 (5.2)	12 (5.2)	13 (5.2)
Number of FDR and SDR relatives with CRC				.36
1 FDR, 0 SDR	390 (81.1)	202 (81.1)	88 (81.0)
≥2 FDR, 0 SDR	31 (6.4)	18 (7.2)	13 (5.6)
1 FDR, 1 SDR	52 (10.8)	22 (8.8)	30 (12.9)
1 FDR, ≥2 SDR	6 (1.3)	6 (2.4)	0 (0.0)

Note. FDR = first-degree relative; SDR = second-degree relative; CRC = colorectal cancer.

Test used: Logistic mixed model to account for familial correlation. Testing is for the association between categories of first-and second-degree relatives with the likelihood of being in the counseling group. Categories with three or more relatives were grouped together due to low counts.

Rural/urban residence was based on Rural–Urban Computing Area (RUCA) codes at the zip code level. RUCA codes were developed by the University of Washington Rural Health Research Center and the United States Department of Agriculture (USDA) Economic Research Service (ERS), with the support of the federal Health Resource and Service Administration’s Office of Rural Health Policy and the ERS, using standard Census Bureau urbanized area and urban cluster definitions in combination with work commuting data to characterize census tracts and zip codes. The 10 RUCA categories were aggregated into urban (1-3) and rural (4-10) as recommended by the Washington, Wyoming, Alaska, Montana, and Idaho (WWAMI) Rural Health Research Center. Logistic mixed model to account for familial correlation. Testing is for the association between categories of first-and second-degree relatives with the likelihood of being in the counseling group. Categories with three or more relatives were grouped together due to low counts.

Employment status was dichotomized as employed (employed for wages, self-employed) and not employed (out of work <1 year, out of >1 year, retired, homemaker, student, unable to work or other).

Private was defined as primary health care coverage that is provided through a current or former employer or union, health insurance plan that you purchase directly from an insurance company, health plan of someone who does not live in the household, and coverage through another source not mentioned. Public was defined as Medicare, Medicaid, or another government plan.

Treatment delivery

Family CARE checklist

A total of 103 (47%) of the TeleCARE sessions were assessed for fidelity according to the Family CARE checklist. Of the sessions, the majority (98.7%) were fully compliant with the checklist. Of the sessions that did not meet full compliance (n = 27), most (96%) were at >90% compliance.

Thirty Family CARE checklists were double coded to assess for inter-rater reliability. The ICC was .93, indicating excellent inter-rater reliability. The two coders demonstrated perfect agreement on 18 of the 30 coded sessions (60%). The largest discrepancy between coders for one session was 5 points.

MI

Two coders (R.S. and W.B.) reviewed 24 of the same tapes for the five counselors. Table 2 provides an overview of MI outcomes according to categories created by. The two coders agreed that counselors demonstrated proficiency in MI Spirit, but modest proficiency in the other three areas of MI. The structured nature of the MI sessions may account for these areas of modest proficiency ratings.

Table 2.

Impressions of Interventions by Study Groups for Participants Who Responded to the Full Version of the 1-Month Post-Intervention Survey.

Communication item	TeleCARE n = 156 (45.1%)		Comparison group n = 190 (54.9%)
Communication item	n (%)		n (%)
Remember receiving print materials/educational pamphlet?
Yes	153 (98.1)		171 (90.0)
No	3 (1.9)		16 (8.4)
Missing/refused	0 (0.0)		3 (1.6)
Did you read print materials/educational pamphlet?
Yes	121 (77.6)		145 (76.3)
Some	30 (19.2)		NA
No/unanswered	5 (3.2)		45 (23.7)
Information evaluation (based on 1-7 Likert-type scale)	Telephone session^a (n = 153)	Print materials^b (tailored and pamphlet; n = 149)	Educational pamphlet^a (n = 144)
	M (SD)	M (SD)	M (SD)
Useful	5.37 (1.65)	5.44 (1.43)	5.09 (1.45)
Interesting	5.39 (1.58)	5.38 (1.39)	5.14 (1.28)
Valuable	5.61 (1.50)	5.50 (1.46)	5.28 (1.33)
Understandable	6.34 (1.11)	6.07 (0.98)	5.70 (1.21)
Valid	6.13 (1.26)	5.94 (1.20)	5.60 (1.30)
Personalized	5.59 (1.62)	5.52 (1.50)	4.49 (1.52)
Easier to think about getting screened	5.31 (1.85)	5.26 (1.59)	4.74 (1.39)
Addressed reasons for not getting screened	5.10 (1.89)	4.92 (1.77)	4.15 (1.69)
Total evaluation score	5.60 (1.26)	5.51 (1.16)	5.02 (1.05)

Subpopulation of those who recalled their telephone session and completed the information evaluation.

Subpopulation of those who read all or some of the printed materials and completed the information evaluation.

BECCI

Two coders scored the BECCI for 30 telephone counseling sessions. The average BECCI score was 40.5 (SD = 1.81, range = 36-44), indicating high proficiency in a clinical assessment of MI. The ICC value for the BECCI was .47, which indicates fair inter-rater reliability. It should be noted that the highly structured nature of the intervention truncated the variation in BECCI scores, resulting in a lowered ICC; this was not necessarily due to low inter-rater agreement. To address this concern, we used additional indicators of the rater’s level of agreement to determine whether the reviewers came to the same clinical conclusion about counselor MI competence. When considering wider ranges of BECCI scores as indications of MI competence, the two reviewers agreed that all counselors demonstrated excellent MI skills (BECCI scores between 40 and 44) on 15 of 30 sessions (50%), and good MI skills (BECCI scores > 38) 83% (25/30) of the time.

Treatment receipt

Table 3 shows descriptive statistics and ANCOVA results for all measures of treatment receipt, including changes in knowledge scores, RBD subscale scores, and decisional uncertainty scale scores from baseline to the 1-month post-intervention assessment.

Table 3.

Changes in Treatment Receipt Variables, Baseline to 1-Month Follow-Up by Intervention Group, Family CARE Study.

Variable	Intervention group	Descriptive statistics on univariate change			Model results^a
Variable	Intervention group	n	M change	SD change	n	Effect estimate	f	p value	ICC
Perceived severity	PRINT	220	−0.55	2.41	408	0.031	0.35	.5576	.393
Perceived severity	TeleCARE	188	−0.32	3.06
Perceived susceptibility	PRINT	209	0.01	1.98	391	0.169	15.08	.0002	.221
Perceived susceptibility	TeleCARE	182	0.79	2.55
Response efficacy	PRINT	213	0.85	3.67	394	0.172	7.46	.0076	0
Response efficacy	TeleCARE	181	1.67	4.02
Self-efficacy	PRINT	213	0.56	2.82	400	0.167	8.16	.0053	.133
Self-efficacy	TeleCARE	187	1.27	3.59
Decision uncertainty	PRINT	195	−13.08	28.47	351	−1.379	19.59	<.0001	.044
Decision uncertainty	TeleCARE	156	−20.43	28.44
Knowledge	PRINT	189	0.22	1.33	344	0.086	17.46	<.0001	.067
Knowledge	TeleCARE	155	0.45	1.07

Note. ICC = inter-class correlation coefficients.

Model results from SAS proc mixed represent ANCOVA model of change, adjusting for baseline. Models include a random family effect to account for the cluster design of the study.

On average, knowledge scores increased in both intervention groups; however, the TeleCARE group had a significantly greater increase in knowledge. The vast majority of participants perceived CRC as serious at baseline, and perceptions about severity were unchanged in 67% of respondents (f = 0.35, p = .56). Perceived susceptibility was virtually unchanged in the PRINT group, but the TeleCARE group had significantly higher scores following the intervention (f = 15.08, p = .0002). Response efficacy scores increased for both intervention groups, but the TeleCARE group had a significantly greater increase (f = 7.46, p = .0076). Self-efficacy scores also increased in both groups, but again, the TeleCARE group demonstrated a significantly greater increase compared with the PRINT group (f = 8.16, p = .0053). Decisional uncertainty change scores decreased in both intervention groups; however, the TeleCARE group showed a significantly greater decrease in uncertainty regarding the decision to have a colonoscopy (f = 19.59, p < .0001).

Discussion

In general, steps to assure treatment fidelity in the Family CARE trial remained consistent with the recommendations for best practices (Bellg et al., 2004; Borrelli, 2011; Borrelli et al., 2005). In our study, best-practice recommendations were adhered to with regard to study design and training of interventionists. Assessment of treatment delivery indicated that counselors were adherent to the TeleCARE intervention’s theoretical tenets.

Assessment of counselor proficiency in MI demonstrated a more complex picture of the intervention delivery. Coders ratings suggested that the genetic counselors demonstrated high levels of MI Spirit (22/24 tapes indicated that counselors met competency), yet they also suggested that specific measures of MI utterances (open questions, complex reflections, reflection-to-question ratios) demonstrated modest proficiency. This seeming difference in proficiency may be due to the fact that MI was incorporated into the intervention more as a style of delivery and less as a specific counseling tool. Traditional MI encourages clinicians to ask questions that can lead to deeper discussion and reflection on the issue at hand; whereas in the current intervention, genetic counselors were constrained by the requirement to address the theoretical tenets of the intervention, which required them to provide a psycho-educational approach. This may have inhibited proficiency counts based on utterance scores. Previous studies (Miller & Rollnick, 2009) have demonstrated that for studies that incorporate MI into structured, multi-theoretical models, the use of traditional utterance counts to examine counselor proficiency may not represent the best method of assessment. For the more global MI assessment scores (MI Spirit and the BECCI), coders agreed that counselors were proficient in MI. These findings indicate that future studies which incorporate MI into intervention structures may find that global ratings of MI are better indicators of counselor proficiency; or, conversely, that a more detailed assessment such as the Motivational Interviewing Skill Code, which assesses interventionist/ participant interactions rather than interventionist utterances, might provide better information about the level of MI proficiency in such studies (Madson & Campbell, 2006; Resnick et al., 2005).

Recommendations for assessing treatment fidelity include administering pre-and post-tests of knowledge as part of treatment receipt assessment (Borrelli, 2011). However, as prior research has demonstrated that increases in knowledge alone do not appreciably affect CRC screening behavior (Manne et al., 2002), we included additional measures of treatment receipt that aligned with our study’s aims and theoretical rationale. All of these measures (e.g., decisional conflict, RBD subscale scores), with the exception of perceived severity, were found to significantly improve at the 1-month follow-up. These findings indicate that our TeleCARE intervention was successful in addressing the theoretical targets. Future plans include assessing whether these theoretical targets mediated the effect of the intervention on colonoscopy uptake. Other cancer screening studies have also shown little variation in perceived severity. This is likely due to a ceiling effect, wherein individuals already perceive cancer as a serious disease and, consequently, scores do not change much over time.

Our study has several limitations. The low number of participants representing ethnic and racial minorities limits the generalizability of our study’s findings. We did not include the treatment fidelity area of “treatment enactment” as suggested in the recommendations for best practices for treatment fidelity (Bellg et al., 2004; Borrelli, 2011). Treatment enactment is a method of assessing whether the skills a participant acquires from an intervention are appropriately implemented, so as to assure intended study outcomes (Borrelli, 2011). The difference between enactment and outcomes is the measurement of enactment throughout the course of the study, instead of at the end of the follow-up period. In certain study designs, such as behavioral intervention trials designed to motivate cancer screening or single-event decision behaviors, it may not be practical to assess for implementation of self-regulatory skills, as these skills may not be measurable. In our study, the primary outcome was colonoscopy uptake by 9 months after the intervention; skills needed to carry out this task are difficult to measure separately from the event itself. For example, steps to obtain a colonoscopy include the following: (a) contacting a primary care physician for a colonoscopy referral, (b) making an appointment with the colonoscopy center, (c) obtaining a prescription for and ingesting the preparation the night before, and (d) finding someone to take the participant to and from his or her appointment. Most of these steps are procedural and do not require a particular skill set. In our case, enactment essentially overlaps with outcome. Therefore, this measure of fidelity may be design-dependent and not always measurable. Future studies that assess treatment fidelity may need to determine whether treatment enactment is possible prior to study implementation. Studies that examine adherence to treatment fidelity may also be advised to take study design into account when assessing inclusion of treatment enactment information.

Conclusion

Reporting treatment fidelity is an important yet widely underutilized process that both assures the legitimacy of study outcomes and creates informed precedence for future research. The Family CARE study adhered to most of the best-practice guidelines outlined previously, demonstrating that our intervention was delivered as intended and positively affected the cognitive processes that are thought to predict desired behavioral outcomes. Given the complex nature of behavioral interventions, we recommend the potential use of multiple treatment receipt assessments, to assure that cognitive intervention targets are assessed. Finally, we recommend that future treatment fidelity assessments consider the appropriateness of including treatment enactment as a required component of the treatment fidelity checklist, as it may not substantially differ from study outcomes in research areas, such as cancer

Footnotes

Acknowledgements

We would like to thank Sandie and Roger Edwards, Rosemary Cress, Christopher Johnson, Deidre Hill, Jan Lowrey, Charles Wiggins, Marc Williams, and the genetic counselors for their assistance in the Family CARE study, and Susan Schulman for assistance with manuscript preparation.

Authors’ Note

This content is solely the responsibility of the authors and does not necessarily reflect the opinions or views of the funding and supporting agencies.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: Family CARE was funded by the National Cancer Institute (1R01CA125194-03; Kinney, PI) and the Huntsman Cancer Foundation. Family CARE was also supported by the Shared Resources (P30 CA042014) at Huntsman Cancer Institute (biostatisticians, genetic counselors, research informatics, Tissue Resource and Applications Core (Huntsman Cancer Institute), and the Utah Population Database [UPDB]); the Utah Cancer Registry, which is funded by Contract No. HHSN261201000026C from the National Cancer Institute’s SEER Program with additional support from the Utah State Department of Health and the University of Utah; the California Department of Public Health as part of the statewide cancer reporting program mandated by California Health and Safety Code Section 103885; the National Cancer Institute’s Surveillance, Epidemiology and End Results Program under Contract N01PC-2010-00034C awarded to the Northern California Cancer Center, Contract N01-PC-35139 awarded to the University of Southern California, and Contract N01-PC-54404 awarded to the Public Health Institute, and the Centers for Disease Control and Prevention’s National Program of Cancer Registries, under agreement U58CCU000807-05 awarded to the Public Health Institute; the Colorado Central Cancer Registry program in the Colorado Department of Public Health and Environment funded by the National Program of Cancer Registries of the Centers for Disease Control and Prevention; the Cancer Data Registry of Idaho supported in part by the National Program of Cancer Registries of the Centers for Disease Control and Prevention; the New Mexico Tumor Registry which is funded by National Cancer Institute Contract No. HHSN261201000033C; the Rocky Mountain Cancer Genetics Network (HHSN261200744000C); the Huntsman Cancer Registry; and the Intermountain Healthcare Oncology Clinical Program and Intermountain Clinical Genetics Institute.

Author Biographies

Rebecca Simmons is a Research Advisor for Population Services International. She is completing her doctoral dissertation on the role of physical activity in cancer screening and development. Her research interests include applied research in behavior change interventions aimed at non-communicable disease prevention, particularly in the developing world.

Scott Walters is Professor of Behavioral and Community Health at the University of North Texas Health Science Center. His research is focused on translating counseling strategies such as motivational interviewing to non-traditional settings and modalities. He also trains providers in motivational counseling techniques and has served as consultant to develop Internet and mobile interventions for a variety of health behaviors.

Lisa Pappas is an M.Stat.with Huntsman Cancer Institute, has twelve years of experience working as an M.S. statistician first in the Department of Internal Medicine and now in the Huntsman Cancer Institute. Ms. Pappas has extensive experience analyzing data sets from the SEER Data Base and other Medicare data bases and provides general statistical support for Huntsman Cancer Institute investigators.

Kenneth M. Boucher is a research associate professor at the University of Utah School of Medicine and director of the biostatistics core facility at Huntsman Cancer Institute. He has more than 18 years of statistical consulting experience in cancer, including clinical trials and projects in epidemiology and genomics.

Watcharaporn Boonyasiriwat is a social psychologist at the faculty of psychology, Chulalongkorn University, Bangkok Thailand. She joined the FCARE project as a research assistant when she was trained at the department of psychology, the University of Utah, US. After earning her PhD in 2010, Watcharaporn returned to Thailand and works on research related to attitudes and persuasion with special interest in health and intergroup relations.

Amanda Gammon is a board-certified genetic counselor with a master’s degree in genetic counseling from University of Colorado at Denver Health Sciences Center. She began working at Huntsman Cancer Institute in July 2007 where she provides genetic counseling to patients in the Family Cancer Assessment Clinic as well as in research settings. Her main research interests include hereditary breast cancer and provision of genetic counseling through alternative modes.

Sally W. Vernon, M.A., Ph.D., is Chair, Department of Health Promotion and Behavioral Sciences and Professor of Behavioral Sciences and Epidemiology at The University of Texas-Houston School of Public Health. Her research focuses on factors related to cancer screening adherence and informed decision making and on developing theory-based interventions to increase the frequency of screening behaviors.

Randall Burt is an MD clinical investigator (gastroenterologist) and practicing physician whose central research and clinical interest has been familial and hereditary colorectal cancers. He was involved with identifying the APC gene, the major disease gene involved in both inherited and common colorectal cancers. He has also done extensive work in clarifying the cancer risk in both the inherited colon cancer syndromes and common familial colon cancer.

Antoinette Stroup is the Director of the New Jersey State Cancer Registry and an Associate Professor of Epidemiology with Rutgers University. Her research interests include cancer surveillance research

References

Bellg

A. J.

Borrelli

Resnick

Hecht

Minicucci

D. S.

Ory

. . . Czajkowski

(2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology, 23, 443-451.

Bennett

G. A.

Roberts

H. A.

Vaughan

T. E.

Gibbins

J. A.

Rouse

(2007). Evaluating a method of assessing competence in motivational interviewing: A study using simulated patients in the United Kingdom. Addictive Behaviors, 32, 69-79.

Borrelli

(2011). The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials. Journal of Public Health Dentistry, 71, S52-S63.

Borrelli

Sepinwall

Ernst

Bellg

A. J.

Czajkowski

Breger

. . . Orwig

(2005). A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology, 73, 852-860.

Carroll

K. M.

Ball

S. A.

Nich

Martino

Frankforter

T. L.

Farentinos

. . .Woody

G. E.

(2006). Motivational interviewing to improve treatment engagement and outcome in individuals seeking treatment for substance abuse: A multisite effectiveness study. Drug Alcohol Dependence, 81, 301-312.

Cohen

D. J.

Crabtree

B. F.

Etz

R. S.

Balasubramanian

B. A.

Donahue

K. E.

Leviton

L. C.

. . .Green

L. W.

(2008). Fidelity versus flexibility: Translating evidence-based research into practice. American Journal of Preventive Medicine, 35(5, Suppl.), S381-S389.

Feldstein

S. W.

Forcehimes

A. A.

(2007). Motivational interviewing with underage college drinkers: A preliminary look at the role of empathy and alliance. American Journal of Drug and Alcohol Abuse, 33, 737-746.

Hawe

Shiell

Riley

(2008). In response to Spillane

Byrne

M. C.

Byrne

Leathem

C. S.

O’Malley

Cupples

M. E.

(2007) Monitoring treatment fidelity in a randomized trial of a complex intervention. Journal of Advanced Nursing 60(3), 343-352. Important considerations for standardizing complex interventions. Journal of Advanced Nursing, 62(2), 267.

Kinney

A. Y.

Boonyasiriwat

Walters

S. T.

Pappas

L. M.

Stroup

A. M.

Schwartz

M. D.

. . .Higginbotham

J. C.

(2014). Telehealth personalized cancer risk communication to motivate colonoscopy in relatives of patients with colorectal cancer: The Family CARE randomized control trial. Journal of Clinical Oncology, 32, 654-662.

10.

Lane

(2002). The Behaviour Change Counselling Index (BECCI): Manual for coding behaviour change counselling. Retrieved from medicine.iupui.edu/IUCAR/roybal/pdf/TCL.BehaviorChange.pdf

11.

Leventhal

Friedman

M. A.

(2004). Does establishing fidelity of treatment help in understanding treatment efficacy? Comment on Bellg et al. (2004). Health Psychology, 23(5), 452-6.

12.

Madson

M. B.

Campbell

T. C.

(2006). Measures of fidelity in motivational enhancement: A systematic review. Journal of Substance Abuse Treatment, 31, 67-73.

13.

Manne

Markowitz

Winawer

Meropol

N. J.

Haller

Rakowski

. . .Jandorf

(2002). Correlates of colorectal cancer screening compliance and stage of adoption among siblings of individuals with early onset colorectal cancer. Health Psychology, 21, 3-15.

14.

Miller

W. R.

Rollnick

(2009). Ten things that motivational interviewing is not. Behavioural and Cognitive Psychotherapy, 37, 129-140.

15.

Moyers

T. B.

Martin

Manuel

J. K.

Hendrickson

S. M.

Miller

W. R.

(2005). Assessing competence in the use of motivational interviewing. Journal of Substance Abuse Treatment, 28, 19-26.

16.

Moyers

T. B.

Martin

Manuel

J. K.

Miller

W. R.

Ernst

(2010). Revised global scales: Motivational interviewing treatment integrity 3.1.1. Retrieved from http://casaa.unm.edu/download/miti3_1.pdf

17.

O’Connor

A. M.

(1993). User manual—Decisional Conflict Scale. Retrieved from http://decisionaid.ohri.ca/docs/develop/User_Manuals/UM_Decisional_Conflict.pdf

18.

Paasche-Orlow

M. K.

Taylor

H. A.

Brancati

F. L.

(2003). Readability standards for informed-consent forms as compared with actual readability. New England Journal of Medicine, 348, 721-726.

19.

Pengchit

Walters

S. T.

Simmons

R. G.

Kohlmann

Burt

R. W.

Schwartz

M. D.

Kinney

A. Y.

(2011). Motivation-based intervention to promote colonoscopy screening: An integration of a fear management model and motivational interviewing. Journal of Health Psychology, 16, 1187-1197.

20.

Pierson

H. M.

Hayes

S. C.

Gifford

E. V.

Roget

Padilla

Bissett

. . .Fisher

(2007). An examination of the motivational interviewing treatment integrity code. Journal of Substance Abuse Treatment, 32, 11-17.

21.

Resnick

Bellg

A. J.

Borrelli

Defrancesco

Breger

Hecht

. . .Czajkowski

(2005). Examples of implementation and evaluation of treatment fidelity in the BCC studies: Where we are and where we need to go. Annals of Behavioral Medicine, 29, 46-54.

22.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.

23.

Simmons

R. G.

Lee

Y. C.

Stroup

A. M.

Edwards

S. L.

Rogers

Johnson

. . .Kinney

A. Y.

(2013). Examining the challenges of family recruitment to behavioral intervention trials: Factors associated with participation and enrollment in a multi-state colonoscopy intervention trial. Trials, 14, Article 116.

24.

Spillane

Byrne

M. C.

Byrne

M. L.

Leathem

C. S.

O’Malley

Cupples

M. E.

(2007). Monitoring treatment fidelity in a randomized controlled trial of a complex intervention. Journal of Advanced Nursing, 60, 343-352.

25.

Tan

Y. Y.

McGaughran

Ferguson

Walsh

M. D.

Buchanan

D. D.

Young

J. P.

, . . . ANECS Group. (2013). Improving identification of lynch syndrome patients: A comparison of research data with clinical records. International Journal of Cancer, 132, 2876-2883.

26.

Thyrian

J. R.

Freyer-Adam

Hannover

Röske

Mentzel

Kufeld

. . .Hapke

(2007). Adherence to the principles of motivational interviewing, clients’ characteristics and behavior outcome in a smoking cessation and relapse prevention trial in women postpartum. Addictive Behaviors, 32, 2297-2303.

27.

Witte

(1992). Putting the fear back into fear appeals: The extended parallel process model. Communication Monographs, 59, 329-349.

28.

Witte

Allen

(2000). A meta-analysis of fear appeals: Implications for effective public health campaigns. Health Education & Behavior, 27, 591-615.

29.

Witte

Cameron

K. A.

McKeon

J. K.

Berkowitz

J. M.

(1996). Predicting risk behaviors: Development and validation of a diagnostic scale. Journal of Health Communication, 1, 317-341.