Abstract
Background:
Fentanyl, a highly potent synthetic opioid, is now involved in the majority of drug overdose deaths in the United States. Discrepancies between substance use self-reports and urine drug screens (UDS) are common, especially when users are unaware of fentanyl adulteration.
Objectives:
The objective of this study was to examine the cognitive processes underlying discrepancies between self-reported fentanyl use and UDS results, specifically focusing on participants unaware of their fentanyl exposure.
Design:
This was a cross-sectional qualitative study.
Methods:
This study conducted cognitive interviews with 10 participants from a larger clinical trial who tested positive for fentanyl in a UDS but reported no fentanyl use. The interviews explored participants’ understanding of survey questions about fentanyl use and their interpretation of UDS results. Responses were analyzed to identify themes related to self-report discrepancies.
Results:
Participants often underreported fentanyl use, not due to social desirability bias, but because they interpreted survey questions as asking about known, rather than suspected, fentanyl use. Many acknowledged their drugs were likely adulterated but only reported use when they were certain of fentanyl’s presence.
Conclusion:
The findings suggest that refining survey questions to capture both intentional and unintentional fentanyl exposure could improve the accuracy of self-reported data. These adjustments are critical to improve the accuracy of data collected to inform surveillance strategies, effective public health policies, and clinical interventions.
Introduction
Fentanyl is a synthetic opioid 50 to 100 times more powerful than morphine and is typically used medically to treat severe pain.1,2 It is also illicitly manufactured and often added to other drugs because of its low cost.3 -5 Its continued saturation of the illicit drug market has contributed to an exponential growth in overdose, with fatalities reaching 107 941 in 2022. 6 This same year, 81.8% of drug overdose deaths involved at least 1 opioid, and illegally-made fentanyl was involved in approximately two-thirds of all overdose deaths. 7 Thus, it is imperative that accurate data on fentanyl use is available to inform surveillance strategies, public health policies, and clinical interventions targeted to individuals who are using fentanyl—whether intentionally or unknowingly.
To this end, researchers are interested in understanding the degree of knowledge possessed by people who use drugs (PWUD) regarding their fentanyl consumption and how this knowledge might impact use behaviors.5,8 While self-reports are often used to understand drug consumption, their utility is limited in situations where PWUD are unaware of, misinformed, or unwilling to share information about the drugs they have taken.9,10 For this reason, self-reported use is often confirmed with point-of-care urine drug screens (UDS).11 -14 However, researchers are often interested in use that occurred in the past 30 or more days and UDS can only detect drug use within a window of a few days. 15 Recognizing these limitations of self-report and UDS related to fentanyl use is important for ensuring the accuracy of data for informing public health surveillance, policy, and healthcare interventions.
The validity of self-reported drug use has been a topic of debate among addiction researchers for more than 50 years.16 -18 While self-reports have been demonstrated to be valid in many situations,19 -21 opioids have become an exception in recent years. Despite some research demonstrating a rise in self-reported fentanyl use, 22 other studies suggest underreporting. For instance, Moreno et al 23 examined 1683 self-report/UDS paired sets and, despite a generally high rate of agreement for other drugs, found a 41.3% mismatch between negative self-report and positive UDS for opioids. As the UDS did not specifically test for fentanyl, the researchers were unable to determine if this discrepancy was because participants were unaware of fentanyl adulteration. Similarly, Martinez et al 24 found every participant (n = 33) in their study was surprised by positive UDS results for fentanyl despite having knowledge of fentanyl adulteration in the drug supply. In the current clinical trial, members of our team found a 37% mismatch between negative self-reported fentanyl use and positive UDS for the first 248 study participants despite high agreement rates for all other drugs, including other opioids. 25 Subsequent confirmatory laboratory testing of 100 urine samples verified fentanyl’s presence. 26 As a next step, we conducted a series of cognitive interviews to gain insight into the origin of these self-report/UDS discrepancies.
Cognitive interviews are an established approach for evaluating the quality of surveys and structured interview items to determine if they elicit information from participants that have been intended by researchers.27 -29 Although techniques vary,28,30 all cognitive interviews rest on a model that describes the survey response process as involving 4 stages 31 : (1) comprehension, (2) information retrieval, (3) judgment/estimation, and (4) response selection. However, cognitive interview studies may focus on 1 or more of these stages depending on the specific goals. 32 Cognitive interviews can be conducted as part of the survey design or during/after data collection as a quality assurance mechanism. 29 Following this latter approach, we conducted a cognitive interview study to identify potential wording issues related to fentanyl self-report questions, with the goal of modifying wording and providing guidance to improve the quality of data collected in future epidemiological and clinical studies.
Methods
Setting
This study was conducted as part of Chestnut Health Systems’ Justice Community Opioid Intervention Network (JCOIN) Research Hub. JCOIN is supported by the National Institute on Drug Abuse through the National Institute of Health’s Helping to End Addiction Long-term Initiative. All 12 JCOIN-supported clinical trial hubs focus on interventions for improving medication for opioid use disorder (OUD) access (eg, methadone, suboxone, naltrexone) among people involved in the criminal legal system, 33 a population at higher overdose risk than the general community.34,35 The JCOIN Hubs that are collecting original client-level data use a set of core survey questions (https://pubmed.ncbi.nlm.nih.gov/33531212/) including self-report for past 90-day use of all major drug classes with specific questions focused on different opioid types, including heroin, methadone, buprenorphine, and fentanyl. The fentanyl question reads: “During the past 90 days, on how many days did you use fentanyl (alone or mixed with other drugs)?”
The Chestnut Hub’s trial focuses on Recovery Management Checkups, an OUD treatment linkage and continuing recovery support intervention.
36
Participants in the study are referred from jails and community providers in participating Illinois and Indiana counties and are screened for study eligibility. To be eligible for the trial, participants must: (a) be 18 years or older; (b) have been arrested within the past 90 days; and (c) either have been diagnosed with an OUD or regularly used opioids in the 90 days before their arrest. Exclusion criteria include (a) having a cognitive impairment that precludes ability to give informed consent and (b) residing outside of the defined service area. A UDS is administered following NIH-recommended protocol
37
at each in-person assessment (baseline and quarterly follow-ups). This protocol requires Chestnut Hub interviewers to conduct the UDS and deliver the results to participants before asking the self-report questions. If the UDS detects a specific drug that the participant stated they either had not or did not know if they used in the past 90 days, a prompt instructs the interviewer to probe further. The specific prompt for a fentanyl mismatch reads:
I need your help, I may have made a mistake or misunderstood. The urine test today indicates that there is some kind of fentanyl in your system, but I wrote down that you had no fentanyl use. Can you think of any reason why the urine test would be positive? Might it have been mixed with something else, you were exposed to it, or maybe you forgot?
The interviewer then asks about past 30-day use since it is closer to the UDS detection window: “During the past 30 days, on how many days did you use fentanyl (alone or mixed with other drugs)?”
As previously stated, an analysis of the first 248 baseline interviews suggested 37% discrepancy rate for self-reports based on UDS results. This trend was persistent across data collection waves and confirmed by laboratory testing. Given that discrepancy rates were approximately 10% or less for other assessed substances, we wanted to identify the root of this incongruity.
Cognitive interviews
We recruited a convenience sample of 10 participants between November 3rd and December 8th of 2023. The sample size was the result of how many interviews were able to be completed within the predetermined timeline necessary to provide useful and actionable feedback to the larger trial versus any pre-determined enrollment goal. To be eligible, enrolled JCOIN participants had to: (a) complete a quarterly follow-up interview for the larger trial while the cognitive interviewer was present at the study’s Chicago-based Research Hub; (b) have a UDS that was positive for fentanyl, and (c) have answered “no” or “I don’t know” to both the 90- and 30-day fentanyl use questions described above. After determining eligibility, a research staff member asked participants if they would complete the cognitive interview following the regular quarterly follow-up. All participants who were invited agreed and completed a single in-person interview at the research office.
Participants completed a semi-structured cognitive interview focusing on their interpretation of and responses to the fentanyl use questions asked in the quarterly trial interview they had just completed. The interviews were conducted solely by the first author who is male and has a doctoral degree in sociology and over 17 years of experience conducting qualitative behavioral health research. This researcher had not been introduced to any trial participants prior to conducting the cognitive interviews.
The interview captured all phases of Tourangeau’s 31 cognitive model (eg, comprehension, information retrieval, judgment/estimation, and response selection) described above. The interviewer began by reading the 90-day fentanyl use question that was asked in the quarterly follow-up, then asked the participant to repeat the question in their own words (comprehension) and describe how someone would know if they took drugs mixed with fentanyl (judgment estimation). At that point, the participant was reminded how they responded to the question during the interview, and asked: (1) why they answered as they did (response selection), (2) how sure they were of their answer (judgment estimation), (3) how difficult the 90-day use question was to answer (judgment estimation), and (4) how well they remembered their drug use over the past 90 days (information retrieval). The interview then shifted to asking if the participant recalled the results of their UDS (information retrieval), and, if not, the interviewer reminded them of the results. The last set of questions focused on the past 30-day fentanyl use question. The participant was again asked to identify the number of days of fentanyl use in the past 30 days and the interviewer probed to understand how they arrived at their answer (response selection). If the participant’s answer was, “0 days” or “don’t know,” the interviewer asked (a) why the participant provided that response given the fentanyl detected by the UDS and (b) why they believed fentanyl was detected. Throughout the process, the interviewer used flexible probing where the participant was asked to expand on their answers immediately after they were provided.29,30,32 To not interrupt participants, he also took brief notes as reminders to probe areas of the discussion during appropriate pauses. For instance, some participants who demonstrated a greater ability to reflect on the line of cognitive interview questioning were asked to provide their own rewording to improve the question or react to rewording suggestions made by the interviewer. Time constraints prevented pilot testing questions before recruitment. Participants received a $35 payment as compensation for their time spent during the cognitive interview. Interviews lasted an average of 20 minutes and were recorded and transcribed verbatim.
Analysis
The first author conducted the analysis using MAXQDA qualitative data analysis software. 38 He followed a pragmatic approach where the data were inductively coded to pertain to the specific questions asked during the interview. These codes were then summarized across interviews to identify common patterns in how participants interpreted and answered the questions.28,29,39 Results as they pertained to each question were then reviewed to develop final interpretations and rewording recommendations for future studies. Thematic saturation was established at the point when no new insights were gained from the available data. The other authors reviewed and provided feedback on the recommendations before they were finalized.
Results
Table 1 summarizes the characteristics of the cognitive interview participants. The sample was primarily male, African American, and non-Hispanic/Latino. The average age was 42, and the length of time using opioids averaged around 25 years. All participants but one identified as regular users of heroin. Half of the sample responded, “0 days” to both fentanyl use questions, and half responded, “I don’t know.”
Cognitive interview participant characteristics.
Represents answer to past 30- and 90-day fentanyl use questions since responses mirrored each other.
Does not include methadone or buprenorphine.
Initial interpretations of 90-day fentanyl use question
When asked to interpret the 90-day fentanyl use question about, most participants said they believed the it was seeking to understand whether they explicitly knew they had taken fentanyl: “You asked me if I had knowledge that I was using fentanyl with heroin in the last 90 days” (African American male, age 56). One participant directly indicated he answered “0 days” despite believing that he was likely using fentanyl every day:
I’m going to repeat the question. During the past 90 days, on how many days did you use fentanyl alone or mixed with other drugs?
Well, I didn’t know it was fentanyl, but they mix it with the heroin. So, every day I use [fentanyl]. . .
So [asking to repeat the question again], during the past 90 days, on how many days did you use fentanyl alone or mixed with other drugs?
How many days have I used fentanyl along with other drugs? Every day. (African American female, age 52)
In this and other cognitive interviews, participants shared the assumption their heroin supply was likely adulterated with fentanyl and clarified that, during clinical trial’s quarterly follow-up interviews, their responses had been based on whether they explicitly knew fentanyl had been present in the drugs they used.
These data demonstrated 2 lines of rationale for why participants lacked this explicit knowledge of fentanyl use required to answer the clinical trial questions as researchers intended. The first rationale was expressed by those who indicated they were not aware they were taking fentanyl because they were not seeking it at the time of use: “I answered none [0 days to the question]. [Because] my intent [at the time of use], [I] was not looking for specifically fentanyl. So, I didn’t have the intent of that [using fentanyl]” (White, Latina female, age 36). The second rationale was shared by those who indicated they had no external means of checking for adulteration because either the person who provided the drugs did not tell them about the fentanyl or they did not have fentanyl test strips for checking: “I don’t know what I was putting [in] my system when I went and bought it. They [people who sell drugs] don’t give you no note and tell you if it’s got fentanyl, or oxycontin, or cornbread, or ham hocks in the heroin” (African American male, age 56). Others pointed to not having the desire to test their drugs to determine adulteration: “You can test the product [heroin] prior to using. But the question is, who’s gonna take out the time to test it when you’re so frantic for the drug?” (African American female, age 53).
Of note, one participant indicated believing the wording in the fentanyl question “how many days did you use fentanyl alone or mixed with other drugs” was asking how many days he used drugs alone (ie, without other people present): “I understand [the question] like, how many times have I gotten high off fentanyl without someone around or near [me]” (African American male, age 37).
Past 90-day recall
Participants indicated they had little difficulty remembering their past 90-day use when answering the question, as most used daily or almost daily. Two participants who used less frequently discussed how they were able to easily remember drug use occasions. The first recalled his single drug use event had resulted in an overdose the evening before the interview: “I only got high once. . .I remember before OD’ing [I] was snorting it and then waking up in the ambulance” (White male, age 29). The second participant, who occasionally used prescription opioids, described his ability to easily recall drug use because it was a rare and significant event: “I remember it down to a ‘T,’ because I don’t really do nothing much of everything every day. So, it’s like I can recall, or I can estimate roughly how many I did. . .So, I could say, yeah, I got a good memory” (African American male, age 26).
Interpretation of urine test results and 30-day self-report follow-up
All but one participant remembered receiving the urine test result before being asked the second fentanyl use question.
So, after you had been asked this question, a little bit further into the interview, they had given you a urine test, right? And the urine test came back and had a fentanyl result. Do you remember what that said on it?
Positive.
Okay. And do you have any issues with trusting the results of that test, or. . .?
No, I’m pretty confident.
That it was positive?
Yeah. (White, Latino male, age 28)
Responses from other participants indicated similar trust in the results of the urine test. Only one participant indicated being surprised by their urine test result. “I was in shock and disbelief [about fentanyl in the UDS] because I couldn’t believe it” (African American male, age 26). This participant was unique in that they indicated only occasional use of Percocet and that he was unaware that counterfeit pills containing fentanyl illicitly are sold as legitimate medication.
While all participants indicated they trusted the urine test result, they still answered “0 days” or “I don’t know” to the 30-day fentanyl use question after receiving the results. As with the 90-day question, participant responses were based on what they believed was in their drugs at the time they used versus knowledge gained after urine testing.
Why did you say ‘unsure’ after receiving the test results that said you were positive?
I don’t know. I guess I was looking at it from a stance like, once I got the bag, being unsure. I mean, obviously, now I know [from the UDS], but I was still going back to the whole original buy of the bag [when I answered the question]. (White male, age 29)
A different logic was provided by one participant who stated that they did not answer “yes” because they could not say with certainty how many times they used fentanyl-adulterated drugs within the window of inquiry:
And why did you answer ‘I don’t know’ when your test said positive?
Because I wouldn’t know how many days [I used] because all I ask when I go buy is, ‘Let me get some [heroin]’. . .I’m not intentionally seeking fentanyl. [I] want heroin. That’s all I ask [for]. (White male, age 28)
Later cognitive interviews revealed modifying the question to asking how many days the participant thinks they might have used fentanyl could result in a different answer:
So, during the past 90 days, how many days do you think you may have used fentanyl?
I would say eight.
Eight. Okay. Why did you come up with eight there?
Because when I relapsed on the 20th, there were several episodes where I overdosed, or became under the influence very quickly, or something like that. And then I had to get Narcan-ed.
Okay. And so, you had used more than eight times, but there were eight times specifically you’re thinking of where you had to have Narcan administered because of something that was going on after you’d taken the drugs.
Yeah. (White male, age 28)
This exchange illustrates how shifting the focus of the question prompted this participant to recall specific drug use events that suggest the presence of fentanyl in the drugs used. However, the following exchange with another participant demonstrates how this approach can also lead to speculative answers with minimal connection to actual events.
During the past 90 days, on how many days do you think you took fentanyl alone or mixed with other drugs?
. . .I would say it’s probably almost every time or the majority of the time, as opposed to saying none. . .I’m saying none because I don’t go out looking for it. And that specific question makes me want to say no, because I don’t go out and look for it, but how many times do I think I’ve used it. . .? I can answer that a little bit easier saying that, honestly, majority of the time, I believe I used it. (White, Latina female, age 36)
Discussion
This study underscores the usefulness of cognitive interviewing for understanding discrepancies in self-reported substance use and UDS results. The trend of underreporting opioid use, particularly fentanyl, is well-documented in existing literature.5,23,24,40 Underreporting of drug use often depends on the contextual factors surrounding the specific substance in question.41,42 The mass adulteration of the current illicit drug supply likely contributes to research participants’ lack of awareness and subsequent underreporting of fentanyl use, and self-report data may underestimate use of substances through unintentional ingestion. 43 Despite an understanding among experts that people are likely underreporting fentanyl use, our study provides new insights into the cognitive and contextual factors contributing to these discrepancies.
Our findings reveal that participants, even those who are aware their drugs likely contain fentanyl, may still underreport. While social desirability bias is often cited as a reason for underreporting drug use,44 -46 our study suggests, in this context, interpretation bias may be a more significant factor. 47 This is because underreporting by participants in our study appeared to be motivated by a desire to accurately report instances of explicitly known fentanyl use, rather than any perceived social threat or embarrassment.
The interpretation bias we identified, where respondents differentiate between known and suspected fentanyl use, highlights the need for more precise questioning. Additionally, considering the complexity of polysubstance use and the evolving nature of synthetic opioid consumption, 15 it is crucial to craft questions that capture both intentional and unintentional exposure. Depending on a specific study’s goals, researchers should phrase questions to explicitly assess participants’ known, unknown, or suspected fentanyl use. As demonstrated by Carroll et al, 3 explicitly framing questions to assess for known or intentional consumption versus assumed or probable consumption will lead to much different answers. Addressing such cognitive and contextual factors in our questions can enhance the reliability of self-reported data that is critical for informing surveillance strategies, public health policies, and interventions aimed at mitigating the overdose crisis.28,42
Future research should prioritize methodological enhancements to address the discrepancies observed between self-reported drug use and UDS data. This could involve exploring alternative survey methods and revising questions to ensure they accurately capture the nuances of fentanyl and opioid use across various contexts. Using qualitative methods, such as cognitive interviews, allows researchers to refine questions, evaluate participant interpretation, and minimize misinterpretation and response bias. 32 In terms of our current clinical trial, the cognitive interviewing process yielded the following recommendations for question revisions in future work:
Formulate more specific questions to clarify participants’ understanding of perceived/potential, intentional, and/or known exposure.
Suggested modifications to existing questions/additional questions:
Regarding limitations, this study was conducted at a single site with participants from the greater Chicago area, and the findings may not generalize to participants in other locations. Inclusion criteria required the detection of fentanyl in a UDS. Given the detection of fentanyl varies depending on multiple factors (eg, timing of recent use, dose, potency, individual metabolism), some individuals may not have been identified for eligibility in the study, leading to sampling bias. The relevance of participants’ experiences was ensured by including those who did not report fentanyl use despite the detection on this substance in their UDS. Future work should include evaluating fentanyl question interpretation using cognitive interviews to examine the transferability of these findings to users of illicit substances in other settings. Although the sample size was small, this is typical for cognitive interview studies. Samples with as few as 5 to 6 participants have been noted to provide useful information for the improvement of questions. 29 In addition, small sample sizes are justified when the sample holds more relevant direct information to the study’s goals, 48 and we used cognitive probing techniques specifically to explore discrepancies in self-reported fentanyl use. In contrast, think-aloud methods are more standardized and require less training, but can also be cognitively demanding and burdensome for participants.27 -29 When conducted by an experienced qualitative interviewer, probing can provide more focus for participants who may struggle with concentration due to a variety of factors, including drug use and trauma. 29 Probing also does not interfere with the response process and generates useful verbal material that may not emerge through think-aloud methods. While only one researcher coded the interviews, summaries including un-edited quotes were shared with and reviewed by the rest of the team. All questions and concerns were discussed and resolved before themes were finalized. Those participants who were asked to provide direct suggestions for rewording questions instead elaborated on their thinking process. It is possible this was too difficult a task to do in an interview, and another approach such as actively working to rewrite questions with participants may have elicited different results. Due to time limitations, final recommendations for question rewording were not reviewed with research participants.
Conclusion
The cognitive interviews highlighted participants in the Chestnut JCOIN Research Hub trial often underreport fentanyl use due to the way they interpret the questions: participants focus on known, rather than suspected, exposure. These findings underscore the necessity of refining survey questions to more accurately capture both intentional and unintentional fentanyl use, thereby enhancing the reliability of self-reported data. Such improvements are vital for informing public health policies and interventions, especially in the context of the ongoing overdose crisis, which has expanded to include a broader range of illicit substances.
Supplemental Material
sj-docx-1-sat-10.1177_29768357251316220 – Supplemental material for “They Don’t Give You No Note If It’s Got Fentanyl”: Using Cognitive Interviews to Understand Self-report and Urine Drug Screen Discrepancies
Supplemental material, sj-docx-1-sat-10.1177_29768357251316220 for “They Don’t Give You No Note If It’s Got Fentanyl”: Using Cognitive Interviews to Understand Self-report and Urine Drug Screen Discrepancies by Dennis P Watson, Katie Clark, Michael L Dennis, Christine E Grella and M. Kate Hart in Substance Use: Research and Treatment
Footnotes
Acknowledgements
The authors would like to thank John Palmer for coordinating interviews for this study, Lyndy Kouns for providing transcription services, and Mona Stivers for providing professional editing services.
Author Contributions
Dennis Watson: Conceptualization, Methodology, Formal Analysis, Investigation, Data Curation, Writing – Original Draft. Katie Clark: Writing – Original Draft. Michael Dennis: Funding Acquisition, Conceptualization, Writing – Review & Editing. Christine Grella: Funding Acquisition, Conceptualization, Writing – Review & Editing. Kate Hart: Writing – Review & Editing, Data Curation.
Funding:
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was supported by the JCOIN cooperative, funded by the National Institute on Drug Abuse (NIDA) and by the National Institutes of Health (NIH) through the NIH HEAL Initiative (UG1DA050065; MPIs: Dennis and Grella). All opinions expressed are the authors’ and do not necessarily represent those of the funding agency.
Declaration of conflicting interests:
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethical Considerations
This study was approved by the Chestnut Health Systems Institutional Review Board in August 2023 (Approval #1179-0823).
Consent to Participate
Written informed consent was obtained by participants.
Consent for Publication
Not applicable.
Data Availability
Due to the sensitivity of information related to substance use and treatment, the dataset is available only upon request and with proper data use agreements in place. All materials developed for the study are available upon request.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
