Abstract
Objective
Varenicline is the most efficacious approved smoking cessation medication, making it one of the most cost-effective clinical interventions for reducing tobacco-related morbidity and mortality. Adhering to varenicline is strongly associated with smoking cessation. Healthbots have the potential to help people adhere to their medications by scaling up evidence-based behavioral interventions. In this protocol, we outline how we will follow the UK's Medical Research Council's guidance to codesign a theory-informed, evidence-based, and patient-centered healthbot to help people adhere to varenicline.
Methods
The study will utilize the Discover, Design and Build, and Test framework and will include three phases: (a) a rapid review and interviews with 20 patients and 20 healthcare providers to understand barriers and facilitators to varenicline adherence (Discover phase); (b) Wizard of Oz test to design the healthbot and get a sense of the questions that chatbot has to be able to answer (Design phase); and (c) building, training, and beta-testing the healthbot (Building and Testing phases) where the Nonadoption, Abandonment, Scale-up, Spread, and Sustainability framework will be used to develop the healthbot using the simplest sensible solution, and 20 participants will beta test the healthbot. We will use the Capability, Opportunity, Motivation-Behavior (COM-B) model of behavior change and its associated framework, the Theoretical Domains Framework, to organize the findings.
Conclusions
The present approach will enable us to systematically identify the most appropriate features for the healthbot based on a well-established behavioral theory, the latest scientific evidence, and end users’ and healthcare providers’ knowledge.
Introduction
Smoking cessation treatment is one of the most cost-effective clinical interventions for reducing cancer morbidity and mortality.1,2 Of the approved smoking cessation medications, varenicline is one of the most effective. 3 While several pharmacological interventions for smoking cessation work better for men than for women,4,5 studies have shown that varenicline is equally effective in both genders for achieving 1-year abstinence. 6 Thus, varenicline may help reduce gender disparities typically seen in rates of smoking cessation. 6
As with many medications, adherence to varenicline is low,7,8 and poor medication adherence is associated with lower smoking abstinence.9–13 Evidence from clinical trials shows that over 30% of people taking varenicline are nonadherent by their second week of treatment, 14 and those who are nonadherent are significantly less likely to be abstinent at multiple time points.12,14–16 Researchers have also shown that age and gender are significant factors that influence adherence to smoking cessation pharmacotherapies, with individuals of older age 8 and males more likely to adhere.17–20
A 2019 Cochrane review found that interventions that provided tailored behavioral supports improved adherence to smoking cessation medications.21,22 However, there is a need for proper implementation of such novel interventions. 23
The use of new technologies has been recognized as essential to achieving personalized, evidence-based, and economically viable healthcare. 24 Advances in natural language processing and near-universal access to smart phone-based messaging systems provide a unique opportunity to connect with patients on a large scale. Simple electronic reminders, such as text messages, have shown improvement in medication adherence 25 but only provide a solution to part of the problem, unintentional nonadherence. In order to provide a more comprehensive solution, one that embeds the behavior change interventions outlined in the Cochrane review, 21 a more complex intervention, with several interacting components, 26 is needed.
Digital health applications (e.g. text messaging and phone applications), focused on helping people adhere to varenicline, have shown promise;27–29 however, they have not been able to tailor to the needs of each user, something that has been proposed as an important mechanism to improve medication adherence. 30 Health-focused apps that have incorporated conversational agents (“healthbots”) might offer a solution, as they can provide tailored behavioral supports by tracking a patient's medication regimen, offering reminders, and answering questions about medication use, including common side effects. Healthbots are computer programs that mimic conversation with users using text or spoken language. 31
When designing a healthbot, it is important to focus on developing an appealing and easy-to-use product, grounded in knowledge about the people and contexts where the innovation will ultimately be deployed.
32
Equally important is to anchor a theoretical underpinning to the intervention,33,34 especially given that the lack of a theoretical base in smoking cessation programs and in adherence interventions has been identified as a gap which might affect the interventions’ success and effectiveness.35–38 The development of the healthbot will follow the guidance of the UK Medical Research Council on complex interventions, which advocates selecting components of a complex intervention by using both theory and evidence and suggests this is done in a number of iterative phases with feedback loops. The
The objective of the study described in this protocol is to co-create a theory-informed, evidence-based, patient-centered healthbot, aimed at helping people adhere to their varenicline regimen, and which healthcare providers (HCPs) feel comfortable recommending. The healthbot developed from this study, once evaluated for its feasibility and effectiveness, may be utilized by patients using varenicline to stop smoking as well as HCPs recommending varenicline, to improve adherence and smoking cessation.
Methods
We will employ a user-centered approach to build the healthbot in order to optimize user experience and achieve the best uptake and utilization.45,46 Following the

Co-creation of a varenicline adherence conversational agent.
Discovery phase
Literature review
Purpose
The aim of this rapid review is twofold: (a) to identify the barriers and facilitators to varenicline adherence, in people using varenicline for smoking cessation (we will use the Theoretical Domain Framework 41 to organize our data extraction) and (b) to identify the behavior change techniques that are associated with helping people adhere to their varenicline treatment (we will use behavior change technique v1 by Michie et al. 42 to organize our data extraction).
Methods
Article inclusion will be based on the population, intervention, comparator, and outcome method for eligibility; the population will be individuals using varenicline for smoking cessation; the intervention will be the use of varenicline; and studies using multiple smoking cessation medications will be included if they reported factors associated with only varenicline users separately. The outcome will be reported factors associated with adherence to varenicline.
Analysis
A narrative synthesis of the included studies will be used to summarize the barriers and facilitators to varenicline adherence (using the Theoretical Domain Framework) as well as to identify the behavior change techniques that are associated with helping people adhere to their varenicline treatment.
Semistructured interviews with people who use varenicline
Purpose
To gain in-depth understanding of the challenges and solutions people who use varenicline encounter and to understand what features users would like to see in a healthbot designed to help them adhere to varenicline.
Recruitment
Recruitment will occur via community boards, CAMH's list of treatment-seeking smokers who consented to being contacted for future studies (the list contains 75,000 contacts), and community agencies partnering with CAMH (such as Rainbow Health Ontario). Interested individuals will contact a team member who will explain the study and assess for inclusion criteria. If eligible, participants will be enrolled in the study.
Eligibility
People taking varenicline to quit smoking or self-identifying as nonadherent to their varenicline treatment (i.e. having difficulty taking varenicline as prescribed by their HCP) in the last 6 months, age ≥18 years.
Interviews
We will interview 20 participants over the phone using a semistructured interview guide, which will probe for common problems taking varenicline and solutions considered appropriate or useful. The interview will follow concepts outlined in the Theoretical Domain Framework 41 and probe for the 11 behavior change techniques that have been identified as possible ways to help people adhere to their medications. 35 Please see attached Appendix I for the patient interview guide. The Theoretical Domain Framework provides a framework to examine key determinants (i.e. theoretical domains) of the target behavior (i.e. medication adherence) that can then help identify a specific behavior change intervention. 51 In order to capture a nuanced and comprehensive account of context, we will use the Theoretical Domain Framework with an intersectionality lens. 52 Intersectionality, which accounts for the interface between social identity factors (e.g. age, gender, ethnicity, geography, and class) and structures of power (e.g. racism, ageism, and sexism),53,54 will offer a novel approach to understand how context shapes the features that should be included in the healthbot. Participants will receive an honorarium in the form of a $30 gift card.
Analysis
Interview recordings will be transcribed verbatim and imported into NVivo. We will analyze the data using a framework analysis approach (a type of thematic analysis)
55
which involves a five-step process: (a) familiarization, (b) identifying a thematic framework, (c) indexing, (d) charting, and (e) mapping/interpretation. The analysis will be an ongoing iterative process. Two research staff will review the transcripts to
Semistructured interviews with healthcare providers who help people who want to quit smoking
Purpose
There is currently little guidance regarding HCPs’ perspectives in recommending digital health solutions to their patients; therefore, understanding the perspectives of HCPs is crucial to facilitate the effective delivery and uptake of the healthbot. These semistructured interviews will explore (a) barriers/facilitators influencing HCP provision of digital health solutions to patients prescribed varenicline; (b) identify theoretical domains to target for behavior change; and (c) select behavior change techniques to include in the healthbot.
Recruitment
We will recruit 20 HCPs who work with people who are trying to quit smoking including those who work in primary care sites, addiction agencies, and pharmacies. We will also invite HCPs that work at the Nicotine Dependence Clinic at CAMH and in smoking cessation programs in Ontario. A purposive sampling strategy will be used to allow for the inclusion of a variety of sites working in different locations and types of settings.
Eligibility
People currently working in a primary care setting, an addiction agency, smoking cessation programs, or registered pharmacy, and as part of their job counsel people who want to quit smoking.
Interviews
We will interview 20 participants, over the phone, using a semistructured interview guide, which will probe for barriers and facilitators for implementing/recommending digital health solutions into their care. The interview will follow concepts outlined in the Theoretical Domain Framework. 41 Please see attached Appendix II for HCP interview guide.
Analysis
Interview recordings will be transcribed verbatim and imported into NVivo and analyzed using framework analysis.
55
The analysis will be an ongoing iterative process. Two research staff will review the transcripts to
Design/Build phase
Wizard of Oz method
Purpose
The purpose of this stage is to prototype and rapidly evolve the healthbot in an efficient manner. 48 Beginning with structures and responses as suggested by the previous phases, a human-powered wizard will appear as if it is an automated healthbot, allowing the fast evaluation and easy modification of the core components of the healthbot.
Recruitment
To ensure that we enroll a diverse group of participants in terms of socioeconomic status, race, gender, age, place of residence, etc., we will use purposeful sampling and recruit through a variety of methods. These will include community boards, CAMH's list of treatment-seeking smokers who consented to being contacted for future studies (list contains 75,000 contacts), community agencies partnering with CAMH, and asking HCPs working in smoking cessation programs to inform their patients. Interested individuals will contact a team member who will explain the study and assess for inclusion criteria. If eligible, participants will be enrolled in the study.
Eligibility
Currently taking varenicline to quit smoking or self-identifying as nonadherent to their varenicline treatment in the last 6 months, age ≥18 years, and has a smartphone with data plan.
Procedures
Forty participants will interact with a healthbot over a smartphone for 10–30 minutes per day for 3 days and/or ask 5–10 questions per day for 3 days. They will be able to send a text (through an existing platform) to the “healthbot” (which will be controlled by a research staff member—the “wizard”—instead of by the actual healthbot). The wizard will be given a library of predefined responses (informed from the interviews and the literature) to provide to participants. Upon user engagement, the wizard will choose an adequate response by clicking on the respective library entry. If none of the listed answers are appropriate for the users’ request or questions, the wizard will reply, “Sorry, I do not have any information available on this topic; please consult with your healthcare provider. If it is an emergency, please call 911.” We expect that the initial library of predefined system prompts (derived from the literature and the interviews) will not be sufficient for the various requests; the results of this phase will help us build this library. At the end of their interaction, a research staff will interview participants over the phone. The interviews will be audio recorded and probe for the users’ experiences with the healthbot and suggest any improvements. Participants will receive an honorarium in the form of a $50 gift card.
Analysis
The text conversations between participants and the wizard, as well as the audio recordings from the brief interviews, will be imported to NVivo for data management and analysis. We will start by examining all the cases where the wizard had to reply, “I do not have any information available on this topic,” the answers that were developed for these questions as well as cases where answers could not be developed, and the evolved library of predefined responses that the healthbot could use. We will then categorize all the questions participants asked the wizard into themes and code keywords that will serve to identify questions the chat should be able to answer. The data will be analyzed using thematic analysis to capture the desired healthbot features.
Building the healthbot (Build phase)
Once we have a list of all behavior change techniques identified in the literature and by participants (Discovery and Design phases), the research team will rate them based on whether it is affordable, practical, effective, acceptable, safe, and equitable57,58 and then decide which behavior change techniques to include in the healthbot. Following guidance from the Nonadoption, Abandonment, Scale-up, Spread, and Sustainability framework, 59 we will build the healthbot choosing the simplest sensible solution for the design. The complexity of healthbots can vary significantly, 60 the simplest are rule-based, meaning they choose responses by searching on keywords and selecting appropriate responses from a predefined library. If users ask questions that do not contain a known keyword, the healthbot will respond with a default message such as, “Sorry, I did not understand.” 61 Despite these limitations, simple healthbots are increasingly used. 61 Conversely, complex healthbots use machine learning to determine the meaning of a user's words, which allows them to select an appropriate response to questions. Although the capabilities of such healthbots are rapidly evolving, they do not yet have full human-level language abilities.60,62 For this study, we will build a rule-based healthbot, which has capabilities to determine the meaning of users’ words. The rules will come from the literature review, the interviews, and Wizard of Oz testing.
Test phase
Purpose
Once the healthbot has been co-created with the help of the interviews and Wizard of Oz study, it will be tested to ensure accurate functioning and resolve any errors. The purpose of this phase will be to refine the healthbot with the help of participant feedback, to prepare for future feasibility and effectiveness research.
Recruitment
Methods similar to the Wizard of Oz study will be utilized to recruit participants for the testing of the healthbot.
Eligibility
People taking varenicline currently to quit smoking or self-identifying as nonadherent to varenicline regimen to quit smoking in the past 6 months, 18 years or older, and have a smartphone with data plan.
Procedures
Twenty participants will interact with the healthbot for 1 hour in order to train, beta test, and validate it. After interacting with the healthbot, they will complete a survey providing feedback on the overall experience of using the healthbot and suggest improvements. Participants will receive a $30 gift card as honorarium.
Analysis
Chatbot analytics will be examined to capture any glitches in the system (crashes, wrong answers given, etc.). Usage data will be downloaded to understand which questions the chatbot was unable to answer. The follow-up survey will be analyzed using descriptive statistics to explore if any changes to the chatbot are needed.
Discussion
Research has shown that offering behavioral supports improves adherence to smoking cessation medications. 21 This study will develop a healthbot as an option to provide these behavioral supports. The healthbot will be built using the latest evidence on barriers and facilitators to varenicline, behavior change theories, and input from end users. By the end of this study, we will have built a healthbot to help people adhere to their varenicline treatment using a systematic, evidence-informed, theory-based approach. Building the healthbot in this way is in line with the UK Medical Research Council complex intervention framework. 33 The results of this study will pave the way for a feasibility evaluation which will then provide clear direction whether to proceed with a randomized controlled trial to examine the effectiveness of the healthbot.
Potential challenges and appropriate mitigation strategies
A common challenge shown in the literature is that the uptake of technology in healthcare is slow63,64 and there seems to be a gap between the development of technology and usefulness in practice, with many technologies further increasing healthcare complexities. 65 Thus, for this study, we are proposing to use several implementation theories and frameworks to make sure we build from existing knowledge and we can gain insights into the different mechanisms by which implementation of the healthbot is more likely to succeed.
Given that the provision of digital health can create equity problems (e.g. if the new technology is not accessible to communities that have traditionally been underserved), we are proposing to build the healthbot with social inclusivity in mind. Our plan includes recruiting a diverse group of people (in terms of socioeconomic status, race, age, and gender) and stratifying by age and gender to help with the codesign of the healthbot. In addition, we will build the healthbot to work on operating systems from 2016 or newer, so that most Canadians (80%) can access it. 66 However, 26% of Canadians with incomes of less than $20,000 and 20% those 60 years or older do not own a smartphone 66 and thus will not be able to benefit from this intervention.
Expected impact
Ontario, as much of the world, is experiencing insufficient resources to meet the demand for care, especially given the treatment backlog created by the pandemic. The use of healthbots is a potential partial solution that could increase access to and quality of care and health information in a cost-effective manner. Thus, as demands for digital healthcare strategies increase, this proposed research will advance our knowledge about using healthbots for medication adherence, using an equity lens. In addition, this project has the potential to scale-up to primary care settings, addiction agencies, public health units, and other jurisdictions nationally and internationally to support tobacco control efforts. The intervention can also extend to other medications at a relatively low cost. Finally, because smoking cessation is one of the most important cancer prevention strategies, 67 and with at least one life saved from a tobacco-related death for every two smokers who quit, 68 our research has the potential to impact cancer control and improve quality of life. In Ontario, the national commitment 69 to achieving 5% prevalence of smoking by 2035, strongly endorsed by the Modernization of Smoke-Free Ontario executive steering committee, 70 will require expansion of available cessation services.
Conclusions
To the best of our knowledge, a theory-informed approach has not been utilized to develop a digital health solution to improve adherence to varenicline for smoking cessation. The findings from the study will have the benefit of utilizing a health equity lens to develop a healthbot for behavior change.
Future research will include a nonrandomized single-arm feasibility study where participants will interact with the co-created healthbot from this study for 12 weeks while taking varenicline for smoking cessation. Based on the findings from the feasibility study, a randomized controlled trial will be conducted to assess the effectiveness of the healthbot.
Supplemental Material
sj-docx-1-dhj-10.1177_20552076231182807 - Supplemental material for Cocreation of a conversational agent to help patients adhere to their varenicline treatment: A study protocol
Supplemental material, sj-docx-1-dhj-10.1177_20552076231182807 for Cocreation of a conversational agent to help patients adhere to their varenicline treatment: A study protocol by Nadia Minian, Kamna Mehra, Jonathan Rose, Scott Veldhuizen, Laurie Zawertailo, Matt Ratto, Julia Lecce and Peter Selby in DIGITAL HEALTH
Supplemental Material
sj-docx-2-dhj-10.1177_20552076231182807 - Supplemental material for Cocreation of a conversational agent to help patients adhere to their varenicline treatment: A study protocol
Supplemental material, sj-docx-2-dhj-10.1177_20552076231182807 for Cocreation of a conversational agent to help patients adhere to their varenicline treatment: A study protocol by Nadia Minian, Kamna Mehra, Jonathan Rose, Scott Veldhuizen, Laurie Zawertailo, Matt Ratto, Julia Lecce and Peter Selby in DIGITAL HEALTH
Footnotes
Contributorship
NM conceived the study with the support of JR, PS, SV, LZ, and MR. KM was involved in gaining ethical approval. NM wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.
Declaration of conflicting interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: NM reports receipt of funding from Canadian Institutes of Health Research, Canadian Cancer Society, and from the Discovery Fund of CAMH. PS reports receipt of funding from Canadian Institutes of Health Research, Canadian Cancer Society, Pfizer Inc., Pfizer Canada, and Ontario Lung Association. PS also reports that through an open tender process, Johnson & Johnson, Novartis, and Pfizer Inc. are vendors of record for having provided smoking cessation pharmacotherapy for research studies at free or discounted rates.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is funded by a Proof of Concept Intervention Grant in Primary Prevention of Cancer (Action Grant) of the Canadian Cancer Society and the Canadian Institutes of Health Research-Institute for Cancer Research (grant #707218) and a Canadian Institutes of Health Research Project Grant: Funding Reference Number: PJT 180405.
Ethical approval
The Research Ethics Board of the Centre for Addiction and Mental Health approved this study (REB number: 50/2022).
Guarantor
NM
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
