Deep canvassing with automated conversational agents: Personalized messaging to change attitudes

Abstract

We test a social media conversational agent for canvassing on the topic of anti-transgender prejudice, replicating and benchmarking treatment effects. In-person deep canvassing is the gold standard for durably changing attitudes on polarizing topics. However, door-to-door canvassing is costly, and many populations may not be feasibly reached in this manner. Campaigns are already conducting outreach using digital tools, including text messages and social media. If appropriately trained agents messaging over social media can achieve a fraction of the effect of in-person canvassing, canvassing may be scaled up to achieve large overall impacts at lower costs. Scripts used in this application are based on those used by transgender allies in the original study. To personalize messaging, the conversational agent uses natural language processing to detect conversational topics, and shares relevant pre-scripted messages of information and third-person experiences, encouraging respondents to engage in perspective-taking with respect to an outgroup. This study demonstrates the potential of automated social media messaging for deep canvassing, with possible applications by governments, public health agencies, and political organizations. Estimated effects are positive and significant under covariate adjustment and reweighting; due to important differential attrition, partial-identification bounds are also reported and include zero.

Deep canvassing has emerged as the gold standard for durably changing attitudes on polarizing topics (Broockman and Kalla, 2016). While traditional canvassing emphasizes one-way delivery of information or persuasive arguments, deep canvassing aims to establish connections between voters and canvassers where both participants contribute to emotionally significant conversations. Deep canvassers share their own personal experiences related to a proposed issue, and invite voters to do the same.

This model has been widely adopted by activist organizations and political campaigns, and studied across topics and settings, including for reducing exclusionary attitudes (Kalla and Broockman, 2020), promoting local campaigning (Górecki and Marsh, 2012), and increasing turnout of female voters (Cheema et al., 2023). Underlying proposed mechanics involve active processing, self-persuasion, cognitive dissonance, and perspective-taking (Brennan and Jackson, 2022).

As voters’ political and social lives have moved online, political outreach has also moved to rely on digital media. Campaign strategies have evolved to include digital communications such as text messaging (Bhatti et al., 2017; Dale and Strauss, 2009), social media engagement (Ohme, 2019; Stier et al., 2020), and even unconventional digital platforms like dating apps (e.g., Tinder’s “Swipe-the-vote” campaign) to reach potential voters.

The COVID-19 pandemic also changed campaign activities, reducing in-person contact. Traditional methods of campaigning, such as door-to-door canvassing, town hall meetings, and large-scale rallies, faced unprecedented challenges due to social distancing requirements and public health concerns. The pandemic underscored the necessity for finding new ways to engage with voters without physical contact, leading to innovations such as the adoption of virtual events, online fundraising efforts, and digital canvassing (Bin-Nashwan et al., 2022; Landman and Splendore, 2020).

In this context of digital transformation, our research aims to explore whether the principles of deep canvassing can be translated to online interactions with an automated conversational agent. We replicate and extend Broockman and Kalla (2016), using modified versions of the trans-ally scripts implemented in the original study, delivered by a Facebook Messenger chatbot. The treatment has statistically significant effects on several measures of transgender acceptance. However, we observe differential attrition between the treatment and control groups (with completion rates of 69.61% compared to 90.49%, respectively), which we attribute to the more involved nature of the deep canvassing treatment condition. To account for potential bias from this attrition, we use pre-test covariates to produce re-weighted and covariate-adjusted estimates, and find that our results are robust to these adjustments. We measured demographics and baseline policy attitudes (healthcare, climate, abortion, immigration) prior to random assignment; we did not elicit pre-treatment measures of our primary outcomes (feelings thermometer, tolerance index). For a more conservative approach, we apply Manski bounds (Manski, 2003). These bounds provide estimates under scenarios where missing data are imputed with minimum and maximum values observed in the survey responses; bounds on such conservative treatment effect estimates include zero.

This research contributes to understanding the processes of persuasion in the deep canvassing model, even when the in-person conversational experience is removed. We also provide additional context for understanding human-computer interactions.

Channels of persuasion

Persuasion through deep canvassing relies on motivated processing (Petty et al., 1986), or “system 2” deliberation over message content (Kahneman, 2013; Stanovich and West, 1999). In particular, subjects are encouraged to engage in perspective-taking with respect to an outgroup (Gehlbach et al., 2012).

However, people may disengage while scrolling social media sites, and online discourse can be more impulsive and vitriolic as compared to in-person discourse (Halpern and Gibbs, 2013; Pennycook and Rand, 2019). Consequently, it is not obvious that the principles of deep canvassing would translate to online settings and digital communication. However, initial evidence is promising. Phone-based deep canvassing efforts have effectively moved vote preferences (John and Brannan, 2008). And personalized messaging exploiting existing social networks can move turnout (Schein et al., 2021).

Digital discourse

Our study contributes to the growing body of research on the use of social media platforms for conducting behavioral interventions and recruiting participants for experimental studies. A key benefit of using social media for recruitment in our study is that it allowed us to reach the very same online population that a real campaign or policy organization would target if they were to implement a similar intervention, ensuring that our findings are directly applicable to the intended audience of such efforts.

We also contribute to research on computer-human interaction. There is evidence that in some settings, respondents may be more open and honest with computer-based correspondents as compared to humans (Lucas et al., 2014). Indeed, users of ELIZA, an early natural language processing computer therapist developed at MIT in 1966, reported feelings of connection and openness with the computer program.

Chatbots and other conversational agents, sometimes backed by AI language models, are already being used or tested by government agencies, research groups, and non-profit organizations to advance policy-relevant objectives. Among these are a dialogue system delivered through a website designed to address questions and concerns about COVID-19 vaccines (Gretz et al., 2023), a chatbot designed to facilitate access to public data in the city of Lviv, Ukraine, Petriv et al. (2020), and “Voting Advice Applications” to provide voters with information about political parties and their platforms during Dutch national elections (Kamoen et al., 2021).

Experimental design and implementation

In the spring of 2023, we ran Facebook advertisements to recruit social media users in Florida into online conversations on policy topics. The primary issue of interest for our study was an update to Florida State Medicaid policy proposed in August of 2022. Previously, Florida State Medicaid had covered medically necessary treatments for gender dysphoria, including puberty blockers, cross-sex hormones, and surgery. In 2022, the Florida Agency for Health Care Administration issued a report claiming that these treatments were “experimental” and “not consistent with generally accepted professional medical standards.” The Florida Legislature then passed a statute explicitly prohibiting the expenditure of state funds, including Medicaid, for “sex reassignment prescriptions or procedures.” ¹ (This law was later struck down by a federal judge in June of 2023 in Dekker v. Weida).

Subjects in a deep canvassing treatment group discussed these updates to the Florida State Medicaid policy. Subjects in a placebo control condition also participated in an online conversation, but instead on the topic of “Ethan’s Law” regarding new legislation on boating safety in Florida. ²

Survey

When users clicked on our advertisements and then agreed to start a conversation, they were connected to a conversation with a research page on Facebook Messenger. For realistic comparison to real policy canvassing interventions, subjects were not paid to engage in conversation with our page. After informed consent, we measured covariates and baseline policy positions before proceeding to the canvassing conversation.

10,377 users consented to participate in the study, but only 5243 completed our pre-experimental survey module on demographic covariates. This level of attrition is common in online surveys (Manfreda et al., 2008) and is particularly unsurprising as our respondents were not paid. We condition analysis on users who completed this section of the survey, as any attrition up to this point was prior to and thus independent of random treatment assignment. The pre-treatment survey module allows low-interest users to self-select out of the survey before they are assigned to a treatment condition. We then assigned eligible respondents who completed the pre-treatment covariates survey uniformly at random to either the deep canvassing treatment group (2560) or the placebo control group (2555). ³

In both conversational conditions, users were asked whether they were aware of the relevant policy, were shown a recent local news video on the policy, and then were asked about their positions on the policy, having seen the video. In the deep canvassing treatment condition, as in the original study, the treatment script provides a longer and more in-depth conversational experience as compared to the placebo control script. In this condition, after users shared their initial positions, the canvasser bot then defined the term “transgender.” While the bot cannot share its own experience of being transgender, it can encourage users to take the perspective of others. To do so, it shared a Vice news video about a transgender Florida resident affected by the legislation. It then prompted its conversational partner to share their own experiences with people who might be affected by the policy. The bot also prompted users to express their responses to or concerns related to the video shown.

The treatment version of the conversational agent integrates a natural language processing model to classify this text related to questions or concerns regarding the video shown. This facilitates a more naturalistic conversation: when a user answers this open ended question, the conversational agent is able to classify their answer, and retrieve a topically relevant response. While this version of the conversational agent is flexible, it is still a retrieval-based chatbot, and is somewhat constrained as a conversational partner; all responses are fully pre-scripted and no novel responses are generated. This design does, however, ensure that the chatbot cannot generate responses that contain unverified information or otherwise perform in ways that are not anticipated by the research team. A general diagram of survey flow is represented in Supplemental Figure S1. Classification is discussed in more detail in the following section. Response texts were selected from FAQ text on the website of the National Center for Transgender Equality (transequality.org); users were informed of the source of the text. Response messages are described in Supplemental Table S13.

For both conversational conditions, we then collected post-treatment response measures, which were embedded in a longer module on policy positions. Our primary response measures are a feelings thermometer on transgender individuals and a combined transgender tolerance scale. For the feelings thermometer, respondents are asked to rate their feelings towards transgender people directly. The tolerance index is constructed via a principal component analysis, transforming a series of items measuring different aspects of transgender tolerance. We use the same index of relevant items and the same re-scaling Broockman and Kalla (2016) describe in their supplementary materials. We also include a secondary response measure for transgender policy acceptance. All response measures are described in more detail in Supplemental Section S1.2.1.

Classification

First, we generated a data set of possible questions or concerns that we expected users could have about transgender related policy and legislation. To do this, we collected relevant question-answer pairs from websites with frequently asked questions sections on LGBTQ-related policies. From these conversational topics, we hand-coded a list of possible “intent” categories, each classifying a type of question or concern users might have on this topic. This process resulted in 24 possible intent categories, and one catch-all default category for text that was not classified in any of the other categories. We then created a training data set with multiple example texts associated with each of the intents, for the purpose of allowing our eventual classifier to recognize text content. Methods to develop this training set are described in Supplemental Section S3.

With the training data set, we fine-tuned a RoBERTa-large model (Liu et al., 2019) to create a classifier for our intent categories. During study implementation, the model was re-trained to include new text inputs while we implemented the survey, and the quality of responses to users’ inputs improved over time. In particular, we added a new category, “impacts of policy,” during the survey, as many users responded based on political topics or how they felt about policymakers’ positions on the relevant issues. Real user input text and the associated classification category are described in Supplemental Table S11. The majority of responses are classified in the default category, either because user content was off-topic or included profanity, or users gave short responses that were not descriptive (e.g., “just i don’t want to pay for this nonsense”; “yeah. what the hell is this world coming to?”; “government has not gone to medical school”; “i’m fairly well versed in this subject.”). However, due to inclusion of new categories, classification improved over time. For descriptive analysis of the model, after the final re-training, we delivered only the treatment version of the survey to a holdout sample of 340 subjects, which are not included in analysis of experimental effects reported below. Among these subjects, 74.7% were assigned the default category, 18.2% were assigned the impacts of policy category, and remaining users were assigned over seven other topic areas (see Supplemental Table S12).

Results

Among the treatment group, 69.61% of the respondents finished the post-treatment survey and provided valid responses, whereas the completion rate in the placebo group is 90.49%; see Table 1. This observed differential attrition rate is not wholly unexpected—typically, online survey respondents show higher dropout rates when faced with longer or more complex tasks, which is the case for the treatment section of our study (Galesic and Bosnjak, 2009)—although it is critical to investigate robustness of our findings to different approaches to adjust for this differential attrition in analysis.

Table 1.

Attrition.

Group	N	Percent	Finish rate
Consented and began pre-experimental module	10 377	−	50.53
Of those who completed pre-experimental module:
Control	2555	49.95	90.49
Treatment	2560	50.05	69.61

Our target estimand is the average treatment effect. In this analysis, we include only respondents who have completed the full post-treatment response module. We report covariate balance among respondents who completed the pre-test module in Supplemental Table S6, and among respondents who completed the post-treatment survey in Supplemental Table S7. We first present simple unadjusted difference-in-means estimates. However, while we do not see evidence of imbalance among those who completed the pre-test module, we note the evident covariate imbalance among completers. To address this, we also present estimates of the average treatment effect under the assumption that missingness is conditionally ignorable using features measured in the first survey module, including demographic characteristics (age, gender, ideology, party identification, political interest, religiosity), and policy positions on healthcare, climate, abortion, and immigration issues. Because the control canvassing intervention is quite short, we do not include pre-test response measures in our pre-treatment covariates. For these covariate-adjusted estimates, we de-mean covariates and interact them with treatment (Lin, 2013). In addition to unadjusted difference-in-means estimates and covariate-adjusted estimates, we also present estimates that are both covariate-adjusted and re-weighted, using inverse probability weighting. This reweighting is to the covariate distribution of all users for whom we collected pre-test measures and to whom a treatment condition was assigned. Note that this is a slightly different target population than the estimand for the first two estimation strategies. To calculate these weights, conditional on completing the pre-test covariate module and having been assigned treatment, we estimate propensity to be in each condition and to not have attrited. Analysis code is available in our pre-analysis plan. Results are also reported in table format in Supplemental Table S1.

Results in Figure 1 indicate that the intervention was effective in moving affect towards transgender individuals. Under the simple difference-in-means estimator, the intervention is associated with an increase in transgender favorability by 0.928 scale points (s.e. = 0.102) as compared to the placebo group; this is equivalent to moving an individual approximately one point on the 0-10 feelings thermometer. The covariate-adjusted estimate is 0.786 scale points (s.e. = 0.082); reweighting to all users assigned treatment, the estimate is 0.773 scale points (s.e. = 0.084).

Figure 1.

Treatment effects: affect, tolerance, and policy support. Point estimates (circles) and 90 and 95% confidence intervals (bars) are represented for different estimators.

Results in Figure 1 indicate that the intervention was also effective in promoting the acceptance of transgender individuals as measured in the transgender tolerance scale. Under the simple difference-in-means estimator, the intervention is associated with an increase in transgender tolerance by 0.181 scale points (s.e. = 0.030) as compared to the placebo group. The covariate-adjusted estimate is 0.113 scale points (s.e. = 0.017); reweighting to all users assigned treatment, the estimate is 0.117 scale points (s.e. = 0.017). These effects are comparable in magnitude to results reported in Broockman and Kalla (2016) who find treatment effects of 0.19 standard deviations among respondents who had conversations with non-transgender canvassers in the first survey after the canvassing intervention, and 0.32 standard deviations 3 months after the intervention (Broockman and Kalla 2016 note that point estimates for the tolerance measure are not directly comparable over time due to changes in factor loadings; for this reason the effect estimates are also not precisely comparable between studies).

Secondary outcome

The intervention was also effective at moving respondents’ positions on policy related to gender-affirming care, as reported in Figure 1, which estimates the treatment effect on support for relevant policy measured by −3 to +3 Likert scales. The treatment group was more supportive of policy measures that would provide gender-affirming health care by 0.229 scale points (s.e. = 0.064) as estimated by the difference-in-means estimator; 0.103 scale points (s.e. = 0.042) as estimated by the covariate-adjusted estimator; and 0.114 scale points (s.e. = 0.042), as estimated by the reweighting estimator. These effects are of the same order but slightly smaller in magnitude compared to policy effects in Broockman and Kalla (2016) after the term “transgender” was defined in study item wording; those effects were on the order of 0.3 point changes in a −3 to +3 Likert scale. Broockman and Kalla (2016) find no immediate treatment effects prior to defining this term, which is why we include the definition within the treatment messaging in this study.

While we did not pre-register sub-group analysis by party, for illustration we report effects separately for Democrats, Republicans, and Independents in Supplemental Figure S2. Effects are directionally similar; treatment effects on the tolerance scale are larger for Republicans than for Democrats, who also start with a much higher control mean on the scale. Effects on policy support are only slightly lower for Republicans than for Democrats, but 95% confidence intervals for the former include zero; as Republicans are a smaller portion of our sample and so standard errors on the estimates are relatively larger.

As an additional robustness check, in Supplemental Table S8, we implement a leave-one-out placebo test following a reviewer’s suggestion: for each covariate, we repeat our covariate-adjusted estimates, dropping that one covariate, and then adjusting on all other pre-treatment covariates. We then test for balance on the held-out variable, and on additional post-treatment measures regarding unrelated policies. Weighted differences are near zero across all covariates, indicating that the covariate-adjusted procedures successfully remove imbalance in pre-treatment characteristics. Supplemental Table S9 tests for spillover to unrelated post-treatment attitudes—tax policy, marijuana legalization, and minimum wage. Only the tax-policy item shows a small, statistically significant difference.

Discussion

Our results broadly align with those of Broockman and Kalla (2016). The original canvassing study finds that brief personal interactions with strangers could reduce prejudice in a field setting; similarly, we find that brief interactions with an automated conversational agent can also reduce prejudice in a digital field setting. While we are not able to report effects separately for canvassers who are themselves transgender as compared to ally canvassers, Broockman and Kalla (2016) find effects for both types of canvassers, as well as for first-time and experienced canvassers. The persistence of these effects, even when the conversational partner is an automated computer program, suggests that the key drivers of change are rooted in respondents’ internal processing and perspective-taking.

In addition to the intended design variation of intervening in online conversations over a social media platform, there are a few key differences in our study design and measurement as compared to the in-person canvassing study. First, measurement for Broockman and Kalla (2016) was conducted via online surveys, ostensibly unconnected to the in-person canvassing interventions. In our study, both the intervention and measurement occur in conversation over the Messenger platform. Due to the nature of our design, we do not contact respondents outside of Messenger nor do we collect any information on respondents outside of this conversation. However, we also embed measurement of relevant positions within much longer modules that include policy questions on a number of additional topics both prior to and after treatment delivery, with the intention of preventing demand effects or other response bias. This also means that we measure immediate effects of the intervention, but are not able to report on treatment effect endurance.

There are, however, important benefits to our chosen delivery method. Due to the low cost and scalability of digital conversations, we are able to intervene on a larger number of respondents, with 5115 study subjects compared to 501 in the in-person original study. Respondents can engage in conversation with the agent on their own schedule; we do not need to rely on respondents who will answer the door or pick up the phone to have a conversation with a stranger. This feature is useful for study design but is also relevant for policy makers and policy oriented organizations who also wish to reach more people at scale. Broockman and Kalla (2016) observe that “over the past century, political campaigns have increasingly relied on mass communication to reach voters” often at the expense of directly “making the case for their positions.” This study demonstrates that these approaches need not be mutually exclusive.

Supplemental material

Supplemental Material - Deep canvassing with automated conversational agents: Personalized messaging to change attitudes

Supplemental Material for Deep canvassing with automated conversational agents: Personalized messaging to change attitudes by Molly Offer-Westort, Jiehan Liu, Nick Feamster, Kartik Garg, Nguyen Phong Hoang, Sudhamshu Hosamane in Research & Politics