Abstract
Campaign costs are rising, making ad execution testing more critical to determine effectiveness prior to media spending. Premarket testing occurs prior to messages’ airing while in-market testing examines message attributes when messages are aired within a real-world setting, where context plays an important role in determining audience response. These types of ad testing provide critical feedback to help develop and deploy campaigns. Due to recent changes in media delivery platforms and audience tobacco use behavior, this study analyzes two nationally representative youth samples, aged 15-21, to examine if pre-market ad testing is an indicator of in-market ad performance for public health campaigns, which rely on persuasive messages to promote or reduce health behaviors rather than selling a product. Using data from the Truth® campaign, a national tobacco use prevention campaign targeted to youth and young adults, findings indicate strong associations between pre-market scores and in-market ad performance metrics.
Significance for public health
Ecological models of health emphasize the efficient use of resources to communicate across multiple platforms to affect population health, but the current crowded media environment makes it difficult to do so. Findings suggest ad testing can provide audience feedback to maximize advertising effectiveness while remaining cost-saving.
Introduction
Tobacco use remains the leading cause of preventable death in the U.S. 1 To combat the tobacco industry's nearly $25 million/day budget to promote cigarettes and other tobacco products, antismoking mass media messages across multiple platforms have been an effective strategy to prevent youth tobacco use.1-4 In 2000, truth Initiative (formally American Legacy Foundation) launched the truth® campaign as the first such campaign to prevent youth tobacco use initiation, by highlighting the tobacco industry's deceptive marketing practices, facts about the ingredients in cigarettes, and the health consequences of smoking. 5
Recent changes in media delivery platforms (the Internet and digital devices) and audience tobacco use behavior, however, highlight the need to update the message architecture of public health campaigns, such as truth, to be successful. Given the highly fragmented media context and the striking disparities in tobacco use by education, race/ethnicity, and geographic region, it is important that tobacco prevention ad messages resonate with audience subgroups. The cost of developing and deploying campaigns is rising, making ad execution testing more critical to determine effectiveness prior to media spending. For example, in market testing for these three ads was four times the costs of pre-market testing (approximately $200,000 vs $50,000). This paper focuses on two forms of ad execution testing, pre-market and in-market testing, and examines the relationship between them to determine campaign success.
Pre-market ad testing, conducted prior to messages’ airing, exposes respondents to a near final ad. Measures assess ad receptivity, message comprehension, knowledge, perceptions, emotional response and attitudes toward the ad's issue, product, or brand. 6 Survey analyses determine whether ad exposure prompted changes in targeted attitudes or perceptions, and are used to optimize ad elements.6-8
In-market testing occurs during an ad's airing, and media penetration, placement, and context factor into message recall, receptivity, and impact on outcome(s).9-11 As messages air on both television and digital platforms, targeted audience feedback is continuously monitored to assess message effectiveness over time.
Ideally, pre-market testing predicts in-market ad performance, notwithstanding the role context plays in ad recall and receptivity. The utility of this predictive relationship would allow for cost-savings in ad development, more efficient ad deployment and, ultimately, a return on investment. The relationship between pre- and in-market ad testing has primarily focused on ads related to consumer products, in which ad performance is assessed through product sales. In these cases, creating awareness is based on the innovations associated with the product being sold. However, evidence is limited for public health campaigns like truth, which typically rely on persuasive messages to promote or reduce health behaviors such as smoking. 12 In a time when advertising is driven by disruption and innovation, it is important to understand how public health campaign messages perform, and how they can stand out in a saturated media marketplace. 13
Previous research on anti-tobacco messages has shown significant associations between perceived message effectiveness (PE) and quitting behaviors, suggesting PE is a sufficient predictor for ads encouraging cessation. 14 The relationship between pre- and inmarket testing could function similarly for campaigns focused on preventing youth from starting smoking in the current media landscape. The purpose of this study is to examine pre-market ad testing as an indicator of in-market ad performance for a public health campaign.
Design and Methods
Study design
This study examines data from one phase of the truth campaign, which aired August 2014-October 2014. It included three ads: 1) Finishers, 2) Unpaid Spokesperson (Unpaid), and 3) Response. All three ads were based on the message architecture outlined by Hornik and Woolf 15 to address attitude and belief constructs related to lower intentions of using tobacco. Finishers introduced the new campaign to inspire youth and young adults to be the generation to end smoking. Unpaid showcased celebrity smokers as unwitting marketers for the tobacco industry; and Response reminded viewers that posting pictures of themselves smoking provides free marketing to the tobacco industry.
Study population and recruitment
Pre-market testing data reflected a cross-sectional sample from Survey Sampling International's online panel of U.S. young adults aged 15-21. Data for Finishers (n = 311), Unpaid Spokesperson (n = 305), and Response (n = 322) were collected from July 8, 2014- July 13, 2014, with 938 total respondents.
In-market testing data was from a continuous tracking survey of young adults aged 15-21, with daily surveys conducted among 140 participants/week. Cross-sectional samples were from Research Now's® online research panel. Study data was restricted to those respondents who took the survey from August 24, 2014 to October 5, 2014 when the three ads were airing, and indicated that they had seen at least one of the three study ads (n = 329).
Survey procedures
Pre- and in-market surveys were conducted online where respondents were screened for age eligibility (15-21 years), asked to review a description of the voluntary study and provide consent. Both studies were approved by the Chesapeake Institutional Review Board.
In the pre-marketing testing study, respondents completed a baseline assessment and then were randomized to view one of the three ads. Respondents were exposed to the ad three times throughout the survey. All survey questions were the same for all respondents, regardless of ad viewed, except for evaluation of the ad specific communication. Respondents answered questions about demographics, recall of the truth logo, tobacco use behavior, and tobacco-related attitudes. Respondents then viewed the ad for the first time and were asked questions related to ad receptivity, emotional response, and the tobacco-related questions asked prior to the ad. After the second viewing of the ad, respondents were asked about the main idea as well as ad-specific attitudes. For the third and final viewing. respondents were asked about specific message elements, including perceptions of the truth brand and recall of the ad's tagline.
Similarly, the in-market survey assessed demographic characteristics, tobacco-related attitudes, behaviors, and ad-specific attitudes. However, the in-market survey included more items, such as media utilization, truth brand recall, tobacco-related perceptions and awareness of specific truth ads. Respondents viewed six screenshots from each ad and were asked if they had seen the ad. If yes, respondents were asked where they saw the ad and how often they had seen it in the past two weeks. The order of the screenshots was randomized to minimize bias.
Measures
Demographic variables included age (15-17 vs. 18-21), gender, self-described financial situation (don't meet basic expenses, just meet basic expenses with nothing leftover, meets needs with a little left over, live comfortably), race/ethnicity (black, Hispanic, white, other), parental education (high school education or less vs. high school diploma/GED or greater), geographic region (West, Midwest/Southeast, Southwest, Northeast/Mid-Atlantic) and smoking status (closed to smoking, at risk for smoking, current smoker).
An ad receptivity index was developed for each of the three ads based on the mean score to the following statements: “This ad captured my attention” and “This ad was meaningful.” Respondents were asked “How much do you agree or disagree with the following statement?” Responses ranged from “strongly disagree” (1) to “strongly agree” (5). The wording for one item was slightly different across surveys; in pre-market testing, it was: “This ad captured my attention” and in in-market testing, it was: “This ad grabbed my attention.” The “grabbed my attention” receptivity item is included in a broader ad receptivity scale that has been validated among adults. 14 Correlations between these two items was .65 to .72 across advertisements.
Statistical analyses
An independent-samples t-test assessed differences in receptivity between pre-market and in-market samples for each ad. Regression analyses examined whether mean ad receptivity differed for pre-market versus in-market data beyond demographic effects. Using SAS Enterprise 7.1, 16 three OLS regression models examined the effect of the pre-market vs in-market samples on ad receptivity for the three ads, while controlling for demographics and smoking status. In each model, differences in the mean estimate between pre-market and in-market were assessed to determine if it was significantly different from zero, which would suggest that there were differences in mean receptivity scores between pre-market and in-market datasets. Significant differences were noted at the 95% confidence level.
Results
Table 1 presents the sample characteristics of both the pre-market and in-market testing data during the Finish It campaign (n=1276). Demographic characteristics were similar across both samples, although the in-market sample had a higher percentage of White participants and Hispanic participants compared to the premarket sample.
Sample Characteristics of Pre-Market and In-Market Data.
Superscripts letters denote significant differences in percentages across groups using the chi-square test of association.
An initial assessment of differences in receptivity between the pre-market and in-market samples was conducted. No statistically significant association between pre-market and in market receptivity index scores were observed for Finishers and Unpaid. For the Response ad, mean receptivity was statistically significantly higher in-market (M=3.76, SD=1.05) compared to pre-market (M=3.43, SD=1.01, P<0.01).
Table 2 presents results from the multivariable linear regression model examining differences in receptivity pre-market and inmarket. Few significant differences were found for ad receptivity when comparing pre-market to in-market scores, controlling for other variables. For the Finishers and Response ads, there was no statistically significant relationship between pre-market and in market receptivity index scores. For the Unpaid Spokesperson ad, mean receptivity was higher in-market compared to pre-market (b=-0.36, P<0.01).
Pre- and In-Market Ad Receptivity Index and Campaign Related Attitudes (Multivariable).
In the pre-market study, this statement was worded as follows: “This ad captured my attention.” In the continuous media tracking study, this statement was worded as follows: “This ad grabbed my attention.” Analyses controlled for age, gender, financial situation, race/ethnicity, parental education, region and smoking status. Significance:
P<0.05
P<0.01
P<0.001.
Discussion
In-market testing can be an expensive and time-consuming undertaking, in addition to the actual development of an effective campaign. 17 No significant differences between pre-market and inmarket ad receptivity scores for Finishers and Response, when controlling for other variables, suggests that, even within the current media landscape, pre-market testing is a robust indicator of inmarket ad performance for public health campaign messages. Specifically, the three ads’ performance during testing show that receptivity to each ad successfully inspires the target audience to engage in a movement to fight against smoking. The significant differences in receptivity between pre- and in-market scores observed for Unpaid were likely a result of the number and type of optimizations made between pre- and in-market testing, including slowing the pace of the ad, featuring different celebrities, and updating the final frame to better emphasize the brand - all of which can affect recall, connection and emotional response. Premarket results for Finishers and Response indicated that these were stronger ads executions (tested more positively), and did not require as many or as extensive optimizations. Developing innovative and memorable messages that change health behaviors and prevent risk can be a marketing challenge. 18 Like other health campaigns, truth's messaging, while still important from a public health standpoint, was no longer as newsworthy as it had been when the campaign launched in 2000. Therefore, truth took an innovative approach to relaying an “old” public health message to gain favorable media attention. Research has shown that, in addition to message relevance and media delivery, contextual factors can also significantly influence the awareness of media messages, thereby heightening or limiting campaign impact.19-24 When a preceding program evokes positive affective states, participants’ evaluations of advertisements are more positive, and more positive cognitive responses are generated when program-evoked affective states are positive rather than negative.25,26 Contextual factors can provide an explanation for why 1) the bivariate analyses, which did not control for other variables, showed a significant difference for the Response ad, and 2) the higher receptivity scores during inmarket testing of Unpaid versus pre-market findings. truth premiered Unpaid and Response during the Video Music Awards (VMA), a highly anticipated and stimulating cultural program with controversial performances and significant social media use. Research indicates that campaign exposure is a tremendously important factor in being able to increase campaign awareness. Leveraging the reach and connectivity of popular televised events, such as the Super Bowl or the VMAs, is an effective strategy for message promotion and can bolster campaign impact.27-31 The contextual factors associated with the VMAs, which are aligned with truth's target audience, and the event's reach as a result of social media and organic earned media (i.e., the online word of mouth in the form of mentions, shares, reposts, reviews, recommendations, or content picked up by 3rd party sites) are a significant contributors to campaign outcomes and could explain the significantly higher recall and receptivity scores in-market for both ads.
There are sample and methodological limitations of note. Online panels recruit demographically-varied participants, but some populations could be excluded or over-represented; and, although characteristics were similar between pre-market and inmarket, the data come from different samples, preventing us from fully examining how pre-market testing results predict in-market metrics. Also, as mentioned, one question used for pre-and in-market testing was not identical. While unlikely, the slight variation may have introduced some bias.
This is the first study to examine associations between marketing metrics to assess the utility of pre-market testing to forecast inmarket ad performance for public health messages, where the outcome is attitude and ultimately behavior change − not product sales. Despite evolving communication and tobacco landscapes, findings suggest traditional ad execution testing is not only appropriate for evaluating public health messages, but associations between pre-market results and in-market ad performance can be cost-effective and maximize performance. Pre- and in-market testing can also be used to assess media platforms for each ad.
Footnotes
Contributions: JR, EH and DV conceived, designed, implemented and supervised the study. HX and LP analyzed and interpreted the data. MG and OG drafted the manuscript. JC and JR contributed to the writing and revisions for intellectual content. All authors contributed to the writing and revisions of the manuscript and provided final approval for publication submission.
Conflict of interest: the authors declare no potential conflict of interest.
Funding: This work was funded and conducted by Truth Initiative (formerly American Legacy Foundation).
Acknowledgements
The authors thank John C. Geraci from Crux Research Inc. and Tracy Budnik from CommSight.