Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults

Abstract

The study of behavioral inhibition system (BIS) and behavioral activation system (BAS) sensitivities has produced a multitude of models and frameworks to better understand decision-making and behavior. We extend this research to the study of a vaccination by exploring the relation of BIS and BAS sensitivities with vaccination willingness, receipt, and word-of-mouth. We also assess whether dimensions of vaccine hesitancy mediate these relations, testing whether they are viable explanatory mechanisms. By conducting a multi-wave survey study, our results support that BIS sensitivity does not meaningfully relate to vaccine hesitancy or vaccination outcomes. On the other hand, BAS sensitivity related to detrimental vaccination outcomes, and these effects were mediated by vaccine hesitancy dimensions associated with perceptions that vaccines cause health risks and are not needed for healthy individuals. Based on extant BIS and BAS theory, these results suggest that certain people may see refraining from vaccination as a nonpunishment.

Keywords

behavioral activation system behavioral inhibition system BIS/BIA preventive behavior vaccination vaccination word-of-mouth vaccine hesitancy

Human affect, cognition, and behavior are governed by two relatively orthogonal neurobiological systems, the behavioral inhibition system (BIS) and the behavioral activation system (BAS) (Carver and Scheier, 2001; Carver and White, 1994; Gable et al., 2000; Heubeck et al., 1998). These two systems are associated with differing cognitive structures, each activating when encountering specific stimuli to guide reactions and responses (Amodio et al., 2008; Anderson and Perone, 2023; Coan and Allen, 2003). The BIS is associated with punishments and nonrewards, whereas the BAS is associated with rewards and nonpunishments. Punishments refers to negative consequences, whereas nonpunishments refers to the removal of a punishment; likewise, rewards refer to positive consequences, whereas nonrewards refers to the removal of a reward. For instance, the BIS is activated when an employee receives a reduction in pay (punishment), but it is also activated when they no longer receive a regularly occurring bonus (nonreward); the BAS is activated when an employee receives an increase in pay (reward), but it is also activated when a long-standing reduction in pay is removed (nonpunishment). While the BIS and BAS activate in everyone, people also have idiosyncratic differences in the emergence of these two systems, wherein the influences of the BIS or BAS may be more pronounced (Beck et al., 2009; Caseras et al., 2003; Jorm et al., 1998; Muris et al., 2005). These differences are believed to be causes of differences in temperament, which influences the development of personality and—the focus of the current article—BIS and BAS sensitivities.

BIS and BIS sensitivities are individual differences that relate to whether people are more attentive, reactive, and responsive to specific types of stimuli (Bryan et al., 2022; Heponiemi et al., 2003; Renn et al., 2014). Those with a greater BIS sensitivity are more attentive, reactive, and responsive to punishments and nonrewards, whereas those with a greater BAS sensitivity are more attentive, reactive, and responsive to rewards and nonpunishments. In turn, these BIS and BAS sensitivities cause people to systematically approach certain outcomes in different manners, resulting in differing motivations and behaviors (Carver and Scheier, 2001; Carver and White, 1994; Gable et al., 2000). For instance, two people working in the same position may have different predominant motivations, as guided by their BIS or BAS sensitivities; a person with a greater BIS sensitivity may be motivated to not lose their job, whereas a person with a greater BAS sensitivity may be motivated by their opportunities for promotion. In turn, the former person may focus on ensuring minimum standards, whereas the latter may strive toward excellence.

Due to their integral nature to human functioning, the study of BIS and BAS sensitivities have produced important insights across several domains of research. Notably, these sensitivities have been repeatedly supported to influence how people perceive the world around them, which subsequently determines resultant behaviors (Franken, 2002; Lerner et al., 2018; O’Connor et al., 2009). In turn, these discoveries have produced sophisticated models of behavioral processes and enabled researchers to explain significant variance in behavioral outcomes (Bijttebier et al., 2009; Koy and Yeo, 2008; Mardaga and Hansenne, 2007). Due to the success of prior research, the current article studies BIS and BAS sensitivities to better understand a particularly important behavior to the modern world—vaccination. In doing so, we suggest that the relations of these outcomes with vaccination outcomes is particularly unclear based on prior theory and present dynamics surrounding vaccination in the context of the current study, the United States.

Public perception and messaging surrounding vaccination are often associated with punishments (Cartmell et al., 2019; James et al., 2021; Lawes-Wickwar et al., 2021). Public health organizations, for example, regularly emphasize the risks of sickness from being unvaccinated, perhaps because research has frequently supported that, counter to the prevention-detection framework, loss-framed messages are more effective than gain-framed messaged for promoting vaccination (Hong and Hashimoto, 2023; Liu et al., 2019; Nan, 2012). More rarely do organizations highlight rewards of vaccination, although they do stress the nonpunishments of vaccination (discussed below). These dynamics would suggest that BIS sensitivity relates to more favorable vaccination outcomes; those with a greater BIS sensitivity may be more focused on the potential punishments of being unvaccinated, and they may subsequently be particularly attracted to becoming vaccinated. At the same time, misinformation during the COVID-19 pandemic often focused on the apparent health risks of vaccines, misguiding people into believing that vaccines themselves pose serious health risks (Basch et al., 2021; Bierwiaczonek et al., 2022; Howard and Davis, 2023; Wardle and Singerman, 2021). These perceived health risks associated with vaccination may cause the behavior to be considered a punishment, as the behavior could be perceived as causing health issues that would otherwise not occur. This would suggest that BIS sensitivity may relate to less favorable vaccination outcomes, as people may be focused on the perceived health risks of vaccines. Those with a greater BIS sensitivity may be more focused on the potential detrimental outcomes of receiving vaccines, and they may be particularly adverse to becoming vaccinated. Therefore, extant theory provides justifications for BIS sensitivity relating to both more and less favorable vaccination outcomes.

Alternatively, public perception and messaging surrounding vaccination are also associated with nonpunishment. Public health organizations regularly emphasize the potential to avoid impending sickness from vaccination (Cartmell et al., 2019; James et al., 2021; Lawes-Wickwar et al., 2021), which could constitute the removal of a punishment. For this reason, BAS sensitivity could be expected to relate to more favorable vaccination outcomes, as these people may be particularly focused on the potential nonpunishments of being vaccinated. At the same time, vaccination was perceived as an almost compulsory behavior by many during the COVID-19 pandemic (Savulescu, 2021; Smith et al., 2021; Vanderslott and Marks, 2021), as the widespread implications of the pandemic caused many people to feel that vaccination was necessary to protect themselves and their family. Several vaccines are also required for schools and colleges, perhaps causing people to further feel that vaccines are almost inevitable. When paired with misinformation surrounding vaccination that heightened perceived risks associated with vaccination, some people may have felt that not becoming vaccinated was avoiding an impending punishment (i.e. nonpunishment) (Cartmell et al., 2019; James et al., 2021; Lawes-Wickwar et al., 2021). For this reason, BAS sensitivity may relate to less favorable vaccination outcomes, as people may be focused on avoiding detrimental outcomes of receiving an almost inevitable vaccine. Therefore, like BIS sensitivity, extant theory similarly provides justifications for BAS sensitivity relating to both more and less favorable vaccination outcomes.

We test each of these possibilities in the current article. That is, we assess whether BIS and BAS sensitivities positively, negatively, or do not relate to vaccination outcomes; however, we also investigate a relevant explanatory mechanism to determine the causes of our studied relations. In developing our arguments above, we highlighted that both those with higher BIS and BAS sensitivity may be particularly focused on perceived risks of vaccination, suggesting that vaccine hesitancy may mediate the relations of BIS and BAS sensitivity with vaccination outcomes. Vaccine hesitancy refers to a state of uncertainty regarding whether to receive vaccines (Bussink-Voorend et al., 2022; Dudley et al., 2020), and it is commonly operationalized by assessing negative perceptions regarding vaccines, such as asking parents about perceived risks of vaccination for their children (Akel et al., 2021; Shapiro et al., 2018). We test whether vaccine hesitancy mediates the relations of BIS and BAS sensitivities with vaccination outcomes in the current article by applying Howard’s (2022a) multidimensional vaccine hesitancy scale (MVHS). The MVHS has significant prior psychometric and validity support (Balgiu et al., 2022; Howard, 2022a, 2022b, 2022c; Howard and Davis, 2023), and its construct coverage is broader than most other measures of vaccine hesitancy. This operationalization includes eight dimensions that represent prominent negative perceptions about vaccination, including beliefs that vaccines are potentially dangerous (Health Risks), expensive (Cost), painful to receive (Physical Pain), a hassle to receive (Inconvenience), potentially dangerous due to specific characteristics of the respondent (Personal Reactions), difficult to obtain (Access), not needed when in good health (Healthy), and difficult to remember (Forget).

In the current article, we explore the mediating effects of these dimensions in a relatively exploratory manner, simultaneously assessing whether all eight dimensions serve as explanatory mechanisms. Our theorizing, however, focused on the possibility that BIS and BAS sensitivities relate to a heightened attention toward the health risks of vaccines. Likewise, prior research has supported that Health Risks and Healthy are the two dominant dimensions within the MVHS, and they most often arise as predictors of outcomes and mediators of relations (Howard, 2022a, 2022b, 2022c; Howard and Davis, 2023). While we assess the possibility that any of the MVHS dimensions are mediators of our tested relations, we also recognize the possibility that Health Risks and perhaps Healthy may arise as the dominant mediators.

Further, we move beyond assessing the sole outcome of vaccine receipt in testing these effects, as other vaccine-related outcomes are valuable within themselves. Instead, we study the outcomes of flu and COVID-19 vaccination willingness; flu, COVID-19, and other vaccination receipt; as well as positive and negative vaccination word of mouth. By doing so, we differentiate whether BIS sensitivity, BAS sensitivity, and vaccine hesitancy produces differing relations with outcomes based on the type of vaccine (flu, COVID-19, and other) or outcome (willingness, receipt, and word-of-mouth). We also are able to test whether these variables relate to whether a person decides to receive a vaccine, but we likewise assess whether they relate to whether a person may influence others to receive a vaccine in studying word-of-mouth.

Together, our study provides several implications for research and practice. First, extant theory produces uncertainties regarding the relations of BIS and BAS sensitivities with vaccination outcomes. Our study provides clarity to these relations, resolving this tension in the current literature and potentially serving as a launching point for future research to investigate nuances of these relations. Second, authors have repeatedly advanced theory by incorporating the BIS and BAS into models and frameworks (Franken, 2002; Lerner et al., 2018; O’Connor et al., 2009), and these advancements can subsequently be applied to understand vaccination outcomes once an initial link is made in the current study. When studied in future research, authors may be able to explain more variance in vaccination outcomes than previously obtained. Third, by studying multiple outcomes, the current article provides more robust evidence for the importance of BIS and BAS sensitivities on vaccination than studying one outcome alone. Perhaps more importantly, the current article determines whether these sensitivities impact only one type of vaccination outcome or vaccination outcomes more broadly. Fourth, our results can lead to improved interventions focused on the constructs most strongly related to vaccination outcomes. Adaptive interventions, for instance, could be created that provide components based on participants’ standing on characteristics known to relate to vaccination outcomes, such as BIS or BAS sensitivities if supported in the current article (Collins et al., 2004; Wang and Miller, 2020). Fifth, we study these relations with a sample of older adults (age >55) due to their heightened risk of sickness from being unvaccinated (Childs et al., 2019). By studying this population, the current article produces implications with a greater practical impacts.

Before presenting our study, however, additional consideration should be given to our chosen measure of vaccine hesitancy. Bussink-Voorend et al. (2022) recently conducted a systematic literature review of vaccine hesitancy conceptualizations and operationalizations in 422 studies, finding a very wide array of definitions based on varying characteristics including cognition/affect, behaviors, and the vaccination decision making process. The authors further found that established instruments of vaccine hesitancy most often assess perceptions of vaccines (i.e. cognition), including the three most commonly applied measures: the parent attitudes about childhood vaccines survey (Opel et al., 2011), the vaccine hesitancy scale (Bussink-Voorend et al., 2022), and the SAGE instrument (Larson et al., 2015). At the same time, the authors called for greater attention in research on conceptualizations and operationalizations that consider vaccine hesitancy to solely be a state of indecision in the vaccination decision making process, which is often operationalized as middling responses on willingness measures.

These dynamics were considered by Howard (2022a) in creating the MVHS. The author recognized the varying definitions in research but utilized an operational definition consistent with extant widespread instruments to address other concerns with their measurement properties. By doing so, the MVHS reflects prominent perceptions about vaccines that relate to indecision, which were discovered and supported via a multiple study process. However, this scale is not the only measure of vaccine hesitancy, and it should be recognized that the current investigation does not address other conceptualizations and operationalizations. Therefore, the present results should be interpreted with this caveat in mind, and as discussed below, future researchers should replicate the current results with alternative measures for vaccine hesitancy.

Method

The current article is accompanied by the raw data generated during and/or analyzed during the study, including files detailing the analyses and either the complete database or other relevant raw data (Supplemental Material A). These files are available in the Figshare repository and accessible as Supplemental Material via the Sage Journals platform. Ethics approval, participant permissions, and all other relevant approvals were granted for this data sharing. The current research procedures were approved by the University of South Alabama IRB [1663301].

Participants

Participants (Age_M = 63.01, Age_SD = 6.43, 63% female, 92% white) were recruited from Prolific in return for monetary compensation. Prolific is an online platform connecting those needing tasks completed with those willing to complete those tasks. Prior studies have supported the validity of results obtained via Prolific when taking certain precautions (Kim et al., 2022; Palan and Schitter, 2018; Stanton et al., 2022), which were taken in the current article. We removed all participants that failed more than one of seven attention checks (e.g. “Please mark agree to show that you are paying attention.”), and we utilized a multi-wave research design. Our sample sizes reported below reflect the sample after removing those who failed our attention checks. Lastly, we restricted participation to only those aged 55 years or older.

Procedure

Participants enrolled via the Prolific platform and immediately completed the first survey that included measures of BIS and BAS sensitivity (n = 292). They completed a second survey 1 week later, which included our vaccine hesitancy measure (n = 265). One week after the second, participants completed a third survey that included our outcome measures (n = 241).

Measures

Due to word count constraints, all measures are reported in Supplemental Material B.

Results

Sociodemographic information separated by vaccination status is provided in Supplemental Material C. Pearson correlations and Cronbach’s alphas are provided in Table 1. Linear regression results of BIS and BAS sensitivity predicting vaccine hesitancy dimensions are provided in Table 2. Regression results of BIS and BAS sensitivities and vaccine hesitancy predicting outcomes are provided in Supplemental Material D. In conducting these regression analyses, we utilized linear regression when analyzing continuous outcomes (willingness and word-of-mouth), whereas we utilized binominal logistic regression when analyzing dichotomous outcomes (receipt). Lastly, we used Hayes’s (2017) PROCESS macro to assess indirect effects. Hayes’s PROCESS macro provides bootstrapped estimates that are among the most accurate approaches to assessing indirect effects for continuous and dichotomous outcomes (Hayes, 2017). When estimating our indirect effects, we analyzed the simultaneous indirect effects of all vaccine hesitancy dimensions. We also included BIS sensitivity as a covariate when assessing the indirect effects of BAS sensitivity (and vice versa), which allows us to test our entire model together. Results of these tests of indirect effects are provided in Table 3.

Table 1.

Correlations and Cronbach’s alphas.

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
1. BIS	0.90
2. BAS	0.06	0.88
3. HR	−0.02	0.19	0.93
4. Cost	0.05	0.05	0.24	0.84
5. Pain	0.19	0.07	0.23	0.32	0.90
6. Incov	0.05	0.07	0.39	0.42	0.31	0.93
7. PR	0.10	0.15	0.45	0.36	0.38	0.41	0.91
8. Acc	−0.01	0.02	0.21	0.24	0.23	0.48	0.36	0.89
9. Heal	−0.03	0.18	0.65	0.26	0.24	0.60	0.46	0.32	0.95
10. Forg	0.09	0.01	0.25	0.38	0.23	0.69	0.34	0.44	0.46	0.91
11. Flu W	0.09	−0.12	−0.60	−0.26	−0.22	−0.37	−0.48	−0.17	−0.71	−0.25	0.86
12. CW	0.11	−0.15	−0.72	−0.26	−0.20	−0.35	−0.46	−0.21	−0.72	−0.23	0.77	0.90
13. Flu R	0.02	−0.05	−0.42	−0.28	−0.22	−0.39	−0.42	−0.27	−0.58	−0.38	0.76	0.47	—
14. CR	0.02	−0.13	−0.62	−0.15	−0.15	−0.31	−0.40	−0.23	−0.62	−0.19	0.59	0.65	0.48	—
15. OR	−0.03	−0.00	−0.15	−0.09	−0.17	−0.22	−0.28	−0.09	−0.32	−0.17	0.29	0.21	0.27	0.20	—
16. PWoM	0.01	0.06	−0.49	−0.07	−0.18	−0.20	−0.29	−0.16	−0.55	−0.16	0.54	0.60	0.43	0.41	0.17	0.85
17. NWoM	−0.04	0.13	0.61	0.16	0.14	0.30	0.32	0.15	0.49	0.13	−0.40	−0.55	−0.33	−0.43	−0.13	−0.27	0.83

Cronbach’s alphas listed on diagonal. All correlations above 0.13 or below −0.13 are statistically significant at a 0.05 level. All correlations above 0.17 or below −0.17 are statistically significant at a 0.01 level.

BIS: behavioral inhibition system; BAS: behavioral activation system; HR: health risks; Cost: cost; Pain: physical pain; Incov: inconvenience; PR: personal reactions; Acc: accessibility; Heal: healthy; Forg: forget; Flu W: flu vaccination willingness; CW: COVID-19 vaccination willingness; Flu R: flu vaccination receipt; CR: COVID-19 vaccination receipt; OR: other vaccination receipt; PWoM: positive word of mouth; NWoM: negative word of mouth.

Table 2.

Regression results of BIS and BAS sensitivities predicting vaccine hesitancy dimensions.

	HR	Cost	Pain	Incov	PR	Acc	Healthy	Forget
1. BIS	−0.03	0.04	0.19**	0.05	0.10	−0.02	−0.04	0.09
2. BAS	0.19**	0.05	0.05	0.07	0.15*	0.02	0.18**	0.01
R ²	0.04**	0.01	0.04**	0.01	0.03*	0.00	0.04	0.01

p < 0.05. **p < 0.01.

Table 3.

Indirect effects of BIS and BAS sensitivities on vaccination outcomes via vaccine hesitancy dimensions.

	HR	Cost	Pain	Incov	PR	Acc	Healthy	Forget	Total
Flu vaccination willingness
1. BIS	−0.02, 0.06	−0.02, 0.01	−0.06, 0.02	−0.02, 0.02	−0.08, 0.01	−0.04, 0.03	−0.05, 0.19	−0.02, 0.02	−0.13, 0.20
2. BAS	−0.17, −0.01*	−0.03, 0.02	−0.03, 0.03	−0.02, 0.03	−0.15, 0.00	−0.04, −0.05	−0.38, −0.06*	−0.04, 0.02	−0.58, −0.11*
COVID-19 vaccination willingness
1. BIS	−0.04, 0.13	−0.02, 0.01	−0.05, 0.03	−0.02, 0.03	−0.05, 0.01	−0.02, 0.01	−0.04, 0.17	−0.01, 0.02	−0.10, 0.26
2. BAS	−0.33, −0.04*	−0.04, 0.02	−0.02, 0.02	−0.02, 0.05	−0.10, 0.00	−0.02, 0.02	−0.35, −0.05*	−0.03, 0.02	−0.65, −0.14*
Flu vaccination receipt
1. BIS	−0.02, 0.06	−0.04, 0.03	−0.15, 0.04	−0.05, 0.05	−0.14, 0.01	−0.04, 0.03	−0.07, 0.25	−0.12, 0.02	−0.31, 0.21
2. BAS	−0.15, 0.05	−0.05, 0.04	−0.07, 0.05	−0.05, 0.06	−0.24, 0.01	−0.05, 0.05	−0.49, −0.05*	−0.05, 0.17	−0.70, 0.01
COVID-19 vaccination receipt
1. BIS	−0.10, 0.34	−0.04, 0.07	−0.04, 0.20	−0.07, 0.06	−0.15, 0.02	−0.06, 0.07	−0.05, 0.20	−0.06, 0.08	−0.16, 0.58
2. BAS	−0.85, −0.09*	−0.06, 0.12	−0.07, 0.11	−0.09, 0.09	−0.28, 0.02	−0.13, 0.06	−0.46, −0.02*	−0.11, 0.09	−1.37, −0.19*
Other vaccination receipt
1. BIS	−0.10, 0.04	−0.05, 0.05	−0.15, 0.05	−0.07, 0.06	−0.12, 0.02	−0.04, 0.08	−0.06, 0.14	−0.05, 0.06	−0.23, 0.12
2. BAS	−0.06, 0.38	−0.05, 0.10	−0.09, 0.09	−0.17, 0.05	−0.32, 0.04	−0.04, 0.14	−0.52, −0.03*	−0.11, 0.07	−0.53, 0.08
Positive word-of-mouth
1. BIS	−0.02, 0.07	−0.01, 0.03	−0.06, 0.01	−0.03, 0.05	−0.03, 0.01	−0.01, 0.01	−0.04, 0.16	−0.02, 0.02	−0.07, 0.18
2. BAS	−0.19, −0.01*	−0.02, 0.05	−0.04, 0.02	−0.04, 0.07	−0.06, 0.02	−0.02, 0.02	−0.33, −0.05*	−0.02, 0.03	−0.44, −0.09*
Negative word-of-mouth
1. BIS	−0.09, 0.03	−0.01, 0.01	−0.02, 0.02	−0.01, 0.04	−0.01, 0.02	−0.01, 0.01	−0.04, 0.01	−0.02, 0.01	−0.12, 0.06
2. BAS	0.03, 0.22*	−.02, 0.01	−.01, 0.01	−0.02, 0.05	−0.03, 0.04	−0.02, 0.01	−0.01, 0.09	−0.02, 0.03	0.05, 0.29*

Indicates that 95% confidence interval excludes zero.

When analyzed alongside BAS sensitivity, BIS sensitivity did not significantly relate to any vaccine hesitancy dimensions or vaccination outcomes (all p > 0.05), and it also did not produce any significant indirect effects (all 95% C.I. include 0). BIS sensitivities were therefore unrelated to the dynamics of vaccination in the current sample.

When analyzed alongside BIS sensitivity, BAS sensitivity significantly related to the vaccine hesitancy dimensions of Health Risks (β = 0.19, 95% C.I. (0.07, 0.31), p < 0.01), Personal Reactions (β = 0.15, 95% C.I. (0.03, 0.27), p = 0.02), and Healthy (β = 0.18, 95% C.I. (0.06, 0.30), p < 0.01). BAS sensitivity also significantly related to flu vaccination willingness (β = −0.13, 95% C.I. (−0.26, −0.00), p = 0.047), COVID-19 vaccination willingness (β = −0.16, 95% C.I. (−0.28, −0.03), p = 0.02), COVID-19 vaccination receipt (O.R. = 0.66, 95% C.I. (0.44, 1.00), p = 0.048), and negative word-of-mouth (β = 0.14, 95% C.I. (0.01, 0.26), p = 0.04).

Further, BAS sensitivity produced several significant indirect effects on our outcomes via the vaccine hesitancy dimensions. Via Health Risks, BAS sensitivity had significant indirect effects on flu vaccination willingness (ab = −0.07, 95% C.I. (−0.17, −0.01)), COVID-19 vaccination willingness (ab = −0.17, 95% C.I. (−0.33, −0.04)), COVID-19 vaccination receipt (ab = −0.35, 95% C.I. (−0.85, −0.09)), positive word-of-mouth (ab = −0.09, 95% C.I. (−0.19, −0.01)), and negative word-of-mouth (ab = 0.12, 95% C.I. (0.03, 0.22)). Via Healthy, BAS sensitivity had significant indirect effects on flu vaccination willingness (ab = −0.21, 95% C.I. (−0.38, −0.06)), COVID-19 vaccination willingness (ab = −0.18, 95% C.I. (−0.35, −0.05)), flu vaccination receipt (ab = −0.22, 95% C.I. (−0.49, −0.05)), COVID-19 vaccination receipt (ab = −0.15, 95% C.I. (−0.46, −0.02)), other vaccination receipt (ab = −0.65, 95% C.I. (−1.11, −0.18)), and positive word-of-mouth (ab = −0.17, 95% C.I. (−0.33, −0.05)). No other dimensions produced significant indirect effects, but BAS sensitivity total indirect effects was significant with flu vaccination willingness (ab = −0.35, 95% C.I. (−0.58, −0.11)), COVID-19 vaccination willingness (ab = −0.40, 95% C.I. (−0.65, −0.14)), COVID-19 vaccination receipt (ab = −0.57, 95% C.I. (−1.37, −0.19)), positive word-of-mouth (ab = −0.25, 95% C.I. (−0.44, −0.09)), and negative word-of-mouth (ab = 0.16, 95% C.I. (0.05, 0.29)).

Lastly, in the final step of our regression analyses including BIS sensitivity, BAS sensitivity, and vaccine hesitancy dimensions predicting vaccination outcomes, Healthy significantly related to six vaccination outcomes; Health Risks significantly related to five, Personal Reactions significantly related to three; Accessibility, Inconvenience, and Forget significantly related to one each (all p < 0.05); and Cost and Physical Pain did not significantly relate to any (all p > 0.05). Therefore, the vaccine hesitancy dimensions of Health Risks and Healthy were again the dominant mediator of relations and predictor of vaccination outcomes.

Discussion

Our results supported that BIS sensitivity does not produce consistent relations with vaccine hesitancy or vaccination outcomes, but BAS sensitivity produced significant negative relations with flu vaccination willingness, COVID-19 vaccination willingness, COVID-19 vaccine receipt. BAS sensitivity also produced a significant positive relation with negative word-of-mouth. These significant relations were mediated by the vaccine hesitancy dimensions of Health Risks and Healthy. Together, these results indicate that BAS sensitivity has more predominant relations with vaccine hesitancy and vaccination outcomes than BIS sensitivity, providing clarity to relations that were otherwise uncertain with modern theory.

Theoretical and practical implications

The present results produce many implications for research and practice. First, BAS sensitivity related to detrimental vaccination outcomes, suggesting that some people may see refraining from vaccination as avoiding a negative outcome (nonpunishment). This finding may be due to the ubiquity of vaccination in the United States (Savulescu, 2021; Smith et al., 2021; Vanderslott and Marks, 2021). Vaccination may be considered a normative behavior for older adults, as most participants in our sample received the flu vaccine, COVID-19 vaccine, and were up to date on their other vaccines. For this reason, certain people may perceive refraining from vaccination as avoiding an impending punishment, causing BAS sensitivity to relate to vaccine hesitancy and detrimental vaccination outcomes. Future research should further investigate this notion. The current article was a relatively exploratory investigation, and more focused studies should determine whether refraining from vaccination is perceived as a nonpunishment by some people. Notably, BIS and BAS sensitivities can be assessed by more sophisticated approaches, such as spatial orienting tasks (van Hemel-Ruiter et al., 2013). These tasks assess the extent that people fix their attention to cues associated with reward and punishment, providing an implicit measurement of BIS and BAS sensitivities. This approach alleviates concerns with participant reactivity, and our relations should be reinvestigated with this measurement approach.

Second, BAS sensitivity did not relate to beneficial vaccination outcomes, indicating that people may not particularly see vaccination as avoiding a negative outcome (nonpunishment). Likewise, BIS sensitivity did not significantly relate to vaccine hesitancy or vaccination outcomes altogether, suggesting that people may not outright view refraining from vaccination or vaccination itself as a negative outcome (punishment). This may again be because vaccination is seen as the status quo. Researchers should compare our results with other preventive behaviors. For instance, face mask wearing is a preventive behavior surrounded by misinformation, whereas less misinformation targeted hand washing during the COVID-19 pandemic (Ayers et al., 2021; Hornik et al., 2021). The present results may be similar when studied with face mask wearing but not hand washing due to their differing associations with misinformation; however, the present results may also be similar when studied with hand washing but not face mask wearing because the former may be seen as a more obligatory behavior. By making these comparisons, future researchers could provide greater insights into the causes of our observed relations.

Third, our results produce novel insights into prior findings. Research has supported that loss-framed messages are more effective at promoting vaccination than gain-framed messages, contrary to the prevention-detection framework (Hong and Hashimoto, 2023; Liu et al., 2019; Nan, 2012). This may be because the BIS is relatively neutral regarding its association with vaccination, and loss-framed messages may be able to trigger the BIS in reacting to potential punishments. On the other hand, the BAS related to detrimental vaccination outcomes in the current study, and it may be more difficult to trigger the BAS to promote vaccination. That is, the BAS may be more resistant to producing favorable vaccination outcomes, such as by emphasizing potential rewards. This rationale corresponds with Nan (2012), who demonstrated that individuals with greater avoidance orientations were more influenced by loss-framed than gain-framed messages, whereas individuals with greater approach orientations did not have a difference in their receptivity to messages. To more fully integrate these findings and understand the association of vaccination with (non)reward and (non)punishment, future research should reassess the influence of message framing while accounting for BIS and BAS sensitivities.

Fourth, the current study can also lead to novel theoretical integrations. Espinoza Oyarce et al. (2021) recently integrated classical and revised research on reinforcement sensitivity theory, drawing attention to the third cognitive system originally proposed by the theory, the flight-flight-freezing system, that is believed to be relatively orthogonal to the other two. The flight-flight-freezing system may prove to be important in predicting vaccination outcomes alongside the BAS system, as its orthogonality may capture variance beyond the other two dimensions. Likewise, the BIS and BAS have been argued to be the cause of personality emergence, and personality traits, including those of the Big Five, can be associated with a respective system (Beck et al., 2009; Caseras et al., 2003; Muris et al., 2005). Vaccine hesitancy and vaccination outcomes have been linked to the Big Five and Dark Triad (Howard, 2022c; Li and Cao, 2022), but reinterpreting these results via the lens of the BIS and BAS may produce insights into the relation of individual differences and vaccination outcomes.

Fifth, while significant support has been provided for the MVHS, it was only recently created (Howard, 2022a). The current article provided further evidence that the dimensions appropriately relate to vaccination outcomes, but we also replicated prior findings in supporting that Health Risks and Healthy were the two dominant explanatory mechanisms (Balgiu et al., 2022; Howard, 2022a, 2022b, 2022c; Howard and Davis, 2023). Future researchers should develop more specific hypotheses involving these two dimensions. BIS and BAS theory has a relevance due to the inherent association with punishments and nonreward, and many of the proposed theoretical integrations above can also be applied to directly study Health Risks and Healthy. Beyond this theoretical perspective, a host of communication theories propose specific processes when facing uncertainty. For instance, uncertainty management theory suggests that people often undergo processes to reduce uncertainty, but they sometimes undergo processes to utilize uncertainty in self-protective manners when it cannot be easily reduced (Lind and Van Den Bos, 2002). Several studies have supported that people engage in information seeking behaviors when vaccine hesitant, perhaps to manage their uncertainty regarding vaccination. Future research should apply these theories to determine whether people engage in these processes when they specifically face uncertainty about the risks (Health Risks) or benefits (Healthy) of vaccination, which may uncover the reasons that these two dimensions are dominant predictors.

Sixth, the present study is the first to administer the MVHS to a sample of older adults, who are particularly at risk for illness (Childs et al., 2019). Our results can provide greater real-world impacts by providing insights into why this population may choose to not receive vaccines. The current article also provided psychometric and validity support for the MVHS with older adults, and future studies can have more assurances in the reliability of their results when administering the MVHS to older adults. Future studies should likewise identify important populations for which vaccine hesitancy may be particularly important to understand, and the MVHS should be investigated in these samples. For instance, understanding vaccine hesitancy in low-receipt countries may be particularly important, and the psychometric properties and validity of the MVHS should be supported with samples from these populations.

Seventh, the current results can aid in the development of interventions, particularly adaptive interventions. Adaptive interventions provide components to participants based on their characteristics (Collins et al., 2004; Wang and Miller, 2020). Participants could, for instance, initially complete measures of the BAS, Health Risks, and Healthy. Participants who score high could then be provided intervention components specifically targeting these constructs, such as information about the minimal health risks of vaccines or the need to receive vaccines even when healthy. Or, in the case of BAS, they could be provided information that minimizes the nonpunishment of refraining from vaccines. By doing so, not only could the intervention be more effective than more general traditional interventions, but it could also conserve time and resources by only providing specific intervention components to targeted participants.

Limitations

We utilized a supported measure of vaccine hesitancy, but other operationalizations exist (Larson et al., 2015; Opel et al., 2011). Researchers should replicate the current results utilizing these other measures. An alternative conceptualization of vaccine hesitancy is solely indecision in the vaccination decision making process, often represented by a range of middling values on measures of willingness (Bussink-Voorend et al., 2022). Researchers should test whether this alternative operationalization produces similar relations with BIS and BAS sensitivities, which could provide insights into the differences of modern conceptualizations and operationalizations of vaccine hesitancy—and how those differences produce different relations for constructs with the same label.

The present results should be replicated with similar constructs and frameworks. Perhaps the most relevant is the 3C, 5C, and 7C models (Betsch et al., 2018; Geiger et al., 2022). These models build upon each other, and they describe varying perceptions regarding vaccines. By showing that the constructs within these models produce similar relations with BIS and BAS sensitivities, researchers can further support that models and frameworks associated with BIS and BAS may be key to understanding vaccination perceptions and behaviors.

We utilized an online data collection platform to obtain our participant responses. While we applied modern best practice in utilizing these types of sources (Kim et al., 2022; Palan and Schitter, 2018; Stanton et al., 2022), our sample may be overly representative of certain types of participants more likely to use these platforms. Authors have suggested that those with higher a higher socioeconomic status and education may be more likely to be recruited from these platforms. For this reason, researchers should replicate the current results with more representative sampling methods, such that all stratum of the general population can be obtained.

We also utilized a time-separated research design. While this design can alleviate some concerns regarding common-method bias, it can only provide modest support for causal effects. We recommend that researchers should replicate the current results with more robust research designs. We recommend the use of panel designs, wherein each variable is measured at each measurement occasion. This design can provide robust support for casual and reverse-causal effects, thereby ensuring the directionality of any hypothesized relations.

Supplemental Material

sj-docx-1-hpq-10.1177_13591053231197645 – Supplemental material for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults

Supplemental material, sj-docx-1-hpq-10.1177_13591053231197645 for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults by Matt C Howard in Journal of Health Psychology

Supplemental Material

sj-docx-2-hpq-10.1177_13591053231197645 – Supplemental material for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults

Supplemental material, sj-docx-2-hpq-10.1177_13591053231197645 for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults by Matt C Howard in Journal of Health Psychology

Supplemental Material

sj-docx-3-hpq-10.1177_13591053231197645 – Supplemental material for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults

Supplemental material, sj-docx-3-hpq-10.1177_13591053231197645 for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults by Matt C Howard in Journal of Health Psychology

Supplemental Material

sj-docx-4-hpq-10.1177_13591053231197645 – Supplemental material for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults

Supplemental material, sj-docx-4-hpq-10.1177_13591053231197645 for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults by Matt C Howard in Journal of Health Psychology

Supplemental Material

sj-docx-5-hpq-10.1177_13591053231197645 – Supplemental material for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults

Supplemental material, sj-docx-5-hpq-10.1177_13591053231197645 for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults by Matt C Howard in Journal of Health Psychology

Supplemental Material

sj-omv-6-hpq-10.1177_13591053231197645 – Supplemental material for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults

Supplemental material, sj-omv-6-hpq-10.1177_13591053231197645 for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults by Matt C Howard in Journal of Health Psychology

Supplemental Material

sj-spv-7-hpq-10.1177_13591053231197645 – Supplemental material for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults

Supplemental material, sj-spv-7-hpq-10.1177_13591053231197645 for Relations of behavioral inhibition and activation system sensitivities with vaccination outcomes: Investigating the mediating role of vaccine hesitancy in older adults by Matt C Howard in Journal of Health Psychology

Footnotes

Data sharing statement

The current article is accompanied by the relevant raw data generated during and/or analysed during the study, including files detailing the analyses and either the complete database or other relevant raw data. These files are available in the Figshare repository and accessible as Supplemental Material via the Sage Journals platform. Ethics approval, participant permissions, and all other relevant approvals were granted for this data sharing.

Declaration of conflicting interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

Ethics approval

The current research procedures were approved by the University of South Alabama IRB [1663301].

Informed consent

Participants were provided an information sheet to maximize confidentiality and anonymity.

ORCID iD

Matt C Howard

References

Akel

Masters

Shih

, et al (2021) Modification of a vaccine hesitancy scale for use in adult vaccinations in the United States and China. Human Vaccines & Immunotherapeutics 17(8): 2639–2646.

Amodio

Master

Yee

, et al (2008) Neurocognitive components of the behavioral inhibition and activation systems: Implications for theories of self-regulation. Psychophysiology 45(1): 11–19.

Anderson

Perone

(2023) Predicting individual differences in behavioral activation and behavioral inhibition from functional networks in the resting EEG. Biological Psychology 177: 108483.

Ayers

Chu

Zhu

, et al (2021) Spread of misinformation about face masks and COVID-19 by automated software on Facebook. JAMA Internal Medicine 181(9): 1251–1253.

Balgiu

Sfeatcu

Țâncu

, et al (2022) The Multidimensional Vaccine Hesitancy Scale: A validation study. Vaccines 10(10): 1755.

Basch

Meleo-Erwin

Fera

, et al (2021) A global pandemic in the time of viral memes: COVID-19 vaccine misinformation and disinformation on TikTok. Human Vaccines & Immunotherapeutics 17(8): 2373–2377.

Beck

Smits

DJM

Claes

, et al (2009) Psychometric evaluation of the behavioral inhibition/behavioral activation system scales and the sensitivity to punishment and sensitivity to reward questionnaire in a sample of eating disordered patients. Personality and Individual Differences 47(5): 407–412.

Betsch

Schmid

Heinemeier

, et al (2018) Beyond confidence: Development of a measure assessing the 5C psychological antecedents of vaccination. PLoS One 13(12): e0208601.

Bierwiaczonek

Gundersen

Kunst

(2022) The role of conspiracy beliefs for COVID-19 health responses: A meta-analysis. Current Opinion in Psychology 46: 101346.

10.

Bijttebier

Beck

Claes

, et al (2009) Gray’s reinforcement sensitivity theory as a framework for research on personality–psychopathology associations. Clinical Psychology Review 29(5): 421–430.

11.

Bryan

Kyron

Page

(2022) BIS sensitivity, BAS sensitivity, and recent suicide attempts. Personality and Individual Differences 191: 111552.

12.

Bussink-Voorend

Hautvast

JLA

Vandeberg

, et al (2022) A systematic literature review to clarify the concept of vaccine hesitancy. Nature Human Behaviour 6: 1634–1648.

13.

Cartmell

Mzik

Sundstrom

, et al (2019) HPV vaccination communication messages, messengers, and messaging strategies. Journal of Cancer Education 34: 1014–1023.

14.

Carver

Scheier

(2001) On the Self-Regulation of Behavior. Cambridge: Cambridge University Press.

15.

Carver

C. S.

White

T. L.

(1994). Behavioral inhibition, behavioral activation, and affective responses to impending reward and punishment: the BIS/BAS scales. Journal of Personality and Social Psychology, 67(2), 319.

16.

Caseras

Àvila

Torrubia

(2003) The measurement of individual differences in behavioural inhibition and behavioural activation systems: A comparison of personality scales. Personality and Individual Differences 34(6): 999–1013.

17.

Childs

Zullo

Joyce

, et al (2019) The burden of respiratory infections among older adults in long-term care: A systematic review. BMC Geriatrics 19(1): 210.

18.

Coan

Allen

(2003) Frontal EEG asymmetry and the behavioral activation and inhibition systems. Psychophysiology 40(1): 106–114.

19.

Collins

Murphy

Bierman

(2004) A conceptual framework for adaptive preventive interventions. Prevention Science 5: 185–196.

20.

Dudley

Privor-Dumm

Dubé

, et al (2020) Words matter: Vaccine hesitancy, vaccine demand, vaccine confidence, herd immunity and mandatory vaccination. Vaccine 38(4): 709–711.

21.

Espinoza Oyarce

Burns

Butterworth

, et al. (2021) Bridging classical and revised reinforcement sensitivity theory research: A longitudinal analysis of a large population study. Frontiers in Psychology 12: 737117.

22.

Franken

IHA

(2002) Behavioral approach system (BAS) sensitivity predicts alcohol craving. Personality and Individual Differences 32(2): 349–355.

23.

Gable

Reis

Elliot

(2000) Behavioral activation and inhibition in everyday life. Journal of Personality and Social Psychology 78(6): 1135–1149.

24.

Geiger

Rees

Lilleholt

, et al (2022) Measuring the 7Cs of vaccination readiness. European Journal of Psychological Assessment 38: 261–269.

25.

Hayes

(2017) Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Approach. New York, NY: Guilford Publications.

26.

Heponiemi

Keltikangas-Järvinen

Puttonen

, et al (2003) BIS/BAS sensitivity and self-rated affects during experimentally induced stress. Personality and Individual Differences 34(6): 943–957.

27.

Heubeck

Wilkinson

Cologon

(1998) A second look at Carver and White’s (1994) BIS/BAS scales. Personality and Individual Differences 25(4): 785–800.

28.

Hong

Hashimoto

(2023) I will get myself vaccinated for others: The interplay of message frame, reference point, and perceived risk on intention for COVID-19 vaccine. Health Communication 38: 813–823.

29.

Hornik

Kikut

Jesch

, et al (2021) Association of COVID-19 misinformation with face mask wearing and social distancing in a nationally representative US sample. Health Communication 36(1): 6–14.

30.

Howard

(2022a) A more comprehensive measure of vaccine hesitancy: Creation of the Multidimensional Vaccine Hesitancy Scale (MVHS). Journal of Health Psychology 27(10): 2402–2419.

31.

Howard

(2022b) Investigating the relation of political orientation and vaccination outcomes: Identifying the roles of political ideology, party affiliation, and vaccine hesitancy. Psychological Reports. Epub ahead of print 7 December 2022. DOI: 10.1177/00332941221144604.

32.

Howard

(2022c) The good, the bad, and the neutral: Vaccine hesitancy mediates the relations of Psychological Capital, the Dark Triad, and the Big Five with vaccination willingness and behaviors. Personality and Individual Differences 190: 111523.

33.

Howard

Davis

(2023) The mediating role of vaccine hesitancy in the relations of COVID-19 conspiracy beliefs and vaccination outcomes: Which dimensions matter? Journal of Health Psychology 28: 241–250.

34.

James

Bokemper

Gerber

, et al (2021) Persuasive messaging to increase COVID-19 vaccine uptake intentions. Vaccine 39(49): 7158–7165.

35.

Jorm

Christensen

Henderson

, et al (1998) Using the BIS/BAS scales to measure behavioural inhibition and behavioural activation: Factor structure, validity and norms in a large community sample. Personality and Individual Differences 26(1): 49–58.

36.

Kim

Fiore

AMM

Chung

, et al (2022) Overcoming online survey data quality challenges in using Qualtrics, Amazon Mechanical Turk, and prolific. International Textile and Apparel Association Annual Conference Proceedings 78(1): 1–2.

37.

Koy

Yeo

(2008) BIS sensitivity, negative affect and performance: Dynamic and multilevel relationships. Human Performance 21(2): 198–225.

38.

Larson

Jarrett

Schulz

, et al (2015) Measuring vaccine hesitancy: The development of a survey tool. Vaccine 33(34): 4165–4175.

39.

Lawes-Wickwar

Ghio

Tang

, et al (2021) A rapid systematic review of public responses to health messages encouraging vaccination against infectious diseases in a pandemic or epidemic. Vaccines 9(2): 72.

40.

Lerner

Hatak

Rauch

(2018) Deep roots? Behavioral inhibition and behavioral activation system (BIS/BAS) sensitivity and entrepreneurship. Journal of Business Venturing Insights 9: 107–115.

41.

Cao

(2022) Your pain, my gain: The relationship between self-report and behavioral measures of everyday sadism and COVID-19 vaccination intention. Current Psychology. 42: 20149–20156.

42.

Lind

Van Den Bos

(2002) When fairness works: Toward a general theory of uncertainty management. Research in Organizational Behavior 24: 181–223.

43.

Liu

Yang

Chu

(2019) Now or future? Analyzing the effects of message frame and format in motivating Chinese females to get HPV vaccines for their children. Patient Education and Counseling 102(1): 61–67.

44.

Mardaga

Hansenne

(2007) Relationships between Cloninger’s biosocial model of personality and the behavioral inhibition/approach systems (BIS/BAS). Personality and Individual Differences 42(4): 715–722.

45.

Muris

Meesters

de Kanter

, et al (2005) Behavioural inhibition and behavioural activation system scales for children: Relationships with Eysenck’s personality traits and psychopathological symptoms. Personality and Individual Differences 38(4): 831–841.

46.

Nan

(2012) Relative persuasiveness of gain- versus loss-framed human Papillomavirus vaccination messages for the present- and future-minded. Human Communication Research 38(1): 72–94.

47.

O’Connor

Stewart

Watt

(2009) Distinguishing BAS risk for university students’ drinking, smoking, and gambling behaviors. Personality and Individual Differences 46(4): 514–519.

48.

Opel

Mangione-Smith

Taylor

, et al (2011) Development of a survey to identify vaccine-hesitant parents: The parent attitudes about childhood vaccines survey. Human Vaccines 7(4): 419–425.

49.

Palan

Schitter

(2018) Prolific.ac—A subject pool for online experiments. Journal of Behavioral and Experimental Finance 17: 22–27.

50.

Renn

Steinbauer

Fenner

(2014) Employee behavioral activation and behavioral inhibition systems, manager ratings of employee job performance, and employee withdrawal. Human Performance 27(4): 347–371.

51.

Savulescu

(2021) Good reasons to vaccinate: Mandatory or payment for risk? Journal of Medical Ethics 47(2): 78–85.

52.

Shapiro

Tatar

Dube

, et al (2018) The vaccine hesitancy scale: Psychometric properties and validation. Vaccine 36(5): 660–667.

53.

Smith

Hodson

Rubin

(2021) Parental attitudes towards mandatory vaccination; A systematic review. Vaccine 39(30): 4046–4053.

54.

Stanton

Carpenter

Nance

, et al (2022) A multisample demonstration of using the prolific platform for repeated assessment and psychometric substance use research. Experimental and Clinical Psychopharmacology 30: 432–443.

55.

Vanderslott

Marks

(2021) Charting mandatory childhood vaccination policies worldwide. Vaccine 39(30): 4054–4062.

56.

van Hemel-Ruiter

de Jong

Oldehinkel

, et al (2013) Reward-related attentional biases and adolescent substance use: The TRAILS study. Psychology of Addictive Behaviors 27(1): 142–150.

57.

Wang

Miller

(2020) Just-in-the-moment adaptive interventions (JITAI): A meta-analytical review. Health Communication 35(12): 1531–1544.

58.

Wardle

Singerman

(2021) Too little, too late: Social media companies’ failure to tackle vaccine misinformation poses a real threat. BMJ 372: n26.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

0.02 MB

0.03 MB

0.01 MB

0.15 MB

1.06 MB