The item wording (or keying) effect is respondents’ differential response style to positively and negatively worded items. Despite decades of research, the nature of the effect is still unclear. This article proposes a potential reason; namely, that the item wording effect is scale-specific, and thus findings are applicable only to a particular measure involved in an investigation. Using multiple scales and several methods, the present study provides strong and converging support for this hypothesis. In light of the results, we reinterpret major findings of the item wording effect and propose possible future directions for research.
AlessandriG.VecchioneM.TisakJ.BarbaranelliC.. 2011. “Investigating the Nature of Method Factors through Multiple Informants: Evidence for a Specific Factor?” Multivariate Behavioral Research46:625–42.
2.
BidermanM. D.NguyenN. T.CunninghamC. J. L.ChenZ.WatsonP. J.. 2013. “Method Factors, Bifactors, and Item Valence.” Paper presented at the 28th annual conference of The Society for Industrial and Organizational Psychology, Houston, TX.
3.
BidermanM. D.NguyenN. T.CunninghamJ. L.GhorbaniM.. 2011. “The Ubiquity of Common Method Variance: The Case of the Big Five.” Journal of Research in Personality45:417–29.
4.
CarminesE. G.ZellerR. A.. 1979. Reliability and Validity Assessment. Beverly Hills, CA: Sage.
5.
Castro-SchiloL.WidamanK. F.GrimmK. J.. 2013. “Neglect the Structure of Multitrait-multimethod Data at Your Peril: Implications for Associations with External Variables.” Structural Equation Modeling20:181–207.
6.
ChangE. C.Maydeu-OlivaresA.D’ZurillaT. J.. 1997. “Optimism and Pessimism as Partially Independent Constructs: Relationship to Positive and Negative Affectivity and Psychological Well-being.” Personality and Individual Differences23:433–40.
7.
ChenF. F.2007. “Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance.” Structural Equation Modeling14:464–504.
8.
CredéM.ChernyshenkoO. S.BagraimJ.SullyM.. 2009. “Contextual Performance and Job Satisfaction-dissatisfaction Distinction: Examining Artifacts and Utility.” Human Performance22:246–72.
9.
CronbachL. J.MeehlP. E.. 1955. “Construct Validity in Psychological Tests.” Psychological Bulletin52:281–302.
10.
DiStefanoC.MotlR. W.. 2006. “Further Investigating Method Effects Associated with Negatively Worded Items on Self-report Surveys.” Structural Equation Modeling13:440–64.
11.
DiStefanoC.MotlR. W.. 2009. “Self-esteem and Method Effects Associated with Negatively Worded Items: Investigating Factorial Invariance by Sex.” Structural Equation Modeling16:134–46.
12.
EidM.2000. “A Multitrait-multimethod Model with Minimal Assumptions.” Psychometrika65:241–61.
13.
GeiserC.KochT.EidM.. 2014. “Data-generating Mechanisms versus Constructively Defined Latent Variables in Multitrait-multimethod Analysis: A Comment on Castro-Schilo, Widaman, and Grimm (2013).” Structural Equation Modeling21:509–23.
14.
GoldbergL. R.JohnsonJ. A.EberH. W.HoganR.AshtonM. C.CloningerC. R.GoughH. C.. 2006. “The International Personality Item Pool and the Future of Public-domain Personality Measures.” Journal of Research in Personality40:84–96.
15.
HerzbergP. Y.GlaesmerH.HoyerJ.. 2006. “Separating Optimism and Pessimism: A Robust Psychometric Analysis of the Revised Life Orientation Test (LOT-R).” Psychological Assessment18:433–38.
16.
HoranP. M.DiStefanoC.MotlR. W.. 2003. “Wording Effects in Self-esteem Scales: Methodological Artifact or Response Style?” Structural Equation Modeling10:444–55.
17.
KamC.2013. “Probing Item Social Desirability by Correlating Personality Items with Balanced Inventory of Desirable Responding (BIDR): A Validity Examination.” Personality and Individual Differences54:513–18.
18.
KamC.MeyerJ. P.. 2012. “Do Optimism and Pessimism Have Different Relationships with Personality Dimensions? A Re-examination.” Personality and Individual Differences52:123–27.
19.
KamC. C. S.MeyerJ. P.. 2015. “Implications of Item Keying and Item Valence for the Investigation of Construct Dimensionality.” Multivariate Behavioral Research50:457–69.
20.
LanceC. E.NobleC. L.ScullenS. E.. 2002. “A Critique of the Correlated Trait–Correlated Method (CTCM) and Correlated Uniqueness (CU) Models for Multitrait-multimethod (MTMM) Data.” Psychological Methods7:228–44.
21.
LiA.BaggerJ.. 2007. “The Balanced Inventory of Desirable Responding (BIDR): A Reliability Generalization Study.” Educational and Psychological Measurement67:525–44.
22.
LindwallM.BarkoukisV.GranoC.LucidiF.RaudseppL.LiukkonenJ.Thøgersen-NtoumaniC.. 2012. “Method Effects: The Problem with Negatively versus Positively Keyed Items.” Journal of Personality Assessment94:196–204.
23.
MagazineS. L.WilliamsL. J.WilliamsM. L.. 1996. “A Confirmatory Factor Analysis Examination of Reverse Coding Effects in Meyer and Allen’s Affective and Continuance Commitment Scales.” Educational and Psychological Measurement56:241–50.
24.
MarshH. W.1989. “Confirmatory Factor Analyses of Multitrait–multimethod Data: Many Problems and a Few Solutions.” Applied Psychological Measurement13:335–61.
25.
MarshH. W.1996. “Positive and Negative Self-esteem: A Substantively Meaningful Distinction or Artifactors?” Journal of Personality and Social Psychology70:810–19.
26.
MarshH. W.ScalasL. F.NagengastB.. 2010. “Longitudinal Tests of Competing Factor Structures for the Rosenberg Self-esteem Scale: Traits, Ephemeral Artifacts, and Stable Response Styles.” Psychological Assessment22:366–81.
27.
MarshallG. N.WortmanC. B.KusulasJ. W.HervigL. K.VickersR. R.Jr. 1992. “Distinguishing Optimism from Pessimism: Relations to Fundamental Dimensions of Mood and Personality.” Journal of Personality and Social Psychology62:1067–74.
28.
MotlR. W.DiStefanoC.. 2002. “Longitudinal Invariance of Self-esteem and Method Effects Associated with Negatively Worded Items.” Structural Equation Modeling9:562–78.
29.
O’NeillT. A.AllenN. J.. 2012. “Team Meeting Attitudes: Conceptualization and Investigation of a New Construct.” Small Group Research43:186–210.
30.
PaulhusD. L.1991. “Measurement and Control of Response Bias.” Pp. 17–59 in Measures of Personality and Social Psychological Attitudes, edited by RobinsonJ. P.ShaverP. R.WrightsmanL. S.. San Diego, CA: Academic Press.
31.
PeabodyD.1967. “Trait Inferences: Evaluative and Descriptive Aspects.” Journal of Personality and Social Psychology7(4, pt. 2):1–18.
32.
PlominR.ScheierM. F.BergemanC. S.PedersenN. L.NesselroadeJ. R.McClearnG. E.. 1992. “Optimism, Pessimism and Mental Health: A Twin/Adoption Analysis.” Personality and Individual Differences13:921–30.
33.
PohlS.SteyerR.. 2010. “Modeling Common Traits and Method Effects in Multitrait-multimethod Analysis.” Multivariate Behavioral Research45:45–72.
34.
PohlS.SteyerR.KrausK.. 2008. “Modelling Method Effects as Individual Causal Effects.” Journal of Royal Statistical Society: Series A (Statistics in Society)171:41–63.
35.
PrattoF.SidaniusJ.StallworthL. M.MalleB. F.. 1994. “Social Dominance Orientation: A Personality Variable Predicting Social and Political Attitudes.” Journal of Personality and Social Psychology67:741–63.
36.
QuiltyL. C.OakmanJ. M.RiskoE.. 2006. “Correlates of the Rosenberg Self-esteem Scale Method Effects.” Structural Equation Modeling13:99–117.
37.
RauchW. A.SchweizerK.MoosbruggerH.. 2007. “Method Effects Due to Social Desirability as a Parsimonious Explanation of the Deviation from Unidimensionality in LOT-R Scores.” Personality and Individual Differences42:1597–607.
38.
R Development Core Team. 2013. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. Accessed December 1, 2013 (http://www.R-project.org).
39.
Robinson-WhelenS.KimC.MacCallumR. C.Kiecolt-GlaserJ. K.. 1997. “Distinguishing Optimism from Pessimism in Older Adults: Is It More Important to Be Optimistic or Not to Be Pessimistic?” Journal of Personality and Social Psychology73:1345.
40.
RosseelY. 2012. “lavaan: An R Package for Structural Equation Modeling.” Journal of Statistical Software48:1–36.
41.
SatorraA.2000. “Scaled and Adjusted Restricted Tests in Multi-sample Analysis of Moment Structures.” Pp. 233–47 in Innovations in Multivariate Statistical Analysis: A Festschrift for Heinz Neudecker, edited by HeijmansR. D. H.PollockD. S. G.SatorraA.. London, UK: Kluwer Academic.
42.
StöberJ.DetteD. E.MuschJ.. 2002. “Comparing Continuous and Dichotomous Scoring of the Balanced Inventory of Desirable Responding.” Journal of Personality Assessment78:370–89.
43.
SwainS. D.WeathersD.NiedrichR. W.. 2008. “Assessing Three Sources of Misresponse to Reversed Likert Items.” Journal of Marketing Research45:116–31.
44.
TomásJ. M.OliverA.. 1999. “Rosenberg’s Self-esteem Scale: Two Factors or Method Effects.” Structural Equation Modeling6:84–98.
45.
TomásJ. M.OliverA.GalianaL.SanchoP.LilaM.. 2013. “Explaining Method Effects Associated with Negatively Worded Items in Trait and State Global and Domain-specific Self-esteem Scales.” Structural Equation Modeling20:299–313.
46.
UzielL.2010. “Rethinking Social Desirability Scales from Impression Management to Interpersonally Oriented Self-control.” Perspectives on Psychological Science5:243–62.
47.
VaggP. R.SpielbergerC. D.O’HearnT. P. J.. 1980. “Is the State-trait Anxiety Inventory Multidimensional?” Personality and Individual Differences1:207–14.
48.
van SonderenE.SandermanR.CoyneJ. C.. 2013. “Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain.” PLoS One31:e68967. doi: 10.1371/journal.pone.0068967.
49.
VautierS.CallahanS.MoncanyD.SztulmanH.. 2004. “A Bistable View of Single Constructs Measured Using Balanced Questionnaires: Application to Trait Anxiety.” Structural Equation Modeling11:261–71.
50.
VecchioneM.AlessandriG.CapraraG. V.TisakJ.. 2014. “Are Method Effects Permanent or Ephemeral in Nature? The Case of the Revised Life Orientation Test.” Structural Equation Modeling: A Multidisciplinary Journal21:117–30.
