Researchers often assume a strong, linear relationship between regular- and reverse-keyed items, with responses on regular-keyed items (e.g., agree) perfectly mirroring those on reverse-keyed items (e.g., disagree). The current research challenges this received view and propounds a possible nonlinear relationship, partly due to the logical tendency of midlevel respondents to disagree with both types of items. In four examples (reported human height, job satisfaction, positive-negative affect, and self-esteem; total N = 50,544), a nonlinear model consistently explained additional item variance beyond a linear model. We further demonstrate that this relationship is moderated by item characteristics such as item extremity (job satisfaction) and item softening (self-esteem). Suboptimal modeling of the relationship may result in the apparent bidmensionality of a construct that characterizes regular- and reverse-keyed items as separate factors. User-friendly syntax for the examination of nonlinearity is provided to enhance the accessibility of the procedure.
AltemeyerR. A.AltemeyerB. (1981). Right-wing authoritarianism. University of Manitoba press.
2.
BäckströmM.BjörklundF.LarssonM. R. (2009). Five-factor inventories have a major general factor related to social desirability which can be reduced by framing items neutrally. Journal of research in personality, 43(3), 335-344. https://doi.org/10.1016/j.jrp.2008.12.013
3.
BartholomewK. J.NtoumanisN.RyanR. M.BoschJ. A.Thøgersen-NtoumaniC. (2011). Self-determination theory and diminished functioning: The role of interpersonal control and psychological need thwarting. Personality and Social Psychology Bulletin, 37(11), 1459–1473.
4.
BriefA. (1998). Attitudes in and around organizations. SAGE.
5.
Castro-SchiloL.GrimmK. J.WidamanK. F. (2016). Augmenting the correlated trait–correlated method model for multitrait–multimethod data. Structural Equation Modeling, 23(6), 798–818.
6.
ClevelandW. S. (1979). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74(368), 829–836.
7.
ColeD. A.CieslaJ. A.SteigerJ. H. (2007). The insidious effects of failing to include design-driven correlated residuals in latent-variable covariance structure analysis. Psychological methods, 12(4), 381. https://doi.org/10.1037/1082-989X.12.4.381
8.
CredéM.ChernyshenkoO. S.BagraimJ.SullyM. (2009). Contextual performance and the job satisfaction–dissatisfaction distinction: Examining artifacts and utility. Human Performance, 22(3), 246–272.
9.
DienerE.WirtzD.TovW.Kim-PrietoC.ChoiD. W.OishiS.Biswas-DienerR. (2010). New well-being measures: Short scales to assess flourishing and positive and negative feelings. Social Indicators Research, 97(2), 143–156.
10.
du PlessisG. A.GuseT. (2017). Validation of the Scale of Positive and Negative Experience in a South African student sample. South African Journal of Psychology, 47(2), 184–197.
11.
GnambsT.SchroedersU. (2020). Cognitive abilities explain wording effects in the Rosenberg Self-Esteem Scale. Assessment, 27(2), 404–418.
12.
HowellA. J.BuroK. (2015). Measuring and predicting student well-being: Further evidence in support of the flourishing scale and the scale of positive and negative experiences. Social Indicators Research, 121(3), 903–915.
13.
IronsonG. H.SmithP. C.BrannickM. T.GibsonW. M.PaulK. B. (1989). Construction of a Job in General scale: A comparison of global, composite, and specific measures. Journal of Applied Psychology, 74(2), 193–200. https://doi.org/10.1037/0021-9010.74.2.193
14.
JacobyW. G. (2000). Loess: A nonparametric, graphical tool for depicting relationships between variables. Electoral Studies, 19(4), 577–613.
15.
JosephD. L.ChanM. Y.HeintzelmanS. J.TayL.DienerE.ScotneyV. S. (2020). The manipulation of affect: A meta-analysis of affect induction procedures. Psychological Bulletin, 146(4), 355–375. https://doi.org/10.1037/bul0000224
16.
JovanovićV. (2015). Beyond the PANAS: Incremental validity of the Scale of Positive and Negative Experience (SPANE) in relation to well-being. Personality and Individual Differences, 86, 487–491.
17.
JovanovićV.LazićM.Gavrilov-JerkovićV.MolenaarD. (2020). The Scale of Positive and Negative Experience (SPANE): Evaluation of measurement invariance and convergent and discriminant validity. European Journal of Psychological Assessment, 36(4), 694–704.
18.
JudgeT. A.BonoJ. E. (2001). Relationship of core self-evaluations traits—self-esteem, generalized self-efficacy, locus of control, and emotional stability—with job satisfaction and job performance: A meta-analysis. Journal of Applied Psychology, 86(1), 80–92.
19.
JudgeT. A.WeissH. M.Kammeyer-MuellerJ. D.HulinC. L. (2017). Job attitudes, job satisfaction, and job affect: A century of continuity and of change. Journal of Applied Psychology, 102(3), 356–374. https://doi.org/10.1037/apl0000181
20.
KamC. C. S. (2020). Testing the assumption of population homogeneity in the measurement of dispositional optimism: Factor mixture modeling analysis. Journal of Personality Assessment, 102(2), 238–249. https://doi.org/10.1080/00223891.2018.1502194
21.
KamC. C. S.FanX. (2020). Investigating Response Heterogeneity in the Context of Positively and Negatively Worded Items by Using Factor Mixture Modeling. Organizational Research Methods, 23(2), 322–341. https://doi.org/10.1177/1094428118790371
22.
KamC. C. S.MeyerJ. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18(3), 512–541.
23.
KamC. C. S.MeyerJ. P.SunS. (2021). Why do people agree with both regular and reversed items? A logical response perspective. Assessment, 28(4), 1110–1124.
24.
KleinA.MoosbruggerH. (2000). Maximum likelihood estimation of latent interaction effects with the LMS method. Psychometrika, 65(4), 457–474.
25.
KochT.KelavaA.EidM. (2017). Analyzing different types of moderated method effects in confirmatory factor models for structurally different methods. Structural Equation Modeling, 25, 179–200. https://doi.org/10.1080/10705511.2017.1373595
26.
LiC. H. (2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949.
27.
LindwallM.BarkoukisV.GranoC.LucidiF.RaudseppL.LiukkonenJ.Thøgersen-NtoumaniC. (2012). Method effects: The problem with negatively versus positively keyed items. Journal of personality assessment, 94(2), 196–204. https://doi.org/10.1080/00223891.2011.645936
28.
LitsonK.GeiserC.BurnsG. L.ServeraM. (2017). Examining trait × method interactions using mixture distribution multitrait–multimethod models. Structural equation modeling, 24(1), 31–51.
29.
MagazineS. L.WilliamsL. J.WilliamsM. L. (1996). A confirmatory factor analysis examination of reverse coding effects in Meyer and Allen’s Affective and Continuance Commitment Scales. Educational and Psychological Measurement, 56(2), 241–250.
30.
MarshH. W.ScalasL. F.NagengastB. (2010). Longitudinal tests of competing factor structures for the Rosenberg Self-Esteem Scale: Traits, ephemeral artifacts, and stable response styles. Psychological Assessment, 22(2), 366–381.
31.
MatthewsC. O. (1927). The effect of position of printed response words upon children’s answers to questions in two-response types of tests. Journal of Educational Psychology, 18(7), 445–457.
32.
MenoldN. (2020). How do reverse-keyed items in inventories affect measurement quality and information processing?Field Methods, 32(2), 140–158.
33.
MeyerJ. P.AllenN. J.SmithC. A. (1993). Commitment to organizations and occupations: Extension and test of a three-component conceptualization. Journal of Applied Psychology, 78(4), 538–551.
34.
MichaelidesM. P. (2019). Negative keying effects in the factor structure of TIMSS 2011 motivation scales and associations with reading achievement. Applied Measurement in Education, 32(4), 365–378.
35.
NadlerJ. T.WestonR.VoylesE. C. (2015). Stuck in the middle: The use and interpretation of mid-points in items on questionnaires. The Journal of General Psychology, 142(2), 71–89.
36.
NewmanD. A. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17(4), 372–411.
PengK.NisbettR. E. (1999). Culture, dialectics, and reasoning about contradiction. American Psychologist, 54(9), 741–754.
39.
PohlS.SteyerR.KrausK. (2008). Modelling method effects as individual causal effects. Journal of the Royal Statistical Society: Series A (Statistics in Society), 171(1), 41–63.
40.
RahmT.HeiseE.SchuldtM. (2017). Measuring the frequency of emotions—validation of the Scale of Positive and Negative Experience (SPANE) in Germany. PLOS ONE, 12(2), Article e0171288.
41.
RosenbergM. (1965). Society and the adolescent self-image. Princeton University Press.
42.
RosseelY. (2012). Lavaan: An R package for structural equation modeling and more. Version 0.5–12 (BETA). Journal of Statistical Software, 48(2), 1–36.
SatorraA.BentlerP. M. (2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66(4), 507–514.
45.
ScheierM. F.CarverC. S.BridgesM. W. (1994). Distinguishing optimism from neuroticism (and trait anxiety, self-mastery, and self-esteem): A reevaluation of the Life Orientation Test. Journal of Personality and Social Psychology, 67(6), 1063–1078.
46.
SchmittD. P.AllikJ. (2005). Simultaneous administration of the Rosenberg Self-Esteem Scale in 53 nations: Exploring the universal and culture-specific features of global self-esteem. Journal of Personality and Social Psychology, 89(4), 623–642.
47.
SchmittN.StuitsD. M. (1985). Factors defined by negatively keyed items: The result of careless respondents?Applied Psychological Measurement, 9(4), 367–373.
48.
SchriesheimC. A.EisenbachR. J. (1995). An exploratory and confirmatory factor-analytic investigation of item wording effects on the obtained factor structures of survey questionnaire measures. Journal of Management, 21(6), 1177–1193. https://doi.org/10.1177/014920639502100609
49.
SchriesheimC. A.EisenbachR. J.HillK. D. (1991). The effect of negation and polar opposite item reversals on questionnaire reliability and validity: An experimental investigation. Educational and Psychological Measurement, 51(1), 67–78.
50.
SimmsL. J.ZelaznyK.WilliamsT. F.BernsteinL. (2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 557–566.
51.
SmithP. C.KendallL. M.HulinC. L.BalzerW. K.KihmJ. A.IrwinJ. L.ParraL. F. (2009). Job descriptive index (2009 revision) and the job in general scales (2009 revision). Bowling Green State University.
52.
SpectorP. E. (1997). Job satisfaction: Application, assessment, causes, and consequences. SAGE.
53.
SpielbergerC. D.GorsuchR. L.LusheneR.VaggP. R.JacobsG. A. (1983). Manual for the State-Trait Anxiety Inventory. Consulting Psychologists Press.
54.
StogdillR. M. (1963). Manual for the leader behavior description questionnaire—Form XII. Bureau of Business Research, Ohio State University.
55.
van SonderenESandermanRCoyneJC (2013) Correction: Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain. PLOS ONE8(9): 10.1371/annotation/af78b324-7b44-4f89-b932-e851fe04a8e5. https://doi.org/10.1371/annotation/af78b324-7b44-4f89-b932-e851fe04a8e5
WatsonD.ClarkL. A.TellegenA. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063–1070.
YildirimM. (2019). Mediating role of resilience in the relationships between fear of happiness and affect balance, satisfaction with life, and flourishing. Europe’s Journal of Psychology, 15(2), 183–198.
60.
ZellE.StrickhouserJ. E.SedikidesC.AlickeM. D. (2020). The better-than-average effect in comparative self-evaluation: A comprehensive review and meta-analysis. Psychological Bulletin, 146(2), 118–149.
