Factor mixture modeling was used to investigate potential response incongruity between positively and negatively worded items. Survey respondents (N = 591) answered questions about job satisfaction and dissatisfaction. Results revealed two classes of respondent: a majority class, who generally do not have problems answering positively and negatively worded items, and a minority class, who have serious trouble with negatively worded items. With the exclusion of the minority class, job satisfaction and dissatisfaction were found to be essentially unidimensional rather than bidimensional as previous research had suggested. These results challenge previous findings regarding the bidimensionality of job satisfaction and cast doubt on the widespread research practice of assuming population homogeneity in survey responses. A detailed flowchart illustrating the analytic procedure and an Mplus syntax program are provided so that researchers can conduct similar investigations on constructs of interest.
BauerD. J.CurranP. J. (2004). The integration of continuous and discrete latent variable models: Potential problems and promising opportunities. Psychological Methods, 9, 3–29.
2.
CampbellD. T.FiskeD. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.
3.
CarminesE. G.ZellerR. A. (1979). Reliability and validity assessment. Thousand Oaks, CA: Sage.
4.
ClarkS. L.MuthénB.KaprioJ.D’OnofrioB. M.VikenR.RoseR. J. (2013). Models and strategies for factor mixture analysis: An example concerning the structure underlying psychological disorders. Structural Equation Modeling, 20, 681–703.
5.
ColeD. A.CieslaJ. A.SteigerJ. H. (2007). The insidious effects of failing to include design-driven correlated residuals in latent-variable covariance structure analysis. Psychological Methods, 12, 381–398.
6.
CorwynR. F. (2000). The factor structure of global self-esteem among adolescents and adults. Journal of Research in Personality, 34, 357–379.
7.
CredéM.ChernyshenkoO. S.BagraimJ.SullyM. (2009). Contextual performance and the job satisfaction-dissatisfaction distinction: Examining artifacts and utility. Human Performance, 22, 246–272.
8.
DalbertC.LipkusI. M.SallayH.GochI. (2001). A just and an unjust world: Structure and validity of different world beliefs. Personality and Individual Differences, 30, 561–577.
9.
de JongeP.SlaetsJ. (2005). Response sets in self-report data and their associations with personality traits. European Journal of Psychiatry, 19, 209–214.
10.
FinchW. H.BronkK. C. (2011). Conducting confirmatory latent class analysis using Mplus. Structural Equation Modeling, 18, 132–151.
11.
FinchamF. D.LinfieldK. J. (1997). A new look at marital quality: Can spouses feel positive and negative about their marriage?Journal of Family Psychology, 11, 489–502.
12.
FornellC.LarckerD. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18, 39–50.
13.
GnambsT.SchroedersU. (2017). Cognitive abilities explain wording effects in the RSEE. Assessment. Advance online publication. doi:10.1177/1073191117746503
14.
GreenbergerE.ChenC.DmitrievaJ.FarruggiaS. P. (2003). Item-wording and the dimensionality of the Rosenberg Self-Esteem Scale: Do they matter?Personality and Individual Differences, 35, 1241–1254.
15.
GuH.WenZ.FanX. (2017). Examining and controlling for wording effect in a self-report measure: A Monte Carlo simulation study. Structural Equation Modeling, 24, 545–555.
16.
HernándezA.DrasgowF.González-RomáV. (2004). Investigating the functioning of a middle category by means of a mixed-measurement model. Journal of Applied Psychology, 89, 687–699.
17.
HerzbergP. Y.GlaesmerH.HoyerJ. (2006). Separating optimism and pessimism: A robust psychometric analysis of the revised Life Orientation Test (LOT-R). Psychological Assessment, 18, 433–438.
18.
HerzbergF.MausnerB.PetersonR. D.CapwellD. F. (1957). Job attitudes: Review of research and opinions. Pittsburgh, PA: Psychological Service of Pittsburgh.
19.
HinesG. H. (1973). Cross-cultural differences in two-factor motivation theory. Journal of Applied Psychology, 58, 375–377.
20.
HoldenR. R.FekkenG. C. (1990). Structured psychopathological test item characteristics and validity. Psychological Assessment, 2, 35–40.
21.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort respond to surveys. Journal of Business and Psychology, 27, 99–114.
22.
HuangJ. L.LiuM.BowlingN. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828–845.
23.
JudgeT. A.WeissH. M.Kammeyer-MuellerJ. D.HulinC. L. (2017). Job attitudes, job satisfaction, and job affect: A century of continuity and of change. Journal of Applied Psychology, 102, 356–374.
24.
KamC. C. S. (2017). Novel insights into item keying/valence effect using latent difference (LD) modeling analysis. Journal of Personality Assessment. Advance online publication. doi:10.1080/00223891.2017.1369095
25.
KamC. C. S.ChanG. H.-h. (2018). Examination of the validity of instructed response items in identifying careless respondents. Personality and Individual Differences, 129, 83–87.
26.
KamC.MeyerJ. P. (2012). Do optimism and pessimism have different relationships with personality dimensions? A re-examination. Personality and Individual Differences, 52, 123–127.
27.
KamC. C. S.MeyerJ. P. (2015a). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18, 512–541.
28.
KamC. C. S.MeyerJ. P. (2015b). Implications of item keying and item valence for the investigation of construct dimensionality. Multivariate Behavioral Research, 50, 457–469.
29.
KamC.MorinA. J.MeyerJ. P.TopolnytskyL. (2016). Are commitment profiles stable and predictable? A latent transition analysis. Journal of Management, 42, 1462–1490.
30.
LindwallM.BarkoukisV.GranoC.LucidiF.RaudseppL.LiukkonenJ.Thøgersen-NtoumaniC. (2012). Method effects: The problem with negatively versus positively keyed items. Journal of Personality Assessment, 94, 196–204.
31.
LitsonK.GeiserC.BurnsG. L.ServeraM. (2017). Examining Trait × Method interactions using mixture distribution multitrait-multimethod models. Structural Equation Modeling: A Multidisciplinary Journal, 24, 31–51.
32.
LubkeG. H.MuthénB. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10, 21–39.
33.
MagazineS. L.WilliamsL. J.WilliamsM. L. (1996). A confirmatory factor analysis examination of reverse coding effects in Meyer and Allen’s Affective and Continuance Commitment Scales. Educational and Psychological Measurement, 56, 241–250.
34.
MarshH. W. (1996). Positive and negative global self-esteem: A substantively meaningful distinction or artifactors?Journal of Personality and Social Psychology, 70, 810–819.
35.
MarshH. W.ScalasL. F.NagengastB. (2010). Longitudinal tests of competing factor structures for the Rosenberg self-esteem scale: Traits, ephemeral artifacts, and stable response styles. Psychological Assessment, 22, 366–381.
36.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
37.
MichaelidesM.KoutsogiorgiC.PanayiotouG. (2015). Method effects on an adaptation of the Rosenberg Self-Esteem Scale in Greek and the role of personality traits. Journal of Personality Assessment, 98, 178–188.
38.
NunnallyJ. C.BernsteinI. H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
39.
NylundK. L.AsparouhovT.MuthénB. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling, 14, 535–569.
40.
PodsakoffP. M.MacKenzieS. B.LeeJ. Y.PodsakoffN. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 8799–8803.
41.
QuiltyL. C.OakmanJ. M.RiskoE. (2006). Correlates of the Rosenberg Self-Esteem Scale method effects. Structural Equation Modeling, 13, 991–917.
42.
RaykovT.MarcoulidesG. A. (2018). On studying common factor dominance and approximate unidimensionality in multicomponent measuring instruments with discrete items. Educational and Psychological Measurement, 78, 504–516.
43.
RaykovT.MarcoulidesG. A.ChangC. (2016). Examining population heterogeneity in finite mixture settings using latent variable modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23, 726–730.
44.
RaykovT.MarcoulidesG. A.LiT. (2015). Evaluation of measurement instrument criterion validity in finite mixture settings. Educational and Psychological Measurement, 76, 1026–1044.
45.
ReiseS. P.KimD. S.MansolfM.WidamanK. F. (2016). Is the bifactor model a better model or is it just better at modeling implausible responses? Application of iteratively reweighted least squares to the Rosenberg Self-Esteem Scale. Multivariate Behavioral Research, 51, 818–838.
46.
RodriguezA.ReiseS. P.HavilandM. G. (2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21, 137–150.
47.
RoszkowskiM.SovenM. (2010). Shifting gears: Consequences of including two negatively worded items in the middle of a positively worded questionnaire. Assessment & Evaluation in Higher Education, 35, 113–130.
48.
SatorraA. (2000). Scaled and adjusted restricted tests in multi-sample analysis of moment structures. In HeijmansR. D. H.PollockD. S. G.SatorraA. (Eds.), Innovations in multivariate statistical analysis (pp. 233–247). Boston, MA: Springer.
49.
SchmittN.StuitsD. M. (1985). Factors defined by negatively keyed items: The result of careless respondents?Applied Psychological Measurement, 9, 367–373.
50.
SchriesheimC. A.EisenbachR. J. (1995). An exploratory and confirmatory factor-analytic investigation of item wording effects on the obtained factor structures of survey questionnaire measures. Journal of Management, 21, 1177–1193.
51.
SchriesheimC. A.EisenbachR. J.HillK. D. (1991). The effect of negation and polar opposite item reversals on questionnaire reliability and validity: An experimental investigation. Educational and Psychological Measurement, 51, 67–78.
52.
SchriesheimC. A.HillK. D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41, 1101–1114.
53.
SliterK. A.ZickarM. J. (2014). An IRT examination of the psychometric functioning of negatively worded personality items. Educational and Psychological Measurement, 74, 214–226.
54.
TomásJ. M.OliverA.GalianaL.SanchoP.LilaM. (2013). Explaining method effects associated with negatively worded items in trait and state global and domain-specific self-esteem scales. Structural Equation Modeling, 20, 299–313.
55.
TourangeauR.RipsL. J.RasinskiK. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
56.
van SonderenE.SandermanR.CoyneJ. C. (2013). Ineffectiveness of reverse wording of questionnaire items: Let’s learn from cows in the rain. PloS One, 8, e68967.
57.
VautierS.PohlS. (2009). Do balanced scales assess bipolar constructs? The case of the STAI scales. Psychological Assessment, 21, 187–193.
58.
VriezeS. I. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17, 228–243.
59.
WangM.HangesP. J. (2011). Latent class procedures: Applications to organizational research. Organizational Research Methods, 14, 24–31.
60.
WanousJ. P.ReichersA. E.HudyM. J. (1997). Overall job satisfaction: How good are single-item measures?Journal of Applied Psychology, 82, 247–252.
61.
WeijtersB.BaumgartnerH. (2012). Misresponse to reversed and negated items in surveys: A review. Journal of Marketing Research, 49, 737–747.
62.
WurptsI. C.GeiserC. (2014). Is adding more indicators to a latent class analysis beneficial or detrimental? Results of a Monte-Carlo study. Frontiers in Psychology, 5, Article 920. doi:10.3389/fpsyg.2014.00920
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
