We investigated the effects that careless responding and acquiescence response bias have on analyses conducted to assess construct dimensionality. Using job satisfaction/dissatisfaction as the focal construct, we measured and controlled for careless responding and acquiescence bias in data obtained from an online survey of employees (N = 666) from different organizations and occupational groups. We found that the negative correlation between factors defined by job satisfaction and dissatisfaction items, respectively, was attenuated by careless responding and acquiescence bias, and was not significantly different from –1.0 when both were controlled. Moreover, the correlations between the two factors and measures of other constructs were similar when careless responding and acquiescence were controlled. These findings challenge recent research purporting to demonstrate that job satisfaction and dissatisfaction are distinct constructs. Recommendations for future investigations of construct dimensionality are provided.
AshtonM. C.LeeK.GoldbergL. R.de VriesR. E. (2009). Higher-order factors of personality: Do they exist?Personality and Social Psychology Review, 13, 79–91.
2.
BakkerA. B.LeiterM. P. (2010). Work engagement: A handbook of essential theory and research. New York, NY: Psychology Press.
3.
BarberL. K.BarnesC. M.CarlsonK. D. (2013). Random and systematic error effects of insomnia on survey behavior. Organizational Research Methods, 16, 616–649.
4.
BarnetteJ. J. (1999). Nonattending respondent effects on internal consistency of self-administered surveys: A Monte Carlo simulation study. Educational and Psychological Measurement, 59, 38–46.
5.
BentlerP. M.JacksonD. N.MessickS. (1971). Identification of content and style: A two-dimensional interpretation of acquiescence. Psychological Bulletin, 76, 186–204.
6.
BillietJ. B.McClendonM. J. (2000). Modeling acquiescence in measurement models for two balanced sets of items. Structural Equation Modeling, 7, 608–628.
7.
BoltD. M.LuY.KimJ.-S. (2014). Measurement and control of response styles using anchoring vignettes: A model-based approach. Psychological Methods. Advance online publication. http://dx.doi.org/10.1037/met0000016
8.
ChernyshenkoO. S.StarkS.CredéM.WadlingtonP.LeeW. (2003, April). Improving the measurement of job attitudes: The development of the IJSI. Paper presented at the 18th annual conference for the Society of Industrial and Organizational Psychologists, Orlando, FL.
9.
ColeD. A.MartinN. M.SteigerJ. H. (2005). Empirical and conceptual problems with longitudinal trait-state models: Introducing a trait-state-occasion model. Psychological Methods, 10, 3–20.
10.
CredéM.ChernyshenkoO. S.BagraimJ.SullyM. (2009). Contextual performance and job satisfaction–dissatisfaction distinction: Examining artifacts and utility. Human Performance, 22, 246–272.
11.
DemeroutiE.MostertK.BakkerA. B. (2010). Burnout and work engagement: A thorough investigation of the independency of both constructs. Journal of Occupational Health Psychology, 15, 209–222.
12.
DienerE.WirtzD.TovW.Kim-PrietoC.ChoiD.OishiS.Biswas-DienerR. (2010). New measures of well-being: Short scales to assess flourishing and positive and negative feelings. Social Indicators Research, 97, 143–156.
13.
FoxS.SpectorP. E.BruursemaK.KesslerS.GohA. (2007, August). Necessity is the mother of behavior: Organizational constraints, CWB and OCB. Paper presented at the meeting of the Academy of Management, Philadelphia, PA.
14.
GoldbergL. R. (1992). The development of markers for the Big-Five factor structure. Psychological Assessment, 4, 26.
15.
HerzbergF.MausnerB.SnydermanB. B. (1959). The motivation to work (2nd ed.). New York, NY: John Wiley.
16.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114.
17.
JohnsonJ. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.
18.
KamC. C. S.MeyerJ. P. (2015). New methodology in nomological network investigation. Manuscript in preparation. The University of Western Ontario.
19.
LockeE. A. (1976). The nature and causes of job satisfaction. In DunnetteM. D. (Ed.)., Handbook of industrial and organisational psychology (pp. 1297–343). Chicago, IL: Rand McNally.
20.
LorgeI. (1937). Gen-like: Halo or reality?Psychological Bulletin, 34, 296–303.
21.
ManiaciM. R.RoggeR. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83.
22.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
23.
OppenheimerD. M.MeyvisT.DavidenkoN. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.
24.
PiedmontR. L.McCraeR. R.RiemannR.AngleitnerA. (2000). On the invalidity of validity scales: Evidence from self-reports and observer ratings in volunteer samples. Journal of Personality and Social Psychology, 78, 582.
25.
PreacherK. J. (2006). Testing complex correlational hypotheses with structural equation models. Structural Equation Modeling, 13, 520–543.
26.
R Development Core Team. (2012). R: A language and environment for statistical computing[Computer software manual]. Vienna, Austria: Author.
27.
RammstedtB.FarmerR. F. (2013). The impact of acquiescence on the evaluation of personality structure. Psychological Assessment, 25, 1137–1145.
28.
RammstedtB.GoldbergL. R.BorgI. (2010). The measurement equivalence of Big Five factor markers for persons with different levels of education. Journal of Research in Personality, 44, 53–61.
29.
RosseelY. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1–36.
30.
SalgadoJ. F. (2002). The Big Five personality dimensions and counterproductive behaviors. International Journal of Selection and Assessment, 10, 117–125.
31.
SatorraA. (2000). Scaled and adjusted restricted tests in multi-sample analysis of moment structures. In HeijmansR. D. H.PollockD. S. G.SatorraA. (Eds.), Innovations in multivariate statistical analysis: A Festschrift for Heinz Neudecker (pp. 233–247). London, UK: Kluwer.
32.
SchmittN.StultsD. M. (1985). Factors defined by negative keyed items: The result of careless respondents?Applied Psychological Measurement, 9, 367–373.
33.
SpectorP. E.FoxS.PenneyL. M.BruursemaK.GohA.KesslerS. (2006). The dimensionality of counterproductivity: Are all counterproductive behaviors created equal?Journal of Vocational Behavior, 68, 446–460.
34.
TourangeauR.RipsL. J.RasinskiK. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
35.
UzielL. (2010). Rethinking social desirability scales: From impression management to interpersonally oriented self-control. Perspectives on Psychological Science, 5, 243–262.
WeijtersB.GeuensM.SchillewaertN. (2010). The stability of individual response styles. Psychological Methods, 15, 96–110.
38.
WigginsJ. S. (1973). Personality and prediction. Reading, MA: Addison-Wesley.
39.
WinklerJ. D.KanouseD. E.WareJ. E. (1982). Controlling for acquiescence response set in scale development. Journal of Applied Psychology, 67, 555–561.
40.
WoodsC. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 186–191.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
