Despite pervasive evidence that general mental ability and personality are unrelated, we investigated whether general mental ability may affect the response process associated with personality measurement. Study 1 examined a large sample of job applicant responses to four personality scales for differential functioning across groups of differing general mental ability. While results indicated that personality items differentially function across highly disparate general mental ability groups, there was little evidence of differential functioning across groups with similar levels of general mental ability. Study 2 replicated these findings in a different sample, using a different measure of general mental ability. We posit that observed differences in the psychometric properties of these personality scales are likely due to the information processing capabilities of the respondents. Additionally, we describe how differential functioning analyses can be used during scale development as a method of identifying items that are not appropriate for all intended respondents. In so doing, we demonstrate procedures for examining other construct-measurement interactions in which respondents’ standings on a specific construct could influence their interpretation of and response to items assessing other constructs.
AckermanP. L.HeggestadE. D. (1997). Intelligence, personality, and interests: Evidence for overlapping traits. Psychological Bulletin, 121, 219–245. doi:10.1037/0033-2909.121.2.219
2.
AtarB.KamataA. (2011). Comparison of IRT likelihood ratio test and logistic regression DIF detection procedures. Hacettepe University Journal of Education, 41, 36–47.
3.
BeckmanH. T.LuegerR. J. (1997). Readability of self-report clinical outcome measures. Journal of Clinical Psychology, 53, 785–789. doi:10.1002/(SICI)1097-4679(199712)53:8<785::AID-JCLP1>3.0.CO;2-F
4.
BockR. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29–51. doi:10.1007/BF02291411
5.
CandellG. L.DrasgowF. (1988). An iterative procedure for linking metrics and assessing bias in item response theory. Applied Psychological Measurement, 12, 253–260. doi:10.1177/014920639902500101
6.
Chamorro-PremuzicT.FurnhamA.AckermanP. L. (2006). Ability and personality correlates of general knowledge. Personality and Individual Differences, 41, 419–429. doi:10.1016/j.paid.2005.11.036
7.
ClarkP. C.LaHuisD. M. (2012). An examination of power and type I errors for two differential item functioning indices using the graded response model. Organizational Research Methods, 15, 229–246. doi:10.1177/1094428111403815
8.
CohenA. S.KimS.-H.WollackJ. A. (1996). An investigation of the likelihood ratio test for detection of differential item functioning. Applied Psychological Measurement, 20, 15–26.
9.
CohenJ. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249–253. doi:10.1177/014662168300700301
10.
CohenJ. (1992). A power primer. Psychological Bulletin, 112, 155–159. doi:10.1037/0033-2909.112.1.155
11.
ColemanM.LiauT. L. (1975). A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60, 283–284. doi:10.1037/h0076540
12.
De BoeckP.WilsonM. (Eds.). (2004). Explanatory item response models: A generalized linear and nonlinear approach. New York, NY: Springer.
13.
DonovanJ. J.DwightS. A.HurtzG. M. (2003). An assessment of the prevalence, severity, and verifiability of entry-level applicant faking using the randomized response technique. Human Performance, 16, 81–106. doi:10.1207/S15327043HUP1601
14.
DrasgowF.LissakR. I. (1983). Modified parallel analysis: A procedure for examining the latent dimensionality of dichotomously scored item responses. Journal of Applied Psychology, 68, 363–373. doi:10.1037/0021-9010.68.3.363
15.
DudleyN. M.OrvisK. A.LebieckiJ. E.CortinaJ. M. (2006). A meta-analytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits. Journal of Applied Psychology, 91, 40–57. doi:10.1037/0021-9010.91.1.40
16.
FleschR. (1948). A new readability yardstick. Journal of Applied Psychology, 32, 221–233. doi:10.1037/h0057532
17.
FlowersC. P.OshimaT. C.RajuN. S. (1999). A description and demonstration of the polytomous-DFIT framework. Applied Psychological Measurement, 23, 309–326. doi:10.1177/01466219922031437
18.
FriedmanD. B.Hoffman-GoetzL. (2006). A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Education and Behavior, 33, 352–373. doi:10.1177/1090198105277329
19.
FurrR. M.BacharachV. R. (2008). Psychometrics: An introduction. Thousand Oaks, CA: Sage Publications, Inc.
20.
GlymourC. (1982). Causal inference and causal explanation. In McLaughlinR. (Ed.), What? Where? When? Why? Essays on induction, space, and time, explanation (pp. 179–191). Boston, MA: Reidel Publishing Company.
21.
GoldbergL. R.JohnsonJ. A.EberH. W.HoganR.AshtonM. C.CloningerC. R.GoughH. C. (2006). The International Personality Item Pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84–96. doi:10.1016/j.jrp.2005.08.007
22.
GottfredsonL. S. (2002). Where and why g matters: Not a mystery. Human Performance, 15, 25–46. doi:10.1207/S15327043HUP1501&02_03
23.
GrazianoW. G.Jensen-CampbellL. A.SteeleR. G.HairE. C. (1998). Unknown words in self-reported personality: Lethargic and provincial in Texas. Personality and Social Psychology Bulletin, 24, 893–905. doi:10.1177/0146167298248008
24.
GunningR. (1968). The technique of clear writing. New York, NY: McGraw-Hill.
25.
HattieJ. A. (1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9, 139–164. doi:10.1177/014662168500900204
26.
HedengrenD.StratmannD. (2012). The dog that didn’t bark: Item non-response effects on earning and health. Unpublished manuscript. Retrieved fromhttp://dx.doi.org/10.2139/ssrn.2194373
27.
HinkinT. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1, 104–121. doi:10.1177/109442819800100106
28.
HittC.TrivittJ.R. (2013). Don’t know? Or don’t care? Predicting educational attainment using survey item response rates and coding speed tests as measures of conscientiousness. Unpublished manuscript, Department of Education Reform, University of Arkansas.
29.
HornJ. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185. doi:10.1007/BF02289447
30.
HuangC. D.ChurchA. T.KatigbakM. S. (1997). Identifying cultural differences in items and traits: Differential item functioning in the NEO personality inventory. Journal of Cross-Cultural Psychology, 28(2), 192–218. doi:10.1177/0022022197282004
31.
JohnsonJ. W.CarterG. W.DavisonH. K.OliverD. H. (2001). A synthetic validity approach to testing differential prediction hypotheses. Journal of Applied Psychology, 86, 774–780. doi:10.1037//0021-9010.86.4.774
32.
JudgeT. A.IliesR. (2002). Relationship of personality to performance motivation: A meta-analytic review. Journal of Applied Psychology, 87, 797–807. doi:10.1037/0021-9010.87.4.797
33.
KimS.-H.CohenA. S.AlagozC.KimS. (2007). DIF detection and effect size measures for polytomously scored items. Journal of Educational Measurement, 44(2), 93–116.
34.
LaHuisD. M.ClarkP.O’BrienE. (2011). An examination of item response theory item fit indices for the graded response model. Organizational Research Methods, 14, 10–23.
35.
LautenschlagerG. J.FlahertyV. L.ParkD. G. (1994). IRT differential item functioning: An examination of ability scale purifications. Educational and Psychological Measurement, 54, 21–31. doi:10.1177/0013164494054001003
36.
McKinleyR. L.MillsC. N. (1985). A comparison of several goodness-of-fit statistics. Applied Psychological Measurement, 9, 49–57. doi:10.1177/014662168500900105
37.
MeadeA. W. (2010). A taxonomy of effect size measures for the differential functioning of items and scales. Journal of Applied Psychology, 95, 728–743. doi:10.1037/a0018966
38.
MeadeA. W.LautenschlagerG. J.JohnsonE. C. (2007). A Monte-Carlo examination of the sensitivity of the DFIT framework for tests of measurement invariance with Likert data. Applied Psychological Measurement, 31, 430–455.
39.
MeadeA. W.WrightN. A. (2012). Solving the measurement invariance anchor item problem in item response theory. Journal of Applied Psychology, 97, 1016–1031. doi:10.1037/a00279341016
40.
MeisenbergG.WilliamsA. (2008). Are acquiescent and extreme response styles related to low intelligence and education?Personality and Individual Differences, 44, 1539–1550. doi:10.1016/j.paid.2008.01.010
41.
OnesD. S.ViswesvaranC.ReissA. D. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81, 660–679. doi:10.1037/0021-9010.81.6.660
42.
OnesD. S.ViswesvaranC.SchmidtF. L. (1993). Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance. Journal of Applied Psychology, 78, 679–703. doi:10.1037/0021-9010.78.4.679
43.
OrlandoM.ThissenD. (2003). Further investigation of the performance of S-χ2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289–298. doi:10.1177/0146621603253011
44.
OshimaT. C.MorrisS. B. (2008). Raju’s differential functioning of items and tests (DFIT). Educational Measurement: Issue and Practice, 27, 43–50.
45.
OshimaT. C.RajuN. S.FlowersC. P. (1997). Development and demonstration of multidimensional IRT-based internal measures of differential functioning of items and tests. Journal of Educational Measurement, 34, 253–272. doi:10.1111/j.1745-3984.1997.tb00518.x
46.
PazS. H.LiuH.FongwaM. N.MoralesL. S.HaysR. D. (2009). Readability estimates for commonly used health-related quality of life surveys. Quality of Life Research, 18, 889–900. doi:10.1007/s11136-009-9506-y
47.
PaulhusD. L. (1984). Two-component models of socially desirable responding. Journal of Personality and Social Psychology, 46, 598–609. doi:10.1037/0022-3514.46.3.598
48.
RajuN. S.van der LindenW. J.FleerP. F. (1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19, 353–368. doi:10.1177/014662169501900405
49.
RaschG. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research.
50.
ReeveB. B.FayersP. (2005). Applying item response theory modeling for evaluating questionnaire item and scale properties. In FayersP.HaysR. (Eds.), Assessing quality of life in clinical trials: Methods of practice (pp. 55–73). New York, NY: Oxford University Press.
51.
ReiseS. P. (1990). A comparison of item- and person-fit methods of assessing model-data fit in IRT. Applied Psychological Measurement, 14, 127–137. doi:10.1177/014662169001400202
52.
RivasG. E. L.StarkS.ChernyshenkoO. S. (2009). The effects of referent item parameters on differential item functioning detection using the free baseline likelihood ratio test. Applied Psychological Measurement, 33, 251–265. doi:10.1177/0146621608321760
53.
RothsteinM. G.GoffinR. D. (2006). The use of personality measures in personnel selection: What does current research support?Human Resource Management Review, 16, 155–180. doi:10.1016/j.hrmr.2006.03.004
54.
SalgadoJ. s. F. (1997). The five factor model of personality and job performance in the European community. Journal of Applied Psychology, 82, 30–43. doi:10.1037/0021-9010.82.1.30
55.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement.
56.
SchmidtF. L.HunterJ. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274. doi:10.1037/0033-2909.124.2.262
57.
SchmitM. J.KihmJ. A.RobieC. (2000). Development of a global measure of personality. Personnel Psychology, 53, 153–193. doi:10.1111/j.1744-6570.2000.tb00198.x
58.
SchwarzN. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93–105. doi:10.1037/0003-066X.54.2.93
59.
SchwarzN.BlessH. (1992). Constructing reality and its alternatives: An inclusion/exclusion model of assimilation and contrast effects in social judgment. In MartinL. L.TesserA. (Eds.), The construction of social judgments. (pp. 217–245). New York, NY: Lawrence Erlbaum Associates, Inc.
60.
SheppardR.HanK.ColarelliS. M.DaiG.KingD. W. (2006). Differential item functioning by sex and race in the Hogan personality inventory. Assessment, 13(4), 442–453.
61.
SHL. (2005). Call Center Solutions technical manual. Roswell, GA: Author.
62.
SHL. (2009a). Dynamic Working With Information (WINFO®). Roswell, GA: Author.
StarkS. (2001). MODFIT: A computer program for model-data fit. Unpublished manuscript, University of Illinois at Urbana-Champaign.
65.
StarkS.ChernyshenkoO. S.DrasgowF. (2004). Examining the effects of differential item (functioning and differential) test functioning on selection decisions: When are statistically significant effects practically important?Journal of Applied Psychology, 89, 497–508. doi:10.1037/0021-9010.89.3.497
66.
StarkS.ChernyshenkoO. S.DrasgowF.WilliamsB. A. (2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring?Journal of Applied Psychology, 91, 25–39. doi:10.1037/0021-9010.91.1.25
67.
StoneC. A. (2000). Monte Carlo based null distribution for an alternative fit statistic. Journal of Educational Measurement, 37, 58–75. doi:10.1111/j.1745-3984.2000.tb01076.x
68.
StoneE. F.StoneD. L.GueutalH. G. (1990). Influence of cognitive ability on responses to questionnaire measures: Measurement precision and missing response problems. Journal of Applied Psychology, 75, 418–427. doi:10.1037/0021-9010.75.4.418
69.
TettR. P.JacksonD. N.RothsteinM. (1991). Personality measures as predictors of job performance: A meta-analytic review. Personnel Psychology, 44, 703–742.
70.
ThissenD. (2001). Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning. Chapel Hill, NC: University of North Carolina at Chapel Hill.
71.
ThissenD.SteinbergL.WainerH. (1993). Detection of differential item functioning using the parameters of item response models. In HollandP. W.WainerH. (Eds.), Differential item functioning (pp. 67–113). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
72.
ThorpeG. L.FaviaA. (2012). Data analysis using item response theory methodology: An introduction to selected programs and applications. Psychology Faculty Scholarship, 20, 1–33.
73.
TimmermanM. E.Lorenzo-SevaU. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16, 209–220. doi:0.1037/a0023353
74.
TourangeauR.RasinskiK. A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, 299–314. doi:10.1037/0033-2909.103.3.299
75.
TourangeauR.RipsL. J.RasinskiK. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
76.
VandenbergR. J.LanceC. E.2000. A review and synthesis of the measurement invariance literature: Suggestions, practices and recommendations for organizational research. Organizational Research Methods, 3, 4–70.
77.
VinchurA. J.SchippmannJ. S.SwitzerF. S. I.RothP. L. (1998). A meta-analytic review of predictors of job performance for salespeople. Journal of Applied Psychology, 83, 586–597. doi:10.1037/0021-9010.83.4.586
78.
WangW.ShihC.WangC. (2009). Effects of anchor item methods on differential item functioning detection with the likelihood ratio test. Applied Psychological Measurement, 69, 713–731. doi:10.1177/0013164409332228
79.
Wonderlic Inc. (2004). The Wonderlic QuickTest series of tests successfully predicts scores on the Wonderlic Personnel Test (WPT). Libertyville, IL: Author.
80.
Wonderlic Inc. (2007). Wonderlic Personnel Test normative report. Libertyville, IL: Author.
81.
WoodsC. M. (2009). Empirical selection of anchors for tests of differential item functioning. Applied Psychological Measurement, 33, 42–57. doi:10.1177/0146621607314044
82.
YenW. M. (1981). Using simulation results to choose a latent trait model. Applied Psychological Measurement, 5, 245–252. doi:10.1177/014662168100500212
83.
YuC. H.PoppS. O.DiGangiS.Janasch-PennellA. (2007). Assessing unidimensionality: A comparison of Rasch modeling, parallel analysis, and TETRAD. Practical Assessment, Research, and Evaluation, 12, 1–19.
84.
ZickarM. J.UryK. L. (2002). Developing an interpretation of item parameters for personality items: Content correlates of parameter estimates. Educational and Psychological Measurement, 62, 19–31. doi:10.1177/0013164402062001002
85.
ZumboB. D.PopeG. A.WatsonJ. E.HubleyA. M. (1997). An empirical test of Roskam’s conjecture about the interpretation of an ICC parameter in personality inventories. Educational and Psychological Measurement, 57, 963–969. doi:10.1177/0013164497057006006
