We applied a new approach to Generalizability theory (G-theory) involving parallel splits and repeated measures to evaluate common uses of the Paulhus Deception Scales based on polytomous and four types of dichotomous scoring. G-theory indices of reliability and validity accounting for specific-factor, transient, and random-response measurement error supported use of polytomous over dichotomous scores as contamination checks; as control, explanatory, and outcome variables; as aspects of construct validation; and as indexes of environmental effects on socially desirable responding. Polytomous scoring also provided results for flagging faking as dependable as those when using dichotomous scoring methods. These findings argue strongly against the nearly exclusive use of dichotomous scoring for the Paulhus Deception Scales in practice and underscore the value of G-theory in demonstrating this. We provide guidelines for applying our G-theory techniques to other objectively scored clinical assessments, for using G-theory to estimate how changes to a measure might improve reliability, and for obtaining software to conduct G-theory analyses free of charge.
AlleyneE.GannonT. A.CiardhaC. Ó.WoodJ. L. (2014). Community males show multiple-perpetrator rape proclivity: Development and preliminary validation of an interest scale. Sexual Abuse: A Journal of Research and Treatment, 26, 82-104.
2.
BeckerG. (2000). How important is transient error in estimating reliability? Going beyond simulation studies. Psychological Methods, 5, 370-379.
3.
BrennanR. L. (2001a). Generalizability theory. New York, NY: Springer.
4.
BrennanR. L. (2001b). Manual for mGENOVA (Ver. 2.1). Iowa City: Iowa Testing Programs, University of Iowa.
5.
BrennanR. L. (2001c). Manual for urGENOVA (Ver. 2.1). Iowa City: Iowa Testing Programs, University of Iowa.
6.
ByrneB. (1996). Measuring self-concept across the lifespan. Washington, DC: American Psychological Association.
7.
CervellioneK. L.LeeY-S.BonannoG. A. (2009). Rasch modeling of the Self-Deception Scale of the Balanced Inventory of Desirable Responding. Educational and Psychological Measurement, 69, 438-458.
8.
ChristophersonK-A.HelsethS.LundT. (2008). A generalizability study of the Norwegian version of KINDL® in a sample of healthy adolescents. Quality of Life Research, 17, 87-93.
9.
CrockerL.AlginaJ. (1986). Introduction to classical and modern test theory. Orlando, FL: Holt, Rinehart and Winston.
10.
CronbachL. J.GleserG. C.NandaH.RajaratnamN. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
11.
DunnT. J.BaguleyT.BrunsdenV. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399-412.
12.
EdwardsA. L. (1957). The social desirability variable in personality assessment and research. New York, NY: Dryden.
13.
FeldtL. S.BrennanR. L. (1989). Reliability. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 105-146). New York, NY: Macmillan.
14.
FreemanA. J.SchumacherJ. A.CoffeyS. F. (2015). Social desirability and partner agreement of men’s reporting of intimate partner violence in substance abuse treatment settings. Journal of Interpersonal Violence, 30, 565-579.
15.
GendreauP.IrvineM.KnightS. (1973). Evaluating response set styles on the MMPI with prisoners: Faking good adjustment and maladjustment. Canadian Journal of Behavioral Science, 5, 183-194.
16.
GreenS. B.YangY. (2009). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74, 121-135.
17.
HaertelE. (2006). Reliability. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 65-110). Westport, CT: Praeger.
18.
HoldenR. R.KronerD. G.FekkenG. C.PophamS. M. (1992). A model of personality test item response dissimulation. Journal of Personality and Social Psychology, 63, 272-279.
19.
HuangC. (2013). Relation between self-esteem and socially desirable responding and the role of socially desirable responding in the relation between self-esteem and performance. European Journal of Psychology of Education, 28, 663-683.
20.
KuentzelJ. G.HendersonM. J.MelvilleC. L. (2008). The impact of social desirability biases on self-report among college student and problem gamblers. Journal of Gambling Studies, 24, 307-319.
21.
LoweryA. E.GreenbergM. A.FosterS. L.ClarkK.CasdenD. R.LoscalzoM.BardwellW. A. (2012). Validation of a need-based biopsychosocial distress instrument for cancer patients. Psycho-Oncology, 21, 1099-1106.
22.
MacCallumR. C.ZhangS.PreacherK. J.RuckerD. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19-40.
23.
MushquashC.O’ConnorB. P. (2006). SPSS and SAS programs for generalizability theory analyses. Behavior Research Methods, 38, 542-547.
24.
NeemannJ.HarterS. (1986). Self-Perception Profile for College Students: Manual and questionnaires. Denver, CO: University of Denver.
25.
NeemannJ.HarterS. (2012). Self-Perception Profile for College Students: Manual and questionnaires (Rev. ed.). Denver, CO: University of Denver.
26.
NugentW. R. (2006). A psychometric study of the MPSI suicidal thoughts subscale. Stress, Trauma, and Crisis, 9, 1-15.
27.
PaulhusD. L. (1991). Measurement and control of response bias. In RobinsonJ. P.ShaverP. R.WrightsmanL. S. (Eds.), Measures of personality and social psychological attitudes (pp. 17-59). San Diego, CA: Academic Press.
28.
PaulhusD. L. (1999). Paulhus Deception Scales (PDS): The Balanced Inventory of Desirable Responding-7 (User’s manual). Toronto, Ontario, Canada: Multi-Health Systems.
29.
PaulhusD. L. (2002). Socially desirable responding: The evolution of a construct. In BraunH.JacksonD.WileyD. (Eds.), The role of constructs in psychological and educational measurement (pp. 39-69). Mahwah, NJ: Lawrence Erlbaum.
30.
PaulhusD. L.TrapnellP. D. (2008). Self-presentation: An agency-communion framework. In JohnO. P.RobinsR. W.PervinL. A. (Eds.), Handbook of personality psychology (pp. 492-517). New York, NY: Guilford.
31.
PaulhusD. L.VazireS. (2007). The self-report method. In RobinsR. W.FraleyR. C.KruegerR. (Eds.), Handbook of research methods in personality psychology (pp. 224-239). New York, NY: Guilford Press.
32.
PenneyS. R.SkillingT. A. (2012). Moderators of informant agreement in the assessment of adolescent psychopathology: Extension to a forensic sample. Psychological Assessment, 24, 386-401.
33.
PoseyC. D.HessA. K. (1984). The fakability of subtle and obvious measures of aggression by male prisoners. Journal of Personality Assessment, 48, 137-144.
34.
ReeveC. L.HeggestadE. D.GeorgeE. (2005). Estimation of transient error in cognitive ability scales. International Journal of Selection and Assessment, 13, 316-320.
35.
RulonP. J. (1939). A simplified procedure for determining the reliability of a test by split-halves. Harvard Educational Review, 9, 99-103.
36.
SchmidtF. L.LeH.IliesR. (2003). Beyond alpha: An empirical investigation of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods, 8, 206-224.
37.
SchmittN. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.
38.
SchretlenD.ArkowitzH. (1990). A psychological test battery to detect prison inmates who fake insanity or mental retardation. Behavioral Sciences & the Law, 8, 75-84.
39.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer (Vol. 1). Newbury Park, CA: Sage.
40.
ShavelsonR. J.WebbN. M.RowleyG. L. (1989). Generalizability theory. American Psychologist, 44, 922-932.
41.
SijtsmaK. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107-120.
42.
SmithP. L. (1978). Sampling error of variance components in small sample multifacet generalizability designs. Journal of Educational Measurement, 3, 319-346.
43.
SpearmanC. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72-101.
44.
StöberJ.DetteD. E.MuschJ. (2002). Comparing continuous and dichotomous scoring of the balanced inventory of desirable responding. Journal of Personality Assessment, 78, 370-389.
45.
ThorndikeR. L. (1951). Reliability. In LindquistE. F. (Ed.), Educational measurement (pp. 560-620). Washington, DC: American Council on Education.
46.
TingN.BurdickR. K.GraybillF. A.JeyaratnamS.LuT. F. C. (1990). Confidence intervals on linear combinations of variance components that are unrestricted in sign. Journal of Statistical Computation and Simulation, 35, 135-143.
47.
VispoelW. P.Forte FastE. E. (2000). Response biases and their relations to sex differences in multiple domains of self-concept. Applied Measurement in Education, 13, 79-97.
48.
VispoelW. P.HeY. (2010, May). Evaluating the multifaceted, hierarchical structure of self-concept using the Self Description Questionnaire III (SDQ-III), Self-Perception Profile for College Students (SPPCS), and Arts Self-Perception Inventory (ASPI). Paper presented at the Annual Meeting of the American Educational Research Association, Denver, CO.
49.
VispoelW. P.KimH. Y. (2014). Psychometric properties for the Balanced Inventory of Desirable Responding: Dichotomous versus polytomous conventional and IRT scoring. Psychological Assessment, 26, 878-891.
50.
VispoelW. P.TaoS. (2013). A generalizability analysis of score consistency for the Balanced Inventory of Desirable Responding. Psychological Assessment, 25, 94-104.
51.
WebbN. M.RowleyG. L.ShavelsonR. J. (1988). Using generalizability theory in counseling and development. Measurement and Evaluation in Counseling and Development, 21, 81-90.
52.
WileyE. W.WebbN. M.ShavelsonR. J. (2013). The generalizability of test scores. In GeisingerK. F.BrackenB. A.CarlsonJ. F.HansenJ. I. C.KuncelN. R.ReiseS. P.RodriguezM. C. (Eds.), APA handbook of testing and assessment in psychology: Vol 1: Test theory and testing and assessment in industrial and organizational psychology (3rd ed., pp. 43-60). Washington, DC: American Psychological Association.
