Moderator hypotheses involving categorical variables are prevalent in organizational and psychological research. Despite their importance, current methods of identifying and interpreting these moderation effects have several limitations that may result in misleading conclusions about their implications. This issue has been particularly salient in the literature on differential prediction where recent research has suggested that these limitations have had a significant impact on past research. To help address these issues, we propose several new effect size indices that provide additional information about categorical moderation analyses. The advantages of these indices are then illustrated in two large databases of respondents by examining categorical moderation in the prediction of psychological well-being and the extent of differential prediction in a large sample of job incumbents.
AguinisH. (1995). Statistical power problems with moderated multiple regression in management research. Journal of Management, 21, 1141–1158.
2.
AguinisH.BeatyJ. C.BoikR. J.PierceC. A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90, 94–107.
3.
AguinisH.CulpepperS. A.PierceC. A. (2010). Revival of test bias research in preemployment testing. Journal of Applied Psychology, 95, 648–680.
4.
AguinisH.CulpepperS. A.PierceC. A. (in press). Differential prediction generalization in college admissions testing. Journal of Educational Psychology.
5.
AguinisH.Stone-RomeroE. F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82, 192–206.
6.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
7.
American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
8.
ArveyR. D.CavanaughM. A. (1995). Using surveys to assess the prevalence of sexual harassment: Some methodological problems. Journal of Social Issues, 51, 117–138.
9.
BaronR. M.KennyD. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182.
10.
BerdahlJ. L.MagleyV. J.WaldoC. R. (1996). The sexual harassment of men? Exploring the concept with theory and data. Psychology of Women Quarterly, 20, 527–547.
BerryC. M.CullenM. J.MeyerJ. M. (2014). Racial/ethnic subgroup differences in cognitive ability test range restriction: Implications for differential validity. Journal of Applied Psychology, 99(1), 21–37.
13.
BoscoF. A.AguinisH.SinghK.FieldJ. G.PierceC. A. (2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100, 431–449.
14.
ClearyT. A. (1968). Test bias: Prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5, 115–124.
15.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
16.
CohenJ. (1990). Things I have learned (so far). American Psychologist, 45, 1304–1312.
17.
CohenJ. (1994). The earth is round (p < 0.05). American Psychologist, 49, 997–1003.
18.
CohenJ.CohenP.WestS. G.AikenL. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. New York, NY: Routledge.
19.
CronbachL. J. (1987). Statistical tests for moderator variables: Flaws in analyses recently proposed. Psychological Bulletin, 102, 414–417.
20.
DrasgowF. (1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issues. Psychological Bulletin, 95, 134–135.
21.
DrasgowF.KangT. (1984). Statistical power of differential validity and differential prediction for detecting measurement nonequivalence. Journal of Applied Psychology, 69, 498–508.
22.
FischerF. T.SchultJ.HellB. (2013). Sex-specific differential prediction of college admissions tests: A meta-analysis. Journal of Educational Psychology, 105, 478–488.
23.
FitzgeraldL. F.MagleyV. J.DrasgowF.WaldoC. R. (1999). Measuring sexual harassment in the military: The sexual experiences questionnaire (SEQ-DoD). Military Psychology, 11, 329–343.
24.
GlassG. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.
25.
GutekB. A. (1985). Sex and the workplace: The impact of sexual behavior and harassment on women, men, and organizations. San Francisco, CA: Jossey-Bass.
26.
HartiganJ. A.WigdorA. K. (Eds.). (1989). Fairness in employment testing: Validity generalization, minority issues, and the General Aptitude Test Battery. Washington, DC: National Academy Press.
27.
HumphreysL. (1952). Individual differences. Annual Review of Psychology, 3, 131–150.
28.
HunterJ. E.SchmidtF. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.
29.
JacobsR.MurphyK. R.SilvaJ. (2013). Unintended consequences of EEO enforcement policies: Being big is worse than being bad. Journal of Business and Psychology, 28, 467–471.
30.
KelleyK.PreacherK. J. (2012). On effect size. Psychological Methods, 17, 137–152.
31.
KenigS.RyanJ. (1986). Sex differences in levels of tolerance and attribution of blame for sexual harassment on a university campus. Sex Roles, 15, 535–549.
32.
KirkR. E. (2006). Effect magnitude: A different focus. Journal of Statistical Planning and Inference, 137, 1634–1646.
33.
KutnerM. H.NachtsheimC. J.NeterJ.LiW. (2005). Applied linear statistical models. New York, NY: McGraw-Hill/Irwin.
34.
LinnR. L. (1973). Fair test use in selection. Review of Educational Research, 43, 139–161.
35.
MazenA. M.GrafL. A.KelloggC. E.HemmasiM. (1987). Statistical power in contemporary management research. Academy of Management Journal, 30, 369–380.
36.
MeadeA. W.FetzerM. (2009). Test bias, differential prediction, and a revised approach for determining the suitability of a predictor in a selection context. Organizational Research Methods, 12, 738–761.
37.
MurphyK. R. (1997). Editorial. Journal of Applied Psychology, 82, 3–5.
38.
MurphyK. R.JacobsR. (2012). Using effect size measures to reform the determination of adverse impact in equal employment litigation. Psychology, Public Policy, and the Law, 18, 477–499.
39.
MurphyK. R.RussellC. J. (in press). Mend it or end it: Redirecting the search for interactions in the organizational sciences. Organizational Research Methods.
40.
NyeC. D. (2011). The development and evaluation of effect size measures for CFA and IRT studies of measurement equivalence (Doctoral dissertation). University of Illinois at Urbana-Champaign.
41.
NyeC. D.BrummelB. J.DrasgowF. (2010). Too good to be true? Understanding change in organizational outcomes. Journal of Management, 36, 1555–1577.
42.
NyeC. D.DrasgowF. (2011). Effect size indices for analyses of measurement equivalence: Understanding the practical importance of differences between groups. Journal of Applied Psychology, 96, 966–980.
43.
OakesM. L. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York, NY: Wiley.
44.
RigerS. (1991). Gender dilemmas in sexual harassment policies and procedures. American Psychologist, 46, 497–505.
45.
RotundoM.SackettP. R. (1999). Effect of rater race on conclusions regarding differential prediction in cognitive ability tests. Journal of Applied Psychology, 84, 815–822.
46.
SackettP. R.LaczoR. M.LippeZ. P. (2003). Differential prediction and the use of multiple predictors: The omitted variables problem. Journal of Applied Psychology, 88, 1046–1056.
47.
SackettP. R.YangH. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112–118.
48.
SchmidtF. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115–129.
49.
Society for Industrial and Organizational Psychology. (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green, OH: Author.
50.
StarkS.ChernyshenkoO. S.DrasgowF. (2004). Examining the effects of differential item (functioning and differential) test functioning on selection decisions: When are statistically significant effects practically important?Journal of Applied Psychology, 89, 497–508.
51.
StarkS.ChernyshenkoO. S.LancasterA. R.DrasgowF.FitzgeraldL. F. (2002). Toward standardized measurement of sexual harassment: Shortening the SEQ-DoD using item response theory. Military Psychology, 14, 49–72.
52.
ThompsonB. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837–847.
53.
WilkinsonL. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.
54.
WillnessC. R.SteelP.LeeK. (2007). A meta-analysis of the antecedents and consequences of workplace sexual harassment. Personnel Psychology, 60, 127–162.
55.
YoungJ. W. (2001). Differential validity, differential prediction, and college admissions testing: A comprehensive review and analysis (Research Report 2001-06). New York, NY: The College Board.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
