The current article notes that the standard application of relative importance analyses is not appropriate when examining the relative importance of interactive or other higher order effects (e.g., quadratic, cubic). Although there is a growing demand for strategies that could be used to decompose the predicted variance in regression models containing such effects, there has been no formal, systematic discussion of whether it is appropriate to use relative importance statistics in such decompositions, and if it is appropriate, how to go about doing so. The purpose of this article is to address this gap in the literature by describing three different yet related strategies for decomposing variance in higher-order multiple regression models—hierarchical F tests (a between-sets test), constrained relative importance analysis (a within-sets test), and residualized relative importance analysis (a between- and within-sets test). Using a previously published data set, we illustrate the different types of inferences these three strategies permit researchers to draw. We conclude with recommendations for researchers seeking to decompose the predicted variance in regression models testing higher order effects.
AguinisH. (2004). Regression analysis for categorical moderators. New York, NY: Guilford.
2.
AguinisH.BeatyJ. C.BoikR. J.PierceC. A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90, 94–107.
3.
AguinisH.PierceC. A.BoscoF. A.MuslinI. S. (2008). First decade of Organizational Research Methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12, 69–112.
4.
AikenL. S.WestS. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.
5.
AustinJ. T.ScherbaumC. A.MahlmanR. A. (2002). History of research methods in industrial and organizational psychology: Measurement, design, analysis. In RogelbergS. G. (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 3–33). Malden, MA: Blackwell.
6.
AzenR.BudescuD. V. (2003). The dominance analysis approach for comparing predictors in multiple regression. Psychological Methods, 8, 129–148.
7.
BingM. N.LeBretonJ. M.DavisonH. K.MigetzD. Z.JamesL. R. (2007). Integrating implicit and explicit social cognitions for enhanced personality assessment: A general framework for choosing measurement and statistical methods. Organizational Research Methods, 10, 346–389.
8.
BudescuD. V. (1993). Dominance analysis: A new approach to the problem of relative importance of predictors in multiple regression. Psychological Bulletin, 114, 542–551.
9.
BudescuD. V.AzenR. (2004). Beyond global measures of relative importance: Insights from dominance analysis. Organizational Research Methods, 7, 341–350.
10.
CasperW. J.EbyL. T.BordeauxC.LockwoodA.LambertD. (2007). A review of research methods in IO/OB work-family research. Journal of Applied Psychology, 92, 28–43.
11.
CohenJ.CohenP.WestS. G.AikenL. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
12.
ColquittJ. A.LePineJ. A.PiccoloR. F.ZapataC. P.RichB. L. (2012). Explaining the justice-performance relationship: Trust as exchange deepener or trust as uncertainty reducer. Journal of Applied Psychology, 97, 1–15.
13.
CortinaJ. M. (1993). Interaction, nonlinearity, and multicollinearity: Implications for multiple regression. Journal of Management, 19, 915–922.
14.
CortinaJ. M.LandisR. S. (2009). When small effect sizes tell a big story, and when large effect sizes don’t. In LanceC. E.VandenbergR. J. (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 287–308). New York, NY: Routledge.
15.
CourvilleT.ThompsonB. (2001). Use of structure coefficients in published multiple regression articles: β is not enough. Educational and Psychological Measurement, 61, 229–248.
16.
DarlingtonR. B. (1968). Multiple regression in psychological research and practice. Psychological Bulletin, 69, 161–182.
17.
EbyL. T.ButtsM.LockwoodA. (2003). Predictors of success in the era of the boundaryless career. Journal of Organizational Behavior, 24, 689–708.
18.
EdwardsJ. R. (1994). The study of congruence in organizational behavior research: Critique and proposed alternative. Organizational Behavior and Human Decision Processes, 58, 51–100.
19.
EdwardsJ. R. (2002). Alternatives to difference scores: Polynomial regression analysis and response surface methodology. In DrasgowF.SchmittN. W. (Eds.), Advances in measurement and data analysis (pp. 350–400). San Francisco, CA: Jossey-Bass.
20.
EdwardsJ. R.ParryM. E. (1993). On the use of polynomial regression equations as an alternative to difference scores in organizational research. Academy of Management Journal, 36, 1577–1613.
GanzachY. (1998). Nonlinearity, multicollinearity and the probability of Type II error in detecting interaction. Journal of Management, 24, 615–622.
23.
GlombT. A.Kammeyer-MuellerJ. D.RotundoM. (2004). Emotional labor demands and compensating wage differentials. Journal of Applied Psychology, 89, 700–714.
24.
JacksonC. L.ColquittJ. A.WessonM. J.Zapata-PhelanC. P. (2006). Psychological collectivism: A measurement validation and linkage to group member performance. Journal of Applied Psychology, 91, 884–899.
25.
JamesL. R. (1998). Measurement of personality via conditional reasoning. Organizational Research Methods, 1, 131–163.
26.
JamesL. R.MulaikS. A.BrettJ. M. (1982). Causal analysis: Assumptions, models, and data. Beverly Hills, CA: Sage.
27.
JohnsonJ. W. (2000). A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivariate Behavioral Research, 35, 1–19.
28.
JohnsonJ. W. (2001). The relative importance of task and contextual performance dimensions to supervisor judgments of overall performance. Journal of Applied Psychology, 86, 984–996.
29.
JohnsonJ. W. (2004). Factors affecting relative weights: The influence of sampling and measurement error. Organizational Research Methods, 7, 283–299.
30.
JohnsonJ. W.LeBretonJ. M. (2004). History and use of relative importance indices in organizational research. Organizational Research Methods, 7, 238–257.
31.
JohnsonR. E.TolentinoA. L.RodopmanO. B.ChoE. (2010). We (sometimes) know not how we feel: Predicting job performance with an implicit measure of trait affectivity. Personnel Psychology, 63, 197–219.
32.
KrasikovaD.LeBretonJ. M.TonidandelS. (2011). Estimating the relative importance of variables in multiple regression models. In HodgkinsonG. P.FordJ. K. (Eds.), International review of industrial and organizational psychology (pp. 119–141). Indianapolis, IN: John Wiley.
33.
LeBretonJ. M.BinningJ. F.AdornoA. J.MelcherK. M. (2004). Importance of personality and job-specific affect for predicting job attitudes and withdrawal behaviors. Organizational Research Methods, 7, 300–325.
34.
LeBretonJ. M.HargisM. B.GriepentrogB.OswaldF. L.PloyhartR. E. (2007). A multidimensional approach for evaluating variables in organizational research and practice. Personnel Psychology, 60, 475–498.
35.
LeBretonJ. M.PloyhartR. E.LaddR. T. (2004). A Monte Carlo comparison of relative importance methodologies. Organizational Research Methods, 7, 258–282.
36.
LeBretonJ. M.TonidandelS. (2008). Multivariate relative importance: Extending relative weight analysis to multivariate criterion spaces. Journal of Applied Psychology, 93, 329–345.
37.
LievensF.ConwayJ. M.De CorteW. (2008). The relative importance of task, citizenship and counterproductive performance to job performance ratings: Do rater source and team-based culture matter?Journal of Occupational and Organizational Psychology, 81, 11–27.
38.
LievensF.HighhouseS.De CorteW. (2005). The importance of traits and abilities in supervisors’ hirability decisions as a function of method of assessment. Journal of Occupational and Organizational Psychology, 78, 453–470.
39.
LindemanR. H.MerendaP. F.GoldR. Z. (1980). Introduction to bivariate and multivariate analysis. Glenview, IL: Scott, Foresman.
40.
LubinskiD.HumphreysL. G. (1990). Assessing spurious “moderator effects”: Illustrated substantively with the hypothesized (“synergistic”) relation between spatial and mathematical ability. Psychological Bulletin, 107, 385–393.
41.
LuoY. D. (2005). How important are shared perceptions of procedural justice in cooperative alliances?Academy of Management Journal, 48, 695–709.
42.
MacCallumR. C.MarC. M. (1995). Distinguishing between moderator and quadratic effects in multiple regression. Psychological Bulletin, 118, 405–421.
43.
MartellR. F.LaneD. M.EmrichC. (1996). Male-female differences: A computer simulation. American Psychologist, 51, 157–158.
44.
McAllisterD. J.KamdarD.MorrisonE. W.TurbanD. B. (2007). Disentangling role perceptions: How perceived role breadth, discretion, instrumentality, and efficacy relate to helping and taking charge. Journal of Applied Psychology, 92, 1200–1211.
45.
MorgesonF. P.NahrgangJ. D. (2008). Same as it ever was: Recognizing stability in the BusinessWeek rankings. Academy of Management Learning and Education, 7, 26–41.
46.
MyersR. H. (1990). Classical and modern regression with applications (2nd ed.). Pacific Grove, CA: Brooks/Cole.
47.
NickersonC.SchwarzN.DienerE.KahnemanD. (2003). Zeroing in on the dark side of the American dream: A closer look at the negative consequences of the goal for financial success. Psychological Science, 14, 531–536.
48.
PedhazurE. J. (1997). Multiple regression in behavioral research: Explanation and prediction (3rd ed.). New York, NY: Harcourt Brace.
49.
PloyhartR. E.HakelM. D. (1998). The substantive nature of performance variability: Predicting interindividual differences in intraindividual performance. Personnel Psychology, 51, 859–901.
50.
PodsakoffP. M.DaltonD. R. (1987). Research methodology in organizational studies. Journal of Management, 13, 419–441.
51.
ScanduraT. A.WilliamsE. A. (2000). Research methodology in management: Current practices, trends, and implications for future research. Academy of Management Journal, 53, 1248–1264.
52.
SchleicherD. J.VenkataramaniV.MorgesonF. P.CampionM. A. (2006). So you didn’t get the job . . . now what do you think? Examining opportunity-to-perform fairness perceptions. Personnel Psychology, 59, 559–590.
53.
ShepperdJ. A. (1991). Cautions in assessing spurious “moderator effects.”Psychological Bulletin, 110, 315–317.
54.
Stone-RomeroE. F.WeaverA. E.GlenarJ. L. (1995). Trends in research design and data analytic strategies in organizational research. Journal of Management, 21, 141–157.
55.
TonidandelS.LeBretonJ. M. (2010). Determining the relative importance of predictors in logistic regression: An extension of relative weights analysis. Organizational Research Methods, 13, 767–781.
56.
TonidandelS.LeBretonJ. M. (2011). Relative importance analysis: A useful supplement to regression analysis. Journal of Business and Psychology, 26, 1–9.
57.
TonidandelS.LeBretonJ. M.JohnsonJ. W. (2009). Statistical significance tests for relative weights. Psychological Methods, 14, 387–399.
58.
Van IddekingeC. H.PloyhartR. E. (2008). Developments in the criterion-related validation of selection procedures: A critical review and recommendations for practice. Personnel Psychology, 61, 871–925.
59.
WilliamsE. J. (1978). Linear hypotheses: Regression. In KruskalW. H.TanurJ. M. (Eds.), International encyclopedia of statistics (pp. 523–541). New York, NY: Free Press.
