Predicting outcomes is critical in many domains of organizational research and practice. Over the past few decades, there have been substantial advances in predictive modeling methods and concepts from the computer science, machine learning, and statistics literatures that may have potential value for organizational science and practice. Nevertheless, treatment of these modern methods in major management and industrial-organizational psychology journals remains minimal. The purpose of this article is to (a) raise awareness among organizational researchers and practitioners with regard to several modern prediction methods and concepts, (b) discuss in nonmathematical terms how they compare to traditional regression-based prediction methods, and (c) provide an empirical example of their application and performance relative to traditional methods. Beyond illustrating their potential for improving prediction, we will also illustrate how these methods can offer deeper insights into how predictor content functions beyond simple construct-based explanations.
AguinisH.BradleyK. J.BrodersenA. (2014). Industrial-organizational psychologists in business schools: Brain drain or eye opener?Industrial and Organizational Psychology: Perspectives on Science and Practice, 7, 284–303.
2.
AguinisH.PierceC. A.BoscoF. A.MuslinI. S. (2009). First decade of organizational research methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12, 69–112.
3.
AguinisH.PierceC. A.CulpepperS. A. (2009). Scale coarseness as a methodological artifact: Correcting correlation coefficients attenuated from using coarse scales. Organizational Research Methods, 12, 623–652.
4.
AntonakisJ.BendahanS.JacquartP.LaliveR. (2010). On making casual claims: A review and recommendations. Leadership Quarterly, 21, 1086–1120.
5.
AustinJ. T.ScherbaumC. A.MahlmanR. A. (2002). History of research methods in industrial and organizational psychology: Measurement, design, analysis. In RogelbergS. G. (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 1–33). Oxford, UK: Blackwell.
6.
BerkR. A. (2006). An introduction to ensemble methods for data analysis. Sociological Methods & Research, 34, 263–295.
BigaA. (2014). Using big and small data to make better people decisions. Presentation to the Personnel Testing Council of Metropolitan Washington, Washington, DC.
9.
BingM. N.WhangerJ. C.DavisonH. K.VanHookJ. B. (2004). Incremental validity of the frame-of-reference effect in personality scale scores: A replication and extension. Journal of Applied Psychology, 89, 150–157
BreimanL. (2001). Random forests. Machine Learning, 45, 5–32.
12.
BreimanL.FriedmanJ. H.OlshenR. A.StoneC. J. (1984). Classification and regression trees. New York, NY: Chapman & Hall.
13.
CampionM. C.PloyhartR. E. (2013). Assessing personality with situational judgment measures: Interactionist psychology operationalized. In ChristiansenN. D.TettR. P. (Eds.), Handbook of personality at work (pp. 439–456). New York, NY: Routledge.
14.
CohenJ. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249–253.
15.
CucinaJ. M.CaputoP. M.ThibodeauxH. F.MacLaneC. N. (2012). Unlocking the key to biodata scoring: A comparison of empirical, rational, and hybrid approaches at different sample sizes. Personnel Psychology, 65, 385–428.
16.
DeanM. A.RussellC. J. (2005). An examination of biodata theory-based constructs in a field context. International Journal of Selection and Assessment, 13, 139–149.
17.
EdwardsJ. R. (1994). The study of congruence in organizational behavior research: Critique and a proposed alternative. Organizational Behavior and Human Decision Processes, 58, 51–100.
18.
EfronB.HastieT.JohnstoneI.TibshiraniR. (2004). Least angle regression. The Annals of Statistics, 32, 407–499.
19.
ElithJ.LeathwickJ. R.HastieT. (2008). A working guide to boosted regression trees. Journal of Animal Ecology, 77, 802–813.
20.
FleesonW. (2007). Situation-based contingencies underlying trait-content manifestation in behavior. Journal of Personality, 75, 825–862.
21.
FreundY.SchapireR. (1996). Experiments with a new boosting algorithm. In Machine Learning: Proceedings of the Thirteenth International Conference (pp. 148–156). San Francisco, CA: Morgan Kaufmann.
FriedmanJ. (2001). Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29, 1189–1232.
24.
FriedmanJ. (2002). Stochastic gradient boosting. Computational Statistics and Data Analysis, 38, 367–378.
25.
GibbyR. E.McCloyR. A.PutkaD. J. (2013, 4). Viewing linkage research through the lenses of current practice and cutting-edge advances. Invited pre-conference workshop presented at the 28th Annual Society for Industrial and Organizational Psychology Conference, Houston, TX.
26.
HambrickD. C. (2007). The field of management’s devotion to theory: Too much of a good thing?Academy of Management Journal, 50, 1346–1352.
27.
HarrellF. EJr. (2001). Regression modeling strategies: With applications to linear models, logistic regression, and survival analysis. New York, NY: Springer.
28.
HastieT.TibshiraniR. (1990). Generalized additive models. London: Chapman and Hall.
29.
HastieT.TibshiraniR.FriedmanJ. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). New York, NY: Springer.
30.
HelfatC. E. (2007). Stylized facts, empirical research and theory development in management. Strategic Organization, 5, 185–192.
31.
HoffmanB. J.MelchersK.BlairC. A.KleinmannM.LaddR. T. (2011). Exercises and dimensions are the currency of assessment centers. Personnel Psychology, 6, 351–395.
32.
HoffmanB. J.WoehrD. J. (2009). Disentangling the meaning of multisource performance rating source and dimension factors. Personnel Psychology, 62, 735–765.
33.
JamesG.WittenD.HastieT.TibshiraniR. (2013). An introduction to statistical learning with applications in R. New York, NY: Springer.
34.
JohnsonJ. W.LeBretonJ. M. (2004). History and use of relative importance indices in organizational research. Organizational Research Methods, 7, 238–257.
35.
KilcullenR. N.RobbinsJ.TrembleT. R. (2009). Development of the CBEF. In PutkaD. J. (Ed.), Initial development and validation of assessments for predicting disenrollment of four-year scholarship recipients from the Reserve Office Training Corps (ARI study report No. 2009-06, pp. 5–10). Arlington, VA: U. S. Army Research Institute for the Behavioral and Social Sciences.
36.
KoppesL. L. (Ed.) (2007). Historical perspectives in industrial-organizational psychology. Mahwah, NJ: Lawrence Erlbaum Associates.
KuhnM.JohnsonK. (2013). Applied predictive modeling. New York, NY: Springer.
39.
LanceC. E. (2008). Why assessment centers do not work the way they are supposed to. Industrial and Organizational Psychology, 1, 84–97.
40.
LanceC. E.HoffmanB. J.GentryB.BaranikL. E. (2008). Rater source factors represent important subcomponents of the criterion construct space, not rater bias. Human Resource Management Review, 18, 223–232.
41.
LegreeP. J.KilcullenR. N.PutkaD. J.WaskoL. E. (2014). Identifying leaders of tomorrow: Validating predictors of leader performance. Military Psychology, 26, 292–309.
42.
LievensF.De CorteW.SchollaertE. (2008). A closer look at the frame-of-reference effect in personality scale scores and validity. Journal of Applied Psychology, 93, 268–279.
43.
LockeE. A. (2007). The case for inductive theory building. Journal of Management, 33, 867–890.
44.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
45.
MaelF. A. (1991). A conceptual rationale for the domain and attributes of biodata items. Personnel Psychology, 44, 763–792.
46.
MischelW.ShodaY. (1995). A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychological Review, 102, 246–268.
47.
MurthaT.C.KanferR.AckermanP. L. (1996). Toward an interactionist taxonomy of personality and situations: An integrative situational-dispositional representation of personality traits. Journal of Personality and Social Psychology, 71, 193–207.
48.
NisbetR.ElderJ.MinerG. (2009). Handbook of statistical analysis and data mining applications. Burlington, MA: Elsevier.
49.
OswaldF. L.PutkaD. J.OckJ. (2014). Weight a minute, what you see in a weighted composite is probably not what you get. In LancC. E.VandenbergR. J. (Eds.), More statistical and methodological myths and urban legends (pp 187–205). New York, NY: Taylor & Francis.
50.
OwensW. A. (1976). Background data. In DunnetteM. D. (Ed.), Handbook of industrial and organizational psychology (pp. 609–644). Chicago, IL: Rand-McNally.
51.
ParkG.SchwartzH. A.EichstaedtJ. C.KernM. L.KowsinskiM.StillwellD. J.…SeligmanM. E. (2015). Automatic personality assessment through social media language. Journal of Personality and Social Psychology, 108, 934–952.
52.
PutkaD. J.OswaldF. (2015). Implications of the Big Data movement for the advancement I-O science and practice. In TonidandelS.KingE.CortinaJ. (Eds.), Big data at work: The data science revolution and organizational psychology (pp. 181–212). New York, NY: Taylor & Francis.
53.
RajuN. S.BilgicR.EdwardsJ. E.FleerP. F. (1997). Methodological review: Estimation of population validity and cross-validity, and the use of equal weights in prediction. Applied Psychological Measurement, 21, 291–305.
R Core Development Team. (2007). R: A language and environment for statistical computing. R Vienna: Foundation for Statistical Computing.
56.
SackettP. R.DreherG. F. (1982). Constructs and assessment center dimensions: Some troubling empirical findings. Journal of Applied Psychology, 67, 401–410.
57.
SchmidtF. L.HunterJ. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199–223.
58.
SchmittN.PloyhartR. E. (1999). Estimates of cross-validity for stepwise regression and with predictor selection. Journal of Applied Psychology, 84, 50–57.
59.
ShapiroD. L.KirkmanB. L.CourtneyH. G. (2007). Perceived causes and solutions of the translation problem in management research. Academy of Management Journal, 50, 249–266.
60.
ShenW.KigerT. B.DaviesS. E.RaschR. L.SimonK. M.OnesD. S. (2011). Samples in applied psychology: Over a decade of research in review. Journal of Applied Psychology, 96, 1055–1064.
61.
SilzerR.ParsonC. (2012). Industrial-organizational psychology journals and the science-practice gap. The Industrial-Organizational Psychologist, 49, 97–117.
62.
SpeerA. B.ChristiansenN. D.GoffinR. D.GoffM. (2014). Situational bandwidth and the criterion-related validity of assessment center ratings: Is cross-exercise convergence always desirable?Journal of Applied Psychology, 99, 282–295.
63.
StokesG. S.MumfordM. D.OwensW. A. (Eds.). (1994). Biodata handbook: Theory, research, and use of biographical information in selection and performance prediction. Palo Alto, CA: CPP Books.
64.
StroblC.MalleyJ.TutzG. (2009). An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods, 14, 323–348.
65.
SuttonC. D. (2005). Classification and regression trees, bagging, and boosting. Handbook of Statistics, 24, 303–329.
66.
ThomasD. R.ZumboB. D.KwanE.SchweitzerL. (2014). On Johnson’s (2000) relative weights method for assessing variable importance: A re-analysis. Multivariate Behavioral Research, 49, 329–338.
67.
TibshiraniR. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, B, 58, 267–288.
68.
TonidandelS.KingE.CortinaJ. (Eds.). (2015). Big data at work: The data science revolution and organizational psychology. New York, NY: Routledge.
69.
VapnikV. (1998). Statistical learning theory. New York, NY: John Wiley and Sons, Inc.
70.
YouyouW.KosinskiM.StillwellD. (2015). Computer-based personality judgments are more accurate than those made by humans. Proceedings of the National Academy of Sciences of the United States of America, 112, 1036–1040.
71.
ZouH.HastieT. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, B, 67, 301–320.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
