Estimating Subgroup Differences in Staffing Research When the Selection Mechanism Is Unknown: A Response to Li’s Case IV Correction

Abstract

When estimating subgroup differences on incumbents, range restriction may bias estimates. Bobko, Roth, and Bobko recognized this problem and developed a Case II and Case III correction for Cohen’s d. Subsequently, Li developed a Case IV correction, which seeks to estimate group differences on a predictor using only incumbent data but must assume that group membership (e.g., ethnicity) plays no role in selection decisions. In this paper, we extend Li’s correction and relax this assumption. In addition, this new correction allows for the estimation of subgroup differences on both the criterion and predictor. Using Monte Carlo simulation, we study the performance of both estimators under situations where Li’s assumptions are violated and demonstrate that this new procedure almost always outperforms Li’s Case IV correction and does so with greater precision. We also provide R code to assist applied researchers in using these corrections.

Keywords

missing data quantitative research criterion and predictive validity strategies reliability and validity ANOVA methods

Get full access to this article

View all access options for this article.

References

Amatya

Demirtas

(2016). BinNor: Simultaneous generation of multivariate binary and normal variates. R package version 2.1. Retrived from https://rdrr.io/cran/BinNor/

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Bobko

Roth

P. L.

Bobko

(2001). Correcting the effect size of d for range restriction and unreliability. Organizational Research Methods, 4(1), 46–61.

Carlsson

Rooth

D.-O

. (2008). Is it your foreign name or foreign qualifications? An experimental study of ethnic discrimination in hiring (IZA Discussion Paper No. 3810) Retrieved from https://www.iza.org/publications/dp/3810/is-it-your-foreign-name-or-foreign-qualifications-an-experimental-study-of-ethnic-discrimination-in-hiring

Collins

L. M.

Schafer

J. L.

Kam

C. M.

(2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330–351.

Dempster

A. P.

Laird

N. M.

Rubin

D. B.

(1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39(1), 1–38.

Derous

Ryan

A. M.

Nguyen

H.-H. D.

(2012). Multiple categorization in resume screening: Examining effects on hiring discrimination against Arab applicants in field and lab settings. Journal of Organizational Behavior, 33(4), 544–570.

Fife

D. A.

Hunter

M. D.

Mendoza

J. L.

(2016). Estimating unattenuated correlations with limited information about selection variables: Alternatives to Case IV. Organizational Research Methods, 19(4), 593–615.

Fife

D. A.

Mendoza

J. L.

Terry

(2012). The assessment of reliability under range restriction: A comparison of α, ω, and test-retest reliability for dichotomous data. Educational and Psychological Measurement, 72(5), 862–888.

10.

Fife

D. A.

Mendoza

J. L.

Terry

(2013). Revisiting Case IV: A reassessment of bias and standard errors of Case IV. British Journal of Mathematical and Statistical Psychology, 66(3), 521–542.

11.

Held

J. D.

Foley

P. P.

(1994). Explanations for accuracy of the general multivariate formulas in correcting for range restriction. Applied Psychological Measurement, 18(4), 355–367.

12.

Hunter

J. E.

Schmidt

F. L.

(2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.

13.

Hunter

J. E.

Schmidt

F. L.

(2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91(3), 594–612.

14.

King

E. B.

Mendoza

S. A.

Madera

J. M.

Hebl

M. R.

Knight

J. L.

(2006). What’s in a name? A multiracial investigation of the role of occupational stereotypes in selection decisions. Journal of Applied Social Psychology, 36(5), 1145–1159.

15.

J. C.

(2015). Cohen’s d corrected for Case IV range restriction: A more accurate procedure for evaluating subgroup differences in organizational research. Personnel Psychology, 68(4), 899–927.

16.

McKay

P. F.

McDaniel

M. A.

(2006). A reexamination of Black-White mean differences in work performance: More data, more moderators. Journal of Applied Psychology, 91(3), 538–554.

17.

Meehl

P. E.

(1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108–141.

18.

Mendoza

J. L.

Mumford

(1987). Corrections for attenuation and range restriction on the predictor. Journal of Educational Statistics, 12(3), 282–293.

19.

Oreopoulos

(2011). Why do skilled immigrants struggle in the labor market? A field experiment with thirteen thousand resumes. American Economic Journal: Economic Policy, 3(4), 148–171.

20.

Pearl

(2000). Causality. Chicago, IL: Cambridge University Press.

21.

R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/

22.

Roth

P. L.

Huffcutt

A. I.

Bobko

(2003). Ethnic group differences in measures of job performance: A new meta-analysis. Journal of Applied Psychology, 88(4), 694–706.

23.

Roth

P. L.

Switzer

F. S.

Van Iddekinge

C. H.

In-Sue

(2011). Toward better meta-analytic matrices: How input values can affect research conclusions in human resource management simulations. Personnel Psychology, 64(4), 899–935.

24.

Rothstein

H. R.

(1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75(3), 322–327.

25.

Sackett

P. R.

Schmitt

Ellingson

J. E.

Kabin

M. B.

(2001). High-stakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative-action world. American Psychologist, 56(4), 302–318.

26.

Schmidt

F. L.

In-Sue

(2006). Increasing the accuracy of corrections for range restriction: Implications for selection procedure validities and other research results. Personnel Psychology, 59(2), 281–305.

27.

SIOP. (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green, OH: Society for Industrial Organizational Psychology.

28.

Thoemmes

Rose

(2014). A cautious note on auxiliary variables that can increase bias in missing data problems. Multivariate Behavioral Research, 49(5), 443–459.

29.

Thorndike

R. L.

(1949). Personnel selection; test and measurement techniques. Oxford, UK: Wiley.

30.

Viswesvaran

Ones

D. S.

Schmidt

F. L.

(1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81(5), 557–574.

31.

Wright

(1918). Color inheritance in mammals: IX, the dog—Many kinds of white patterns found—Albinism resembles that of other mammals in reducing red more than black—Inheritance of black and tan requires further data—Red and liver simple recessives. Journal of Heredity, 9(2), 87–90.