Employee selection often involves a series of sequential tests (or hurdles). However, validation strategies under this complex design are not found in the literature. Missing is a discussion of the statistical properties important in establishing criterion-related validity in multiple-hurdle designs. The authors address this gap in the literature by suggesting a general statistical model for range restriction corrections. Because the multiple-hurdle design includes as special cases predictive and concurrent designs, the corrections apply also to these designs. The general correction model is based on algorithms from the missing data literature. Two missing data procedures are examined: the estimation-maximization procedure and the Bayesian multiple imputation (MI) procedure. These procedures are large-sample equivalent and often yield similar results. The MI procedure, however, has the added advantage of providing easily obtainable standard errors. A hypothetical example of a multiple-hurdle design is used to illustrate the procedures.
Allison, P. D. (2001). Missing data. Thousand Oaks, CA: Sage.
2.
Amemiya, T. (1984). Tobit models: A survey. Journal of Econometrics, 24, 3-61.
3.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
4.
Bobko, P. (1995). Correlation and regression: Principles and applications for industrial/ organizational psychology and management. New York: McGraw-Hill.
5.
Chasteen, C. S., & Mendoza, J. L. (2003, April). Restriction in range issues in validation designs: Modern tools for old problems. Paper presented at the national meetings of the American Educational Research Association, Chicago, IL.
6.
Collins, L. M., Schafer, J. L., & Kam, C. M. (2001). A comparison of inclusive and restrictive strategies in modern missing-data procedures. Psychological Methods, 6, 330-351.
7.
Efron, B. (1994). Missing data, imputation, and the bootstrap. Journal of the American Statistical Association, 89, 463-475.
8.
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, 39(series B), 1-38.
9.
Fan, X., & Thompson, B. (2001). Confidence intervals about score reliability coefficients, please: An EPM guidelines editorial. Educational and Psychological Measurement, 61, 517-531.
10.
Ghiselli, E. E. (1964). Theory of psychological measurement. New York: McGraw-Hill.
11.
Ghiselli, E. E. (1966). The validity of occupational aptitude tests. New York: John Wiley.
12.
Graham, J. W., & Schafer, J. L. (1999). On the performance of multiple imputation for multivariate data with small sample size. In R. Hoyle (Ed.), Statistical strategies for small sample research(pp. 1-29). Thousand Oaks, CA: Sage.
13.
Greener, J. M.,& Osburn, H. G. (1979). An empirical study of the accuracy of corrections for restriction in range due to explicit selection. Applied Psychological Measurements, 3, 31-41.
14.
Greener, J. M., & Osburn, H. G. (1980). Accuracy of corrections for restriction in range due to explicit in heteroscedastic and nonlinear distributions. Educational and Psychological Measurement, 40, 337-346.
15.
Gross, A. L. (1990). A maximum likelihood approach to test validation with missing data and censored dependent variables. Psychometrika, 55, 533-549.
16.
Gross, A. L.,& McGanney, M. L. (1987). The restriction of range problem and nonignorable selection process. Journal of Applied Psychology, 72, 604-610.
17.
Heckman, J. (1976). The common structure of statistical models of truncation, sample selection and limited dependent variables, and a simple estimator for such models. Annals of Economics and Social Measurement, 5, 475-492.
18.
Hoffman, C. (1995). Applying range restriction corrections using published norms: Three case studies. Personnel Psychology, 48, 913-923.
19.
Horton, N. J.,& Lipsitz, S. R. (2001). Multiple imputation in practice: Comparison of software packages for regression models with missing variables. American Statistician, 55, 244-254.
20.
James, L. R., Demaree, R., Muliak, S. A.,& Mumford, M. D. (1988). Validity generalization: A rejoinder to Schmidt, Hunter, and Raju. Journal of Applied Psychology, 73, 443-452.
21.
Kolmstetter, E. (2003). I-Os making an impact: TSA transportation security screener skill standards, selection system, and hiring process. Industrial-Organizational Psychologist, 40(4), 39-46.
22.
Lawley, D. N. (1943). A note on Karl Pearson’s selection formulae. Proceedings of the Royal Society of Edinburgh, 62, 28-30.
23.
Li, K. H., Raghunathan, T. E., & Rubin, D. B. (1991). Large-sample significance levels from multiply imputed data using moment-based statistics and an F reference distribution. Journal of the American Statistical Association, 86, 1065-1073.
24.
Linn, R. L., Harnisch, D. L.,& Dunbar, S. B. (1981). Correction for range restriction: An empirical investigation of conditions resulting in conservative corrections. Journal of Applied Psychology, 66, 655-663.
25.
Linn, R. T. (1968). Range restriction problems in the use of self-selected groups for test validation. Psychological Bulletin, 69, 69-73.
26.
Little, R. J. A. (1995). Modeling the dropout mechanism in repeated-measures studies. Journal of the American Statistical Association, 90, 1112-1121.
27.
Little, R. J. A., & Rubin, D. B (1987). Statistical analysis with missing data. New York: John Wiley.
28.
Little, R. J. A.,& Schenker, N. (1995). Missing data. In Arminger, G., Clogg, C. C.,& Sobel, M. E. (Eds.), Handbook of statistical modeling for the social and behavioral sciences(pp. 39-75). New York: Plenum.
29.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
30.
McLachlan, G. J., & Krishnan, T. (1997). The EM algorithm and extensions. New York: John Wiley.
31.
Mendoza, J. L. (1993). Fisher transformations for correlations corrected for selection and missing data. Psychometrika, 58, 601-615.
32.
Mendoza, J. L., & Mumford, M. D. (1987). Corrections for attenuation and range restriction. Journal of Educational Statistics, 12, 282-293.
33.
Mendoza, J. L., Stafford, K. L.,& Stauffer, J. M. (2000). Large-sample confidence intervals for validity and reliability coefficients. Psychological Methods, 5, 356-369.
34.
Meng, X. L., & Rubin, D. B. (1991). Using the EM to obtain asymptotic variance-covariance matrices: The SEM algorithm. Journal of the American Statistical Association, 86, 899-909.
35.
Messick, S. J. (1998). Alternative models of assessment, uniform standards of validity. InM. D. Hakel (Ed.), Beyond multiple choice: Evaluating alternatives to traditional testing for selection(pp. 59-74). Mahwah, NJ: Lawrence Erlbaum.
36.
Olkin, I., & Finn, J. D. (1995). Correlation redux. Psychological Bulletin, 118, 155-164.
37.
Ones, D. S.,& Viswesvaran, C. (2003). Job-specific applicant pools and national norms for personality scales implications for range-restriction corrections in validation research. Journal of Applied Psychology, 88, 570-577.
38.
Pearson, K. (1903). Mathematical contributions to the theory of evolution XI: On this influence of natural selection on the variability and correlation of organs. Philosophical Transactions of the Royal Society, London, Series A, 200, 1-66.
39.
Rao, J. N. K.,& Shao, J. (1992). Jackknife variance estimation with survey data under hot-deck imputation. Biometrika, 79, 811-822.
40.
Ree, M. J., Carretta, T. R., Earles, J. A.,& Albert, W. (1994). Sign changes when correcting for range restriction: A note on Pearson’s and Lawley’s selection formulas. Journal of Applied Psychology, 79, 298-301.
41.
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581-592.
42.
Rubin, D. B. (1978). Multiple imputations in sample surveys—A phenomenological Bayesian approach to nonresponse. Proceedings of the Survey Research Methods Section American Statistical Association, 20-34.
43.
Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York: John Wiley.
44.
Rubin, D. B. (1996). Multiple imputation after 18+ years. Journal of the American Statistical Association, 91, 473-489.
45.
Sackett, P. R., Laczo, R. M.,& Arvey, R. D. (2002). The effects of range restriction on estimates of criterion interrater reliability: Implications for validation research. Personnel Psychology, 55, 807-825.
46.
Sackett, P. R.,& Ostgard, D. L. (1994). Job-specific applicant pools and national norms for cognitive ability tests: Implications for range restriction corrections in validation research. Journal of Applied Psychology, 79, 680-684.
47.
Sackett, P. R.,& Yang, H. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112-118.
48.
Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapma & Hall.
49.
Schafer, J. L.,& Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147-177.
50.
Schafer, J. L., & Olsen, M. K. (1998). Multiple imputation for multivariate missing-data problems: A data analyst’s perspective. Multivariate Behavioral Research, 33, 545-571.
51.
Schafer, J. L.,& Schenker, N. (2000). Inferences with imputed conditional means. Journal of the American Statistical Association, 95, 144-154.
52.
Switzer, F. S., Roth, P. L., & Switzer, D.M. (1998). Systematic data loss in HRM settings: A Monte Carlo analysis. Journal of Management, 24, 763-777.
53.
Tanner, M. A.,& Wong, W.H. (1987). The calculation of the posterior distributions by data augmentation. Journal of the American Statistical Association, 82, 528-550.
