Most study samples show less variability in key variables than do their source populations due most often to indirect selection into study participation associated with a wide range of personal and circumstantial characteristics. Formulas exist to correct the distortions of population-level correlations created. Formula accuracy has been tested using simulated normally distributed data, but empirical data are rarely available for testing. We did so in a rare data set in which it was possible: the 6-Day Sample, a representative subsample of 1,208 from the Scottish Mental Survey 1947 of cognitive ability in 1936-born Scottish schoolchildren (70,805). 6-Day Sample participants completed a follow-up assessment in childhood and were re-recruited for study at age 77 years. We compared full 6-Day Sample correlations of early-life variables with those of the range-restricted correlations in the later-participating subsample, before and after adjustment for direct and indirect range restriction. Results differed, especially for two highly correlated cognitive tests; neither reproduced full-sample correlations well due to small deviations from normal distribution in skew and kurtosis. Maximum likelihood estimates did little better. To assess these results’ typicality, we simulated sample selection and made similar comparisons using the 42 cognitive ability tests administered to the Minnesota Study of Twins Reared Apart, with very similar results. We discuss problems in developing further adjustments to offset range-restriction distortions and possible approaches to solutions.
AlexanderR. A. (1990). Correction formulas for correlations restricted by selection on an unmeasured variable. Journal of Educational Measurement, 27, 187-189.
2.
AlexanderR. A.CarsonK. P.AlligerG. M.CronshawS. F. (1989). Empirical distributions of range restricted SDx in validity studies. Journal of Applied Psychology, 74, 253-258.
3.
BerryC. M.SackettP. R. (2007). Revisiting interview-cognitive ability relationships: Attending to specific range restriction mechanisms in meta-analysis. Personnel Psychology, 60, 837-874.
4.
BerryC. M.ZhaoP. (2015). Addressing criticisms of existing predictive bias research: Cognitive ability test scores still overpredict African-Americans’ job performance. Journal of Applied Psychology, 100, 162-179.
5.
BobkoP.RothP. L.BobkoC. (2001). Correcting the effect size of d for range restriction and unreliability. Organizational Research Methods, 4, 46-61.
6.
BotellaJ.SueroM.GambaraH. (2010). Psychometric inferences form a meta-analysis of reliability and internal consistency coefficients. Psychological Methods, 15, 386-397.
7.
BrettC. E.DearyI. J. (2014). Realising health data linkage from a researcher’s perspective: Following up the 6-Day Sample of the Scottish Mental Survey 1947. Longitudinal and Life Course Studies, 5, 283-298.
8.
BryantN. D.GokhaleS. (1972). Correcting correlations for range restriction due to selection on an unmeasured variable. Educational and Psychological Measurement, 32, 305-310.
9.
ChernyshenkoO. S.OnesD. S. (1999). How selective are psychology graduate programs? The effect of the selection ratio on GRE score validity. Educational and Psychology Measurement, 59, 951-961.
10.
DearyI. J.BrettC. E. (2015). Predicting and retrodicting intelligence between childhood and old age in the 6-Day Sample of the Scottish Mental Survey 1947. Intelligence, 50, 1-9.
11.
DearyI. J.PattieA.StarrJ. M. (2013). The stability of intelligence from age 11 to age 90 years: The Lothian Birth Cohort of 1921. Psychological Science, 24, 2361-2368.
12.
DearyI. J.WhitemanM. C.StarrJ. M.WhalleyL. J.FoxH. C. (2004). The impact of childhood intelligence on later life: Following up the Scottish Mental Surveys of 1932 and 1947. Journal of Personality and Social Psychology, 86, 130-147.
13.
DeFriesJ. C.VandenbergS. G.McClearnG. E.KuseA. R.WilsonJ. R.AshtonG. C.JohnsonR. E. (1974). Near identity of cognitive structure in two ethnic groups. Science, 183, 338-339.
14.
DesaiM. (1952). The test-retest reliability of the progressive matrices test. British Journal of Medical Psychology, 25, 48-53.
15.
EndersC. (2011). Missing not at random models for latent growth curve analyses. Psychological Methods, 16, 1-16.
16.
GlassG. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
17.
GlassG. V. (1982). Meta-analysis: An approach to the synthesis of research results. Journal of Research in Science Teaching, 19, 93-112.
18.
GreenerJ. M.OsburnH. G. (1980). Accuracy of corrections for restriction in range due to explicit selection in heteroscedastic and nonlinear distributions. Educational and Psychological Measurement, 40, 337-346.
19.
GrossA. L.FleischmanL. (1983). Restriction of range corrections when both distribution and selection assumption is violated. Applied Psychological Measurement, 7, 227-237.
20.
HagglundG.LarssonR. (2006). Estimation of the correlation coefficient based on selected data. Journal of Educational and Behavioral Statistics, 31, 377-411.
21.
HakstianA. R.BennettR. W. (1977). Validity studies using the Comprehensive Ability Battery (CAB) I: Academic achievement criteria. Education and Psychological Measurement, 37, 425-437.
22.
HakstianA. R.CattellR. B. (1975). The comprehensive ability battery. Champaign, IL: Institute for Personality and Ability Testing.
23.
HansonR. K.HunsleyJ.ParkerK. H. (1988). The relationship between WAIS subtest reliability, g-loadings, and meta-analytically derived validity estimates. Journal of Clinical Psychology, 44, 557-562.
24.
HoffmanC. C. (1995). Applying range restriction corrections using published norms: Three case studies. Personnel Psychology, 48, 913-922.
25.
HunterJ. E.SchmidtF. L.LeH. (2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91, 594-612.
26.
JohnsonW.BouchardT. J. (2011). The MISTRA data: Forty-two mental ability tests in three batteries. Intelligence, 39, 82-88.
27.
JohnsonW.BouchardT. J.McGueM.SegalN. L.TellegenA.KeyesM.GottesmanI. I. (2007). Genetic and environmental influences on the Verbal-Perceptual-Image Rotation (VPR) model of the structure of mental ability in the Minnesota Study of Twins Reared Apart. Intelligence, 35, 542-562.
28.
JohnsonW.BrettC. E.CalvinC.DearyI. J. (2016). Childhood characteristics and participation in Scottish Mental Survey 1947 Follow-Up Study: Implications for participation in aging studies. Intelligence, 54, 70-79.
29.
JohnsonW.CorleyJ.StarrJ. M.DearyI. J. (2011). Psychological and physical health at age 70 in the Lothian Birth Cohort 1936: Links with early life IQ, SES, and current cognitive function and home environment. Health Psychology, 30, 1-11.
30.
KuseA. R. (1977). Familial resemblances for cognitive abilities estimated from two test batteries in Hawaii (Unpublished doctoral dissertation). University of Colorado at Boulder, Boulder, CO.
31.
LawleyD. N. (1943). A note on Karl Pearson’s selection formulae. Proceedings of the Royal Society of Edinburgh, 62, 28-30.
32.
LeH.SchmidtF. L. (2006). Correcting for indirect range restriction in meta-analysis: Testing a new meta-analytic procedure. Psychological Methods, 11, 416-438.
33.
LittleR. J.RubinD. (1987). Statistical analysis with missing data. New York, NY: Wiley.
34.
LittmanA. J.WhiteE.SatiaJ. A.BowenD. J.KristalR. (2006). Reliability and validity of two single items of psychosocial stress. Epidemiology, 17, 398-403.
35.
LonnqvistJ. E.PaunonenS.VerkasaloM.LeikasS.Tuulio-HenrikssonA.LonnqvistJ. (2007). Personality characteristics of research volunteers. European Journal of Personality, 21, 1017-1030. doi:10.1002/per655
36.
MacPhersonJ. S. (1958). Eleven-year-olds grow up. London, England: University of London Press.
37.
MaxwellJ. (1969). Sixteen years on: A follow-up of the 1947 Scottish Survey. London, England: University of London Press.
38.
MendozaJ. L. (1993). Fisher transformations and correlations corrected for selection and missing data. Psychometrika, 58, 601-615.
39.
MendozaJ. L.MumfordM. (1987). Corrections for attenuation and range restriction on the predictor. Journal of Educational Statistics, 12, 282-293.
40.
MicceriT. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166.
41.
MoorsJ. J. (1986). The meaning of kurtosis: Darlington re-examined. American Statistician, 40, 283-284.
42.
National Records of Scotland. (2013). Vital events Reference Tables, 2013. Retrieved from National Records of Scotland website: www.gro-scotland.gov.uk
43.
NishiwakiY.ClarkH.MortonS. M.LeonD. A. (2005). Early life factors, childhood cognition and postal questionnaire response rate in middle age: The Aberdeen Children of the 1950s study. BMC Medical Research Methodology, 5. doi:10.1186/1471-2288-5-16
44.
ReeM. J.CarrettaT. R.EarlesJ. A.AlbertW. (1994). Sign changes when correcting for range restriction: A note on Pearson’s and Lawley’s selection formula. Journal of Applied Psychology, 79, 298-301.
45.
RothP. L.BobkoP.SwitzerF. S.IIIDeanM. A. (2001). Prior selection causes biased estimates of standardized ethnic group differences: Simulation and analysis. Personnel Psychology, 2001, 591-617.
46.
SackettP. R.LaczoR. M.ArveyR. D. (2002). The effects of range restriction on estimates of criterion interrater reliability: Implications for validation research. Personnel Psychology, 55, 807-825.
47.
SavaleiV.RhemtullaM. (2012). On obtaining estimates of the fraction of missing information from full information maximum likelihood. Structural Equation Modeling, 19, 477-494.
48.
SchmidtF. L.HunterJ. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.
49.
Scottish Council for Research in Education. (1933). The intelligence of Scottish children. London, England: University of London Press.
50.
Scottish Council for Research in Education. (1949). The trend of Scottish intelligence. London, England: University of London Press.
51.
Scottish Council for Research in Education. (1953). Social implications of the 1947 Scottish Mental Survey. London, England: University of London Press.
52.
SegalN. L. (2000). Entwined lives: Twins and what they tell us about human behavior. New York, NY: Plume.
53.
TarescavageA. M.FischlerG. L.CappoB. M.HillD. O.CoreyD. M.Ben-PorathY. S. (2015). Minnesota Multiphasic Personality Inventory-2-Restructured Form (MMPI-2-RF) predictors of police officer problem behavior and collateral self-report test scores. Psychological Assessment, 27, 125-137.
54.
TaylorA. (2004). The consequences of selective participation on behavioral genetic findings: Evidence from simulated and real data. Twin Research, 7, 485-504.
55.
TellegenA.WallerN. G. (2008). Exploring personality through test construction: Development of the Multidimensional Personality Questionnaire. In BoyleJ. G.MatthewsG.SaklofskeD. H. (Eds.), The SAGE handbook of personality and assessment: Vol. 2. Personality measurement and testing (pp. 261-292). Thousand Oaks, CA: Sage.
56.
ThorndikeR. L. (1949). Personnel selection. New York, NY: Wiley.
57.
VolkenT. (2013). Second-stage no-response in the Swiss Health Survey: Determinants and bias in outcomes. BMC Public Health, 13, 167. doi:10.1186/1471-2458-13-167
58.
WatkinsM. P. (1979). Similarities between newlywed spouses (assortative marriage) with respect to specific cognitive abilities, socioeconomic status, and education. Berkeley: University of California.
59.
WechslerD. (1955). Manual for the Wechsler Adult Intelligence Scale. New York, NY: Psychological Corporation.
60.
WeinerI. B. (2003). Handbook of psychology. Hoboken, NJ: Wiley.
61.
WilcoxR. R. (1996). Statistics for the social sciences. San Diego, CA: Academic Press.
62.
WilcoxR. R. (1997). Introduction to robust estimation and hypothesis testing. San Diego, CA: Academic Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
