Rating scales are most frequently used to assess the criterion in studies of validity in personnel selection. However, only a few articles report the interrater reliability for these scales. This paper presents four meta-analyses in which the inter-rater reliability was estimated for Civil Composite Criterion, Military Composite Criterion, Total Composite Criterion, and Single Civil Criterion. Mean reliabilities were .64, .53, .62, and .40, respectively. The implications of these findings for single and meta-analytic studies are discussed.
Get full access to this article
View all access options for this article.
References
1.
GhiselliE. E. (1966) The validity of occupational aptitude tests. New York: Wiley.
2.
GhiselliE. E. (1973) The validity of aptitude tests in personnel selection. Personnel Psychology, 23, 461–478.
3.
HartiganJ. A.WicdorA. K. (1989) Fairness in employment testing: validity generalization minority and the General Aptitude Test Battery. Washington, DC: National Research Council.
4.
HunterJ. E. (1980) Validity generalization for 12000 fobs: an application of synthetic validity and validity generalization to the General Aptitude Test Battery (GATB). Washington, DC: US Employment Service, US Department of Labor.
5.
HunterJ. E.HirshH. R. (1987) Applications of meta-analysis. In CooperC. L.RobertsonI. T.(Eds.), International review of industrial and organizational psychology. London: Wiley. pp. 321–357.
6.
HunterJ. E.HunterR. F. (1984) Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72–98.
7.
HunterJ. E.SchmidtF. L. (1990) Methods of meta-analysis, correcting error and bias in research findings. Newbury Park, CA: Sage.
8.
KingL. M.HunterJ. E.SchmidtF. L. (1980) Halo in multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology, 65, 507–516.
9.
RobertsonI. T.KinderA. (1993) Personality and job competences: the criterion-related validity of some personality variables. Journal of Occupational and Organizational Psychology, 66, 225–244.
10.
RothsteinH. R. (1990) Interrater reliability of job performance ratings: growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75, 322–327.
11.
SalgadoJ. F. (submitted) Sample size in validity studies of personnel selection. Journal of Occupational and Organizational Psychology.
12.
SchmidtF. L.HunterJ. E. (1977) Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529–540.
13.
SchmidtF. L.HunterJ. E. (1980) The future of criterion-related validity. Personnel Psychology, 33, 41–60.
14.
SchmidtF. L.HunterJ. E. (1981) Employment testing: old theories and new research findings. American Psychologist, 36, 1128–1137.
15.
SchmidtF. L.HunterJ. E. (1995) The measurement errors in the applied research: answers to 27 scenarios. (Unpublished manuscript, Department of Management and Organizations, Univer. of Iowa).
16.
SchmittN.GoodingR. Z.NoeR. D.SckirschM. (1984) Meta-analyses of validity studies published between 1964 and 1982 and the investigation of study characteristics. Personnel Psychology, 37, 407–422.
17.
SchmidtF. L.HunterJ. E.PearlmanK. E.ShaneG. S. (1979) Further test of the Schmidt-Hunter Bayesian validity generalization procedure. Personnel Psychology, 32, 257–281.
18.
SchmidtF. L.PearlmanK. E.HunterJ. E.HirschH. R. (1985) Forty questions about validity generalization and meta-analysis. Personnel Psychology, 38, 697–798.
19.
BarrickM. R.MountM. K. (1993) Autonomy as moderator of the relationship between the Big Five personality dimensions and job performance. Journal of Applied Psychology, 78, 111–118.
20.
BarrickM. R.MountM. K.StraussJ. P. (1993) Conscientiousness and performance of sales representatives: test of the mediating effects of goal setting. Journal of Applied Psychology, 78, 715–722.
21.
BormanW. C.HansonM. A.OpperS. H.PulakosE. D.WhiteC. A. (1993) Role of early supervisory experience in supervisor performance. Journal of Applied Psychology, 78, 443–449.
22.
CampionM. A.CampionJ. E.HudsonJ. P.Jr. (1994) Structured interviewing: a note of incremental validity and alternative question types. Journal of Applied Psychology, 79, 998–1002.
23.
CampionM. A.PurssellE. D.BrownB. K. (1988) Structured interviewing: raising the psychometric properties of the employment interview. Personnel Psychology, 41, 25–42.
24.
DayD. V.SilvermanS. B. (1989) Personality and job performance: evidence of incremental validity. Personnel Psychology, 42, 25–36.
25.
HoganJ.HoganR.BuschL. M. (1984) How to measure service orientation. Journal of Applied Psychology, 69, 167–173.
26.
McEvoyG. M.BeattyR. W. (1989) Assessment center and subordinate appraisals of managers: a seven-year examination of predictive validity. Personnel Psychology, 42, 37–52.
27.
MotowidloS. J.CarterG. W.DunnetteM. D.TippinsN.WarnerS.BennettJ. P.VaughanM. J. (1992) Studies of the structured behavioral interview. Journal of Applied Psychology, 77, 571–587.
28.
MotowidloS. J.PackardJ. S.ManningM. R. (1986) Occupational stress: its causes and consequences for job performance. Journal of Applied Psychology, 71, 618–629.
29.
PulakosE. D.BormanW. C.HoughL. M. (1988) Test validation for scientific understanding: two demonstrations of an approach to studying predictor-criterion linkages. Personnel Psychology, 41, 703–716.
30.
SchmittN.RoeR. A.MerittR.FitzgeraldM. P. (1984) Validity of assessment center rating for the prediction of performance ratings and school climate of school administrators. Journal of Applied Psychology, 69, 207–213.
31.
SchmittN.SchneiderJ. R.CohenS. A. (1990) Factors affecting validity of a regionally administered assessment center. Personnel Psychology, 43, 1–12.
32.
SiegelL. (1983) The miniature job training and evaluation approach: additional findings. Personnel Psychology, 36, 41–56.
