Sage Journals: Discover world-class research

Abstract

Intraclass correlation coefficients (ICCs) often are used to index the level of interrater agreement for job analysis ratings. Typically, ICCs are calculated from a one-way ANOVA design for each of several rated job characteristics in which multiple job analysts rate multiple jobs. ICCs are highest when between mean square (BMS) is large relative to within mean square (WMS), but they can be low either when (a) WMS is large, in which case ICCs appropriately indicate low interrater agreement, or (b) BMS is small, in which case ICCs reflect the attenuating influence of range restriction on the grouping factor (i.e., jobs). Using two job analysis data sets, we found that (a) ICCs covaried significantly and negatively with WMS components and (b) ICCs covaried significantly and positively with BMS. These restults indicate that ICCs cannot be interpreted unambiguously as indices of interrater agreement because they are substantially affected by the level of between-group variance. Interpretational confounding for ICCs can be circumvented by (a) using statistics designed specifically to index within-group interrater agreement, or (b) applying corrections for attenuation due to range restriction.

Get full access to this article

View all access options for this article.

References

Cain

P. S.

Green

B. F.

(1983). Reliabilities of selected ratings available from the Dictionary of Occupational Titles. Journal of Applied Psychology, 68, 155–165.

Christal

R. E.

(1974). The United States Air Force occupational research project. (AFHRL-TR-73-75). Lackland AFB, TX: Air Force Human Resources Laboratory, Occupational Research Division.

Christal

R. E.

Madden

J. M.

Harding

F. D.

(1960). Reliability of job evaluation ratings as a function of number of raters and length of job descriptions. (WADD-TN-60–257). Personnel Laboratory, Write Air Development Division, Air Research and Development Command, Lackland, AFB, TX.

Cornelius

E. T.

III Lyness

K. S.

(1980). A comparison of holistic and decomposed judgment strategies in job analyses by incumbents. Journal of Applied Psychology, 65, 155–163.

Hazel

J. T.

(1966). Reliability of job ratings as a function of time spent on evaluations. Journal of Industrial Psychology, 4, 16–19.

Hazel

J. T.

Cowan

D. K.

(1966). Evaluation of airman jobs by four categories of raters. (PRL-TR-66–3). Lackland AFB, TX: Personnel Research Laboratory, Aerospace Medical Division.

James

L. R.

Demaree

R. G.

Wolf

(1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.

Kavanagh

M. J.

Lance

C. E.

(1990). Transferability of skills methodology. (Draft Technical Paper). Brooks AFB, TX: Air Force Human Resource Laboratory, Manpower and Personnel Division.

Lahey

M. A.

Downey

R. G.

Saal

F. E.

(1983). Intraclass correlations: There's more that meets the eye. Psychological Bulletin, 93, 586–595.

10.

Lance

C. E.

Kavanagh

M. J.

Gould

R. B.

(1990). Development and convergent validation of a methodology for estimating cross-job transferability of skills. (Manuscript submitted for publication).

11.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

12.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.

Do Intraclass Correlation Coefficients Index Interrater Agreement?

Abstract

Get full access to this article

References