Estimating Reliability with Small Samples: Increased Precision with Averaged Correlations.

Abstract

Both analytic and Monte Carlo methods were used to show that empirical estimates of the standard error of an averaged correlation agree quite well with theoretical expressions derived from Pearson's asymptotic equations. Where discrepancies did exist, particularly with small sample sizes, the theoretical formula slightly overestimated the actual (empirical) standard error, thus making a test of significance conservative. It was shown that by testing repeatedly with a small sample and averaging the resulting relations, one may obtain precision equivalent to single estimates from much larger samples. Furthermore, the greatest increase in precision occurred when the population correlation was small, the situation where greatest precision is needed. These results should prove of value for human factors evaluations, which often have limited subject resources.

Get full access to this article

View all access options for this article.

References

Allen, M. J. , and Yen, W. M. (1979). Introduction to measurement theory. Monterey. CA: Brooks-Cole.

Alvares, K. M. , and Hulin, C. L. (1972). Two explanations of temporal changes in ability-skill relationships: A literature review and theoretical analysis. Human Factors, 14, 295–308.

Bittner, A. C., Jr. (1982). Averaged correlations between parallel measures: Reliability estimation. In Proceedings of the Eighth sychology in the DoD Symposium (USAFA-TR-82-10) (pp. 321–327). Colorado Springs, CO: U.S. Air Force Academy.

Bittner, A. C., Jr. , Carter, R. C. , Kennedy, R. S. , Harbeson, M. M. , and Krause, M. (1984). Performance evaluation tests for environmental esearch (PETER): Evaluation of 112 measures (NBDL-84R006). New Orleans: Naval Biodynamics Laboratory.

Bittner, A. C., Jr. , Dunlap, W. P. , and Jones, M. B. (1982). Averaged cross-correlations with differentially stable variables: Fewer subjects required with repeated measures. In Proceedings of the Human Factors Society 26th Annual Meeting (pp. 349–353). Santa Monica, CA: Human Factors Society.

Dunlap, W. P. , Jones, M. B. , and Bittner, A. C. (1983). Average correlation vs. correlated averages. Bulletin of the Psychonomic Society, 21, 213–216.

Dunlap, W. P. , Silver, N. C. , Hunter, R. E. , and Bittner, A. C., Jr. (1985). Averaged cross-correlations: A methodology for validity assessment in small samples. In R. Eberts and C. G. Eberts (Eds.), Trends in ergonomics/human factors II (pp. 13–21). Amsterdam: Elsevier.

Epstein, S (1980). The stability of behavior: II. Implications for psychological research. Americun Psychologist, 35, 790–806.

Harmon, H. H. (1976). Modern factor analysis (3rd cd.). Chicago: University of Chicago Press.

10.

Kendall, S. M. , and Stuart, A. (1979). The advanced theory of statistics, Vol. I. (4th ed.). New York: Macmillan.

11.

Pearson, K. , and Filon, L. N. G. (1898). Mathematical contributions to the theory of evolution: IV. On the probable errors of frequency constants and on the influence of random selection on variation and correlation. Philosophical Transactions of the Royal Society of London, Series A, 191, 229–311.

12.

Rushton, J. P. , Brainerd, C. J. , and Pressley, M. (1983). Behavioral development and construct validity: The principle of aggregation. Psychological Bulletin, 94, 18–38.

13.

Schmidt, F. L. , and Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 87, 529–540.

14.

Schmidt, F. L. , Hunter, J. E. , and Urry, V. W. (1976). Statistical power in criterion-related validity studies. Journal of Applied Psychology, 61, 473–485.

15.

Steiger, J. H. (1980). Tests for comparing clements of a correlation matrix. Psychological Bulletin, 87, 245–251.

16.

Sutcliffe, J. P. (1980). On the relationship of reliability to statistical power. Psychological Bulletin, 88, 509–515.

17.

Trattner, M. H. , and O'Leary, B. S. (1980). Sample size for specified power in testing for differential validity. Journal of Applied Psychology, 65, 127–134.