The Reliability of a Promotional Job Knowledge Examination Scored by Numberof Items Right and by Four Confidence Weighting Procedures and its Corresponding Concurrent Validity Estimates Relative to Performance Criterion Ratings

Abstract

For a sample of 153 male police officers, five sets of scores on a multiple-choice job knowledge examination were derived from the use of one conventional scoring procedure representing the number of items correct and from four confidence weighting methods. These five sets of scores were correlated with ratings of supervisors on each of eight job performance dimensions and with standings on each of eight factor scales of a standardized personality inventory. Evidence concerning the realization of higher test reliabilities from use of confidence weighting procedures in relation to those obtained from the conventional scoring method was equivocal. Increments in validity coefficients obtained from the confidence weighting scoring methods were slight. None of the five sets of scores registered statistically significant correlations with any one of eight personality scales. It would appear that in the context of the particular public service job studied the employment of confidence weighting techniques in achievement test scoring would yield at best only small gains in either reliability or concurrent validity. The use of the fourth confidence weighting procedure that statistically controlled for variance in a job knowledge examination demonstrated not one statistically significant correlation with any one of the personality factor scales but afforded statistically significant validity coefficients with five of the eight criterion measures-a circumstance which suggests the possible presence of a unique construct deserving of further systematic research in a variety of employment settings.

Get full access to this article

View all access options for this article.

References

Comrey, A. (1970). Comrey Personality Scales. San Diego: Educational and Industrial Testing Service.

de Finetti, B. (1965). Methods for discriminating levels of partial knowledge concerning a test item. British Journal of Mathematical and Statistical Psychology, 13, 87-123.

Guilford, J. P. and Fruchter, B. (1973). Fundamental statistics in psychology and education (5th ed.). New York: McGraw-Hill.

Hansen, R. (1971). The influence of variables other than knowledge on a probabilistic test. Journal of Educational Measurement, 8, 9-14.

Kristof, W. (1964). Testing differences between reliability coefficients. British Journal of Statistical Psychology, 17, 105-11 1.

Magnusson, D. (1967). Test theory. Reading, MA: Addison-Wesley.

Michael, J. J. (1968). The reliability of a multiple-choice examination under various test-taking instructions. Journal of Educational Measurement, 5, 307-314.