The present research was designed to study whether amount of information and kind of information available to the judges affect the consensus among judges. Two experiments were performed, where the consensus was analyzed by the lens model equation, which has been related to the non-criterion situation by Naylor and Schenck (1966). The amount of information (two, four, and six tests) was found to have an effect on the consensus, whereas type of information available (access to the validities of the tests and their intercorrelations or not) did not have a significant effect on the consensus. The results imply that consensus among pairs of judges decreases when number of tests increase. However, the difference between four and six tests was not significant in one of the experiments. The changes in consensus as the number of tests increased was mainly a function of a decreased matching between a pair of judges with respect to the linear aspects of their judgment processes. A further analysis of the linear aspects of the judges' judgment processes indicated that policy similarity contributed more to consensus than policy consistency. The contribution of the nonlinear aspects of the judgment process to consensus was negligible in these experiments.
Get full access to this article
View all access options for this article.
References
1.
Anderson, N. H.Application of an additive model to impression formation. Science, 1962 , 138, 817-818.
2.
Anderson, N. H.Averaging versus adding as a stimulus combination rule in impression formation. Journal of Experimental Psychology, 1965, 70, 394-400.
3.
Brehmer, B.The structure of policy conflicts . Umeå Psychological Reports, The University of Umeå, 1970, No. 29.
4.
Brodie, C. M.Clinical prediction of personality traits displayed in specific situations. Journal of Clinical Psychology , 1964, 20, 459-461.
5.
Bryan, J. H., Hunt, W. A. , and Walker, R. E.Reliability of estimating intellectual ability from transcribed interviews . Journal of Clinical Psychology, 1966, 22, 360.
6.
Dana, R. H., Cocking, R. R., and Dana, J. M.The effects of experience and training on accuracy and configural analysis . Journal of Clinical Psychology, 1970, 26, 28-32.
7.
Goldberg, L. R.Reliability of peace corps selection boards: A study of interjudge agreement before and after board discussions. Journal of Applied Psychology, 1966, 50, 400-408.
8.
Goldberg, L. R.Simple models or simple processes? Some research on clinical judgments. American Psychologist , 1968, 23, 483-496.
9.
Goldberg, L. R.Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inferences . Psychological Bulletin, 1970, 73, 422-432.
10.
Gunderson, E. K. E.Determinants of reliability in personality ratings. Journal of Clinical Psychology, 1965, 21, 164-169.
11.
Hammond, K. R., and Brehmer, B.Distrust among nations: A challenge to scientific inquiry . Institute of Behavioral Science, University of Colorado: Program on Cognitive Processes, 1969, No. 116.
12.
Hammond, K. R., Hursch, C. J., and Todd, F. J.Analyzing the components of clinical inference. Psychological Review, 1964, 71, 438-456.
13.
Hoffman, P. J.The paramorphic representation of clinical judgment. Psychological Bulletin, 1960, 57, 116-131.
14.
Hoffman, P. J., Slovic, P. , and Rorer, L. G.An analysis-of-variance model for the assessment of configural cue utilization in clinical judgment. Psychological Bulletin, 1968, 69, 338-349.
15.
Howard, K. I.Ratings of projective test protocols as a function of degree of inference. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1963, 23, 267-275.
16.
Huff, F. W., and Friedman, H.A study of the effects on agreement among clinicians of redundant and new information, confidence, and time available for assessments . The Journal of General Psychology, 1967 , 76, 49-57.
17.
Hunt, W. A. and Jones, N. F.Clinical judgment of some aspects of schizophrenic thinking . Journal of Clinical Psychology, 1958, 14, 235-239.
18.
Hunt, W. A., Walker, R. E. , and Jones, N. F.The validity of clinical ratings for estimating severity of schizophrenia . Journal of Clinical Psychology, 1960, 16, 391-393.
19.
Hursch, C. J., Hammond, K. R., and Hursch, J. L.Some methodological considerations in multiple-cue probability studies. Psychological Review, 1964, 71, 42-60.
20.
Lee, J. C. and Tucker, R. B.An investigation of clinical judgment: A study in method. Journal of Abnormal and Social Psychology, 1962, 4, 272-280.
21.
Little, K. B. and Shneidman, E. S.Congruencies among interpretations of psychological test and anamnestic data. Psychological Monographs, 1959, 73, (6, Whole No. 476).
22.
Magnusson, D.The effect of training on ratings from a projective technique. Reports from the Psychological Laboratories, The University of Stockholm , 1962, No. 122.
23.
Magnusson, D., Gerzen, M. , and Nyman, B.The generality of behavioral data I: Generalization from observations on one occasion . Multivariate Behavioral Research, 1968, 3, 295-320.
24.
Magnusson, D., Heffler, B. , and Nyman, B.The generality of behavioral data II: Replication of an experiment on generalization from observations on one occasion. Multivariate Behavioral Research , 1968, 3, 415-422.
25.
Magnusson, D. and Heffler, B.The generality of behavioral data III: Generalization potential as a function of the number of observation instances. Multivariate Behavioral Research, 1969, 4, 29-42.
26.
Magnusson, D. and Ryder, B.Subjective confidence and interobserver agreement for interview data as a function of interview length. Psychological Laboratories, The University of Stockholm, 1967 (Mimeographed prepublication).
27.
Meehl, P. E.The cognitive activity of the clinician. American Psychologist, 1960, 15, 19-27.
28.
Miller, P.McC. Do labels mislead? A multiple cue study, within the framework of Brunswik's probabilistic functionalism . Organizational Behavior and Human Performance, 1971, 6, 480-500.
29.
Naylor, J. C. and Schenck, E. A.ρm as an "error-free" index of rater agreement. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1966, 26, 815-824.
30.
Newton, J. R.Judgment and feedback in a quasi-clinical situation. Journal of Personality and Social Psychology, 1965, 1, 336-342.
31.
Nystedt, L.A further study on predictive efficiency as a function of amount of information. Reports from the Psychological Laboratories, The University of Stockholm, 1971, No. 335.
32.
Nystedt, L. and Magnusson, D.Predictive efficiency as a function of amount of information . Multivariate Behavioral Research, 1972, 7, 441-450.
33.
Nystedt, L. and Magnusson, D.Cue relevance and feedback in a clinical prediction task. Organizational Behavior and Human Performanee, 1973, 9, 100-109.
34.
Slovic, P.Cue-consistency and cue-utilization in judgment. American Journal of Psychology.1966, 79, 427-434.
35.
Slovic, P.Analyzing the expert judge: A descriptive study of a stockbroker's decision processes. Journal of Applied Psychology, 1969, 53, 255-263.
36.
Weitman, M.Some variables related to bias in clinical judgment. Journal of Clinical Psychology, 1960, 18, 504-506.
37.
Wiggins, N. and Hoffman, P. J.Three models of clinical judgment. Journal of Abnormal and Social Psychology, 1968, 73, 70-77.
38.
Winslow, C. N. and Rapersand, I.Postdiction of the outcome of somatic therapy from the Rorschach records of schizophrenic patients. Journal of Consultin g Psychology, 1964, 28, 243-247.
