Reviewer agreement and the predictors of publication judgments were investigated for first-submission manuscripts to the Personality and Social Psychology Bulletin during a 3 1/2—year period (i.e., one editor’s tenure). Among the findings were the following: Reviewers’ judgments of manuscripts were multirather than unidimensional; reviewer agreement about methodology and overall recommendation was greater among high-prestige than mixed-prestige reviewers; authors with high prestige and authors with low professional experience submitted longer manuscripts than their counterparts; author prestige and text length were positively related to publication judgments of reviewers and editors; and author gender was related to editor’s decisions with female authors receiving less favorable decisions than males. The possible mediation of these findings and their implications for understanding the peer-review process in personality and social psychology are discussed.
Get full access to this article
View all access options for this article.
References
1.
Bakanic, V. , McPhail, C., & Simon, R. J. (1987). The manuscript review and decision-making process. American Sociological Review, 52, 631-642.
2.
Baron, R. M. , & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychology research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173-1182.
3.
Beyer, J. (1978). Editorial policies and practices among leading journals in four scientific fields. Sociological Quarterly, 19, 68-88.
4.
Browne, M. W. (1984). The decomposition of multitrait-multimethod matrices. British Journal of Mathematical and Statistical Psychology, 37, 1-21.
5.
Browne, M. W. (1989). Relationships between an additive and multiplicative model for multitrait-multimethod matrices. In R. Coppi & S. Bolasco (Eds.), Multiway data analysis (pp. 507-520). Amsterdam: Elsevier Science.
6.
Browne, M. W. , & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods and Research, 21, 230-258.
7.
Bruce, D. , & Bahrick, H. P. (1992). Perceptions of past research. American Psychologist, 47, 319-328.
8.
Ceci, S. L. , & Peters, D. (1984). How blind is blind review?American Psychologist, 39, 1491-1494.
9.
Chase, J. M. (1970). Normative criteria for scientific publication. American Sociologist, 4, 262-265.
10.
Cicchetti, D. V. (1980). Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. American Psychologist, 35, 300-303.
11.
Cicchetti, D. V. , & Eron, L. D. (1979). The reliability of manuscript reviewing for the Journal of Abnormal Psychology. Proceedings of the American Statistical Association (Social Statistics Section), 22, 596-600.
12.
Crandall, R. (1978). Interrater agreement is not so bad!American Psychologist, 33, 623-624.
13.
Crane, D. (1967). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. American Sociologist, 32, 195-201.
14.
Fiske, D. W. , & Fogg, L. (1990). But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. American Psychologist, 45, 591-598.
15.
Fogg, L. , & Fiske, D. W. (1993). Foretelling the judgments of reviewers and editors. American Psychologist, 48, 293-294.
16.
Gaston, J. (1978). The reward system in British and American science. New York: Wiley.
17.
Gilbert, J. R. , Williams, E. S., & Lundberg, G. D. (1994). Is there gender bias in JAMA’s peer review process?Journal of the American Medical Association, 272, 139-142.
18.
Goldberger, M. L. , Naher, B. A., & Flattau, P. E. (1995). Research-doctorate programs in the United States: Continuity and change. Washington, DC: National Academy Press.
19.
Goodrich, D. W. (1945). An analysis of manuscripts received by the editors of the American Sociological Review from May 1, 1944 to September 1, 1945. American Sociological Review, 10, 716-725.
20.
Gordon, M. D. (1980). The role of referees in scientific communication. In J. Hartley (Ed.), The psychology of written communication (pp. 263-275). New York: Kogan Page.
21.
Hagstrom, W. O. (1974). Competition in science. American Sociological Review, 39, 1-18.
22.
Hargins, L. L. , & Herting, J. R. (1990). A new approach to referees’ assessments of manuscripts. Social Science Research, 19, 1-16.
23.
Hendrick, C. (1976). Editorial comment. Personality and Social Psychology Bulletin, 2, 207-208.
24.
Hendrick, C. (1977). Editorial comment. Personality and Social Psychology Bulletin, 3, 1-2.
25.
Howard, G. S. , Cole, D. A., & Maxwell, S. E. (1987). Research productivity in psychology based on publication in the journals of the American Psychological Association. American Psychologist, 42, 975-986.
26.
King, A. (1986). How not to lie with statistics: Avoiding common mistakes in quantitative political science. American Journal of Political Science, 30, 666-687.
27.
Linder, D. E. (1977). Evaluation of the Personality and Social Psychology Bulletin by its readers and authors. Personality and Social Psychology Bulletin, 3, 583-591.
28.
Lloyd, M. E. (1990). Gender factors in reviewer recommendations for manuscript publication. Journal of Applied Behavior Analysis, 23, 539-543.
29.
Lock, S. (1982). A difficult balance. London: Nuttfield Provicial Hospital Trust.
30.
Luskin, R. C. (1991). Abusus non tollit usum: Standardized coefficients, correlations, and R2. American Journal of Political Science, 35, 1032-1046.
31.
Marsh, H. W. , & Ball, S. (1981). Interjudgmental reliability of reviews for the Journal of Educational Psychology. Journal of Educational Psychology, 73, 872-880.
32.
Marsh, H. W. , & Ball, S. (1989). The peer review process used to evaluate manuscripts submitted to academic journals. Journal of Experimental Education, 57, 151-169.
33.
O’Brien, R. M. (1991). The reliability of composites of referee assessments of manuscripts. Social Science Research, 20, 319-328.
34.
Peters, D. P. , & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5, 743-750.
35.
Petty, R. E. , & Cacioppo, J. T. (1986). The elaboration likelihood model of persuasion. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 19, pp. 123-205). New York: Academic Press.
36.
Scarr, S. , & Weber, B. L. (1978). The reliability of reviews for the American Psychologist. American Psychologist, 33,935.
37.
Scott, W. A. (1974). Inter-referee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. American Psychologist, 29, 698-702.
38.
Sternberg, R. J. , Hojjat, M., Brigockas, M. G., & Grigorenko, E. L. (1997). Getting in: Criteria for acceptance of manuscripts in Psychological Bulletin, 1993-1996. Psychological Bulletin, 121, 321-323.
39.
Whitehurst, G. J. (1984). Interrater agreement for journal manuscript reviews. American Psychologist, 39, 22-28.
40.
Wilson, T. D. , DePaulo, B. M., Mook, D. G., & Klaaren, K. J. (1993). Scientists’ evaluations of research: The biasing effects of the importance of the topic. Psychological Science, 4, 322-325.
41.
Wolff, W. M. (1973). Publication problems in psychology and an explicit evaluation schema for manuscripts. American Psychologist, 28, 257-261.
42.
Yoels, W. (1973). On publishing or perishing: Fact or fiction. American Sociologist, 8, 128-130.
43.
Zuckerman, H. , & Merton, R. K. (1971). Patterns of evaluation in science: Institutionalization, structure, and functions of the referee system. Minerva, 9, 66-100.
