Reviewer evaluations and recommendations for 853 manuscript submissions, over a span of 4 years, are analyzed for evidence of particularistic bias, reviewer agreement, and predictive validity for forecasting a published manuscript's citation impact. Attributes of the submitters, their affiliated institutions, and the reviewers have little consistent association with reviewers' recommendations or editorial decision outcomes. Furthermore, reviewers' recommendations demonstrate a reasonable degree of agreement. However, neither reviewers' evaluative ratings across five dimensions nor publication recommendations can predict the number of citations that a published article subsequently receives. Strengths and limitations of various features of the manuscript review process, as well as the importance of monitoring the process for particularistic biases and evidence of predictive validity, are discussed.
Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.
2.
Bakanic, V., McPhail, C., & Simon, R. J. (1987). The manuscript review and decision-making process. American Sociological Review, 52, 631-642.
3.
Beyer, J. M., Chanove, R. G., & Fox, W. B. (1995). The review process and the fates of manuscripts submitted to AMJ. Academy of Management Journal, 38, 1219-1260.
4.
Campion, M. A. (1993a). Are there differences between reviewers on the criteria they use to evaluate research articles?The Industrial-Organizational Psychologist, 31, 29-39.
5.
Campion, M. A. (1993b). Article review checklist: A criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 705-719.
6.
Ceci, S. J., & Peters, D. (1984). How blind is blind review?American Psychologist, 39, 1491-1494.
7.
Chalmers, T. C., Frank, C. S., & Reitman, D. (1990). Minimizing the three stages of publication bias. Journal of the American Medical Association, 203, 1392-1395.
8.
Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
9.
Crane, D. (1972). Invisible colleges: Diffusion of knowledge in scientific communities. Chicago: University of Chicago Press.
10.
Daft, R. L. (1985). Why I recommended that your manuscript be rejected and what you can do about it. In L. L. Cummings & P. J. Frost (Eds.),Publishing in the organizational sciences(pp. 193-209). Homewood, IL: Irwin.
11.
Ernst, E., & Kienbacher, T. (1991). Chauvinism. Nature, 352, p. 560-560.
12.
Finn, R. H. (1970). A note on estimating the reliability of categorical data. Educational and Psychological Measurement, 30, 71-76.
13.
Fiske, D. W., & Fogg, L. (1990). But the reviewers are making different criticisms of my paper!American Psychologist, 45, 591-598.
14.
Gilliland, S. W., & Beckstein, B. A. (1996). Procedural and distributive justice in the editorial review process. Personnel Psychology, 49, 669-691.
15.
Gilliland, S., & Cortina, J. M. (1997). Reviewer and editor decision making in the journal review process. Personnel Psychology, 50, 427-453.
16.
Hasselback, J. R. (1996). Directory of management faculty. New York: McGraw-Hill.
17.
Jacobson, R. L. (1986). Scholars fault journals and college libraries in survey by council of learned societies. Chronicle of Higher Education, 32, 21-23.
18.
Keiser, J., Utzinger, J., Tanner, T., & Singer, S. H. (2004). Representation of authors and editors from countries with human development indexes in the leading literature on tropical medicine: Survey of current evidence. British Medical Journal, 328, 1229-1232.
19.
Kerr, S., Tolliver, J., & Petree, D. (1977). Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal, 20, 132-141.
20.
Kurst, C. (1992). Guide to doctoral programs in business and management. St. Louis: American Assembly of Collegiate Schools of Business.
21.
Laband, D. N., & Piette, M. J. (1994). A citation analysis of the impact of blinded peer review. Journal of the American Medical Association, 272, 147-149.
22.
Lloyd, M. (1990). Gender factors in reviewer recommendations for manuscript publication. Journal of Applied Behavioral Analysis, 23, 539-543.
23.
Long, R. G., Bowers, W. P., Barnett, T., & White, M. C. (1998). Research productivity of graduates in management: Effect of academic origin and academic affiliation. Academy of Management Journal, 41, 704-714.
24.
Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press.
25.
Moss, P. A. (1994). Can there be validity without reliability?Educational Researcher, 23, 5-12.
26.
Mosteller, F. R., & Rourke, R. E. (1973). Sturdy statistics:Nonparametrics & order statistics. Reading, MA: Addison-Wesley.
27.
Oromaner, M. (1974). Collaboration and impact: The careers of multi-authored publications. Social Science Information, 14, 147-155.
28.
Patterson, S. C. (1994). The itch to publish in political science. In R. J. Simon & J. F. Fyfe (Eds.), Editors as gatekeepers: Getting published in the social sciences (pp. 3-20). Boston: Rowman and Littlefield.
29.
Prescott, S., & Csikszentmihalyi, M. (1977). Institutional status and publication rates in professional journals. Quarterly Journal of Ideology, 1, 3-37.
30.
Presser, S. (1980). Collaboration and the quality of research. Social Studies of Science, 10, 95-101.
31.
Rennie, D. (1993). More peering into editorial peer review. Journal of the American Medical Association, 270, 2856-2858.
32.
Riggs, J. E. (1995). Priority, rivalry, and peer review. Journal of Child Neurology, 10, 255-256.
33.
Rosenthal, R. (1982). Reliability and bias in peer-review practices. Behavioral and Brain Sciences, 5, 235-236.
34.
Sackett, P. R., & Yang, H. (2000). Connection for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112-118.
35.
Walsh, W. B., & Betz, N. E. (1990). Tests and assessment. Englewood Cliffs, NJ: Prentice Hall.
36.
Watkins, M. W. (1979). Chance and interrater agreement on manuscripts. American Psychologist, 34, 796-798.
37.
Whitehurst, G. J. (1984). Interrater agreement for journal manuscript reviews. American Psychologist, 39, 22-28.
38.
Whitehurst, G. J. (1985). On lies, damned lies, and statistics: Measuring interrater agreement. American Psychologist, 40, 568-569.
39.
Yankauer, A. (1991). How blind is blind review?American Journal of Public Health, 81, 843-845.
40.
Zuckerman, H. (1988). The sociology of science. In N. J. Snelser (Ed.), Handbook of sociology (pp. 511-571). Newbury Park, CA: Sage.
41.
Zuckerman, H., & Merton, R. K. (1973). Age, aging, and age structure in science. In R. K. Merton (Ed.), The sociology of science (pp. 497-559). Chicago: University of Chicago Press.
