This paper analyses peer review deliberations in four evaluation panels that differ in terms of scope and disciplinary heterogeneity. Based on evaluation reports and discussions with panel members, it illustrates a variety of ways in which reviewers bridge their epistemological differences and achieve consensus on the quality of research proposals. The analysis demonstrates that peer review panels are forums in which communication across disciplinary boundaries occurs and interdisciplinary judgments arise. At the same time, disciplinary gate-keeping and incommensurabilities may set limits on such communication. The comparison of deliberative processes sheds light on how collective judgments are shaped and constrained by the disciplinary set-up of the panels in which the reviewers operate and in which the intersubjective dynamics of the deliberations unfold. Based on these findings, the paper considers conditions that may enhance disciplinary interaction and complementary judgments in the peer review of proposals, and thereby expands the prospects for interdisciplinary research.
Boix MansillaVFellerIGardnerH (2006) Quality assessment in interdisciplinary research and education. Research Evaluation15(1): 69–74.
2.
BoltanskiLThévenotL (1999) The sociology of critical capacity. European Journal of Social Theory2(3): 359–377.
3.
BrenneisD (1994) Discourse and discipline at the National Research Council: A bureaucratic Bildungsroman. Cultural Anthropology9(1): 23–36.
4.
BruunHHukkinenJHuutoniemiKKleinJT (2005) Promoting interdisciplinary research: The case of the Academy of Finland. Publications of the Academy of Finland 8/05. The Academy of Finland, Helsinki.
5.
ChubinDEHackettEJ (1990) Peerless Science: Peer Review and U.S. Science Policy. Albany, NY: State University of New York Press.
6.
CicchettiDV (1991) The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences14(1): 119–135.
7.
ColeJRColeS (1981) Peer Review in the National Science Foundation: Phase Two of a Study. Washington, DC: National Academy Press.
8.
ColeS (1992) Making Science: Between Nature and Society. Cambridge, MA: Harvard University Press.
9.
ColeSColeJRSimonGA (1981) Chance and consensus in peer review. Science214(4523): 881–886.
10.
ColeSRubinLColeJR (1978) Peer Review in the National Science Foundation: Phase One of a Study. Washington, DC: National Academy Press.
11.
EisenhartM (2002) The paradox of peer review: Admitting too much or allowing too little?Research in Science Education32(2): 241–255.
12.
EspelandWNStevensML (1998) Commensuration as a social process. Annual Review of Sociology24: 313–343.
13.
FellerI (2006) Multiple actors, multiple settings, multiple criteria: Issues in assessing interdisciplinary research. Research Evaluation15(1): 5–15.
14.
FuchsS (2001) Against Essentialism. A Theory of Culture and Society. Cambridge, MA: Harvard University Press.
15.
FullerS (2002) Knowledge Management Foundations. Boston, MA and Oxford: Butterworth-Heinemann.
16.
General Accounting Office (1994) Peer Review Reforms Needed to Ensure Fairness in Federal Agency Grant Selection. Report to the Chairman, Committee on Governmental Activities, US Senate. Washington, DC: General Accounting Office.
17.
GriggL (1999) Cross-Disciplinary Research. A Discussion Paper. Commissioned Report No. 61. Canberra: Australian Research Council.
18.
GulbrandsenJM (2000) Research quality and organisational factors: An investigation of the relationship. Doctoral dissertation, Department of Industrial Economics and Technology Management, Norwegian University of Science and Technology, Trondheim.
19.
HackettEJChubinDE (2003) Peer Review for the 21st Century: Applications to Education Research. Prepared for a National Research Council Workshop, Washington, DC.
20.
HemlinS (1991) Quality in science: Researchers’ conceptions and judgments. Doctoral Dissertation, Department of Psychology, University of Göteborg, Göteborg, Sweden.
21.
HemlinS (2009) Peer review agreement or peer review disagreement: Which is better?Journal of Psychology of Science and Technology2(1): 5–12.
22.
HirschauerS (2010) Editorial judgments: A praxeology of ‘voting’ in peer review. Social Studies of Science40(1): 71–103.
23.
HuutoniemiK (2010) Evaluating interdisciplinary research. In: FrodemanRKleinJTMitchamC (eds) Oxford Handbook of Interdisciplinarity. Oxford: Oxford University Press, 309–319.
24.
HuutoniemiKKleinJTBruunHHukkinenJ (2010) Analyzing interdisciplinarity: Typology and indicators. Research Policy39: 79–88.
25.
KleinJT (2008) Evaluation of interdisciplinary and transdisciplinary research: A literature review. American Journal of Preventive Medicine35: S116–S123.
26.
LamontM (2009) How Professors Think: Inside the Curious World of Academic Judgment. Cambridge, MA: Harvard University Press.
27.
LamontMFournierMGuetzkowJMallardGBernierR (2007) Evaluating creative minds: The assessment of originality in peer review. In: SalesAFournierM (eds) Knowledge, Communication, and Creativity. London: Sage, 166–181.
28.
LamontMHuutoniemiK (2011) Comparing customary rules of fairness: Evaluative practices in various types of peer review panels. In: CamicCGrossNLamontM (eds) Social Knowledge in the Making. Chicago: University of Chicago Press, 209–232.
29.
LamontMMallardGGuetzkowJ (2006) Beyond blind faith: Overcoming the obstacles to interdisciplinary evaluation. Research Evaluation15(1): 43–55.
30.
LangfeldtL (2004) Expert panels evaluating research: Decision-making and sources of bias. Research Evaluation13(1): 52–62.
31.
LangfeldtL (2006) The policy challenges of peer review: Managing bias, conflict of interests and interdisciplinary assessments. Research Evaluation15(1): 31–41.
32.
LaudelG (2006) Conclave in the Tower of Babel: How peers review interdisciplinary research proposals. Research Evaluation15(1): 57–68.
33.
LaudelGOriggiG (2006) Introduction to a special issue on the assessment of interdisciplinary research. Research Evaluation15(1): 2–4.
34.
MallardGLamontMGuetzkowJ (2009) Fairness as appropriateness: Negotiating epistemological differences in peer review. Science, Technology, & Human Values34(5): 573–606.
35.
MarshHWJayasingheUWBondNW (2008) Improving the peer review process for grant applications: Reliability, validity, bias, and generalizability. American Psychologist63(3): 160–168.
36.
MertonRK (1996) On Social Structure and Science. Chicago: University of Chicago Press.
37.
Muller MirzaNPerret-ClermontA-NTartasVIannacconeA (2009) Psychosocial processes in argumentation. In: Muller MirzaNPerret-ClermontA-N (eds) Argumentation and Education: Theoretical Foundations and Practices. New York: Springer, 67–90.
38.
OlbrechtMBornmannL (2010) Panel peer review of grant applications: What do we know from research in social psychology on judgment and decision-making in groups?Research Evaluation19(4): 293–304.
39.
PomerantzA (1984) Agreeing and disagreeing with assessments: Some features of preferred/dispreferred turn shapes. In: AtkinsonJMHeritageJ (eds) Structures of Social Action: Studies in Conversation Analysis. Cambridge: Cambridge University Press, 57–101.
40.
PorterALRossiniFA (1985) Peer review of interdisciplinary research proposals. Science, Technology, & Human Values10(3): 33–38.
41.
Research Evaluation (2006) Special issue on interdisciplinary research assessment. Research Evaluation15(1).
42.
RoyR (1985) Funding science: The real defects of peer review and an alternative to it. Science, Technology, & Human Values10(3):73–81.
43.
RussellMG (1983) Peer review in interdisciplinary research: Flexibility and responsiveness. In: EptonSRPayneRLPearsonAW (eds) Managing Interdisciplinary Research. New York: John Wiley & Sons, 184–202.
44.
ShimadaKAkagiMKazamakiTKobayashiS (2007) Designing a proposal review process to facilitate interdisciplinary research. Research Evaluation16(1): 13–21.
45.
ThorngateWDawesRMFoddyM (2009) Judging Merit. New York: Psychology Press.
46.
TravisGDLCollinsHM (1991) New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology, & Human Values16(3): 322–341.
47.
WeinbergAM (1962) Criteria for scientific choice. Minerva1(2): 158–171.
48.
WhitleyR (1984) The Intellectual and Social Organization of the Sciences. Oxford: Clarendon Press.
