‘I Am Not a Rubber Stamp’

Abstract

• Summary: In the United Kingdom a formal assessment of all universities’ research is undertaken through the Research Assessment Exercise (RAE) approximately once every five years. The results are published and lead to each academic unit receiving a ranking (on a seven-point scale). Funding for research follows these rankings. Initiatives designed to review research quality have an obligation to clearly describe criteria on which assessments are made. Critical appraisals of published material prepared by RAE review panels should be based on criteria that permit rigorous appraisal of different kinds of publications. Criteria for reviewing the quality of different kinds of research are readily available. In empirical studies, such critical appraisals would reveal the likelihood that the research method used could answer questions raised. Efforts made by a non-UK adviser appointed to the 2001 RAE Panel for Social Policy and Administration and Social Work to identify criteria used by RAE organizers to review different kinds of publications were unsuccessful, resulting in the conclusion that panel members do not have a written description of specific criteria they share and that they share with others which describe exactly how they assess the quality of different kinds of research studies (e.g. controlled clinical trials, cohort studies, qualitative research and so on) and the quality of conceptual work. The troubling implications of this finding are discussed.

• Findings: No clear criteria allowing rigorous appraisal of different kinds of research were provided by the RAE in spite of repeated requests for the information. Rankings by RAE panel members seem to be made based on vague criteria and surrogates such as ‘esteem indicators’ (e.g. papers presented at conferences) that may not reflect quality of research. Productivity does not necessarily reflect research quality. Without the use of criteria that yield rigorous appraisals of research quality in the review process, it is likely that stereotypes about which programs produce the highest-quality research will simply be perpetuated. Lack of clear criteria for reviewing research seems to encourage preparation of propagandistic promotional text replete with grand claims, including opinions stated as facts. Government bodies responsible for distributing research funds should give careful consideration of how to fairly and accurately evaluate research quality. At least in this area, it seems that such consideration has not occurred.

• Applications: Lack of success in finding clear descriptions of criteria led to the troubling conclusion that no such specific criteria were used. The implications of these findings are concerning. They question the basis upon which research funding is allocated in the UK and indicate that if future rankings of research are undertaken, they should be based on explicit and open published criteria that permit an accurate assessment of the match between questions raised and methods used to pursue answers, available to all parties well in advance. Without such criteria staff lose an opportunity to hone their critical appraisal skills, rigorously appraise their research and candidly examine outcomes.