Abstract
Let’s assume there’s an academic article criticizing a social institution – schools, say. In the article the author raises a host of complaints about schools – they are inefficient, they educate too many different kinds of children, it’s not even clear that they do what they say they do. To illustrate his case, the author draws on a range of different examples: a secondary school in the UK here, a primary school in Germany there, a technical college in Canada, a school (unspecified) in the US. When presenting this data, the author rarely, if ever, provides context; he does not distinguish between primary and secondary schools, private and public institutions, or religious and secular organizations. There is an assumption that all schools, regardless of country, educational level, funding and systemic features, can be assessed in the same terms, and that broader, social and structural aspects are essentially irrelevant to questions about how these schools do whatever it is that they are meant to do.
Given its severe limitations, it is unlikely that this analysis would go unchallenged by social scientists pointing out that such casual, de-contextualized comparisons tell us little about how educational institutions actually work and the benefits and drawbacks of different educational systems. I will not belabour the metaphor, but I do find it curious that while such an impoverished attack on schools (or almost any other social institution) would be met with a negative response from social scientists, it is perfectly acceptable to make similar claims about the bodies responsible for prospective research ethics review – research ethics committees, institutional review boards, research ethics board – and not be challenged. Indeed when making such claims about the ethical review of social scientific research, authors find themselves pushing at an open door, at least as far as their colleagues are concerned.
My response to Schrag’s article sees his position as typical of the majority of social science critiques of ethics review, which tend to generalize from the issues and problems associated with ethics review in a particular jurisdiction (for example, higher education institutions in the US) and claim that these problems are representative of ethics review in general, even in contexts that are quite different (the UK NHS, for example) (Schrag, 2011). My main point is not that ethics review systems are perfect and error-free, but rather that different ethics review systems will have their own distinct problems, will be prone to their own kinds of errors, and that the best critiques of ethics review will be those that avoid sloppy elision between different systems either by focusing on the problems of one particular system, or by comparative work that emphasizes difference and variation. In addition to criticizing Schrag’s unsupported assumptions about the universal characteristics of research ethics review bodies, this article also explores his failure to adequately engage with ethnographic research into the decision-making of such bodies, despite his complaints that we know too little about how these bodies reach their decisions.
Assumed isomorphism
In their classic article exploring the way in which organizations come to resemble each other, DiMaggio and Powell propose that institutional change be seen in terms of ‘isomorphism’, the process whereby in response to external environmental factors different institutions, over time, come to adopt similar structures, processes and attitudes (DiMaggio and Powell, 1983). I would like to suggest that a variation of this – what we might call ‘assumed isomorphism’ – tends to characterize much of the social science critique of ethics review, including Schrag’s article (Schrag, 2011). It is assumed isomorphism that underpins both critics’ willingness to equate RECs in the UK NHS with IRBs in US hospitals and REBs in Canadian universities, and the blind spot that assumes that the structural, functional and organizational differences between these different bodies and the systems within which they reside are not the cause of their (supposedly) pathological features.
Even a cursory analysis of ethics review systems underlines how important their organization is to the decisions made by ethics review bodies in different jurisdictions, and hence the behaviours that the social science critique objects to. Take, for example, the issue of institutional affiliation, i.e. whether an ethics review body has formal links with a research institution or not. There is considerable evidence that the institutional nature of US IRBs plays an important part in IRB decision-making, and that this institutional connection can raise problems for social scientific research on putatively controversial topics (usually research on sexual behaviour). The ‘institutional protection’ role of US IRBs – protecting their host universities or hospitals from litigation or bad press – is now well established, with some IRBs being ‘quite clear and above board that their main concern is protection of the institution from damage’ (Lincoln and Tierney, 2004: 220). The unexceptional nature of this role is such that a chair of one of the University of Chicago’s IRBs can confidently assert that:
A third important function served by Institutional Review Boards is to protect the institutional interest … Private universities are largely corporations. The respect for academic freedom within these corporations is desirable, but not indispensable … Institutional Review Boards can forestall the public image problems and protect the institution’s reputation by weeding out politically sensitive studies before they are approved. (Moss, 2007: 803–804)
It is not unreasonable to suggest that the institutional affiliation of US IRBs – an organizational characteristic which is not true for all research ethics review bodies – may well play a major part in the perceived pathologies of ethics review of social science, yet it is largely overlooked by critics (Schrag included) in favour of generalized statements about the application of ‘inappropriate principles’ or ‘silly restrictions’. A rigorous critique would not just list the supposedly ‘silly’ things ethics review bodies do, but rather try to explain how the structure of and systems around ethics review bodies lead them to do these things. A rigorous critique would note that in the NHS, for example, RECs have become progressively ‘de-institutionalized’ over the past 30 years, with formal links between committees and hospitals being eroded to the extent that UK NHS RECs have now had to rename themselves to sever even the appearance of institutional affiliation. Consequently, we might expect institutional factors to play little or no role in the way RECs in the NHS make their decisions. We might also expect a different pattern of problems from US IRBs when it comes to the review of social science.
An even more rigorous critique might note that, in contrast to the NHS, the University REC system in the UK (ESRC, 2005) does resemble the US IRB system in terms of institutional affiliations, and as a result we might expect institutional protection to play a role in these RECs’ decisions. Unfortunately, given that assumed isomorphism means that the social science critique tends to ignore differences between REC systems in different countries, it is unlikely that the differences between systems within an individual jurisdiction will be taken into account.
Thus the core of my criticism of Schrag’s article is that he succumbs to the assumed isomorphism typical of the social science critique of ethics review, and fails to make a convincing case that the problems he identifies with ethics review of social science are relevant to ethics review bodies in general as opposed to those in only one, specific, jurisdiction (i.e. the US IRB system).
That Schrag means his criticisms to apply to ethics review bodies as a whole – as opposed to just IRBs in the US – is, I think, a reasonable conclusion. His terminology is of ‘ethics committees’ (used on almost every page of text in the article) rather than ‘institutional review board’ (which occurs only three times). He refers to US-based ethics review bodies as ‘ethics committees’, for example, when suggesting that Jeffrey Botkin of the US Secretary’s Advisory Committee on Human Research Protections argued that in the US, ‘ethics committees were designed to fix systemic problems in the conduct of medical research’ (Schrag, 2011: 124).
Yet despite his claims being made about ethics review of social science as a whole, the evidence Schrag draws on is overwhelmingly US-based. I estimate that of the 34 citations Schrag uses to make empirical claims about ethics review, 28 (82%) are US based examples, 1 with just 5 cases (15%) coming from non-US systems (Canada, UK and Australia). 2 Like many other US-based critics of ethics review, Schrag’s assumptions about the isomorphic nature of ethics review bodies (‘all such bodies think the same’) is coloured by a US-centric viewpoint (‘all such bodies think like US IRBs’). Yet given this imbalance between US and non-US examples, one cannot reasonably claim that Schrag’s conclusions about the ethics review of social science in general are based on empirical foundations – however true such a claim might be about how social science research is dealt with in the US system.
There are a number of other examples in the article where putative deficiencies of the US IRB system are deemed to be true of ethics review bodies across the board. A good example comes when Schrag discusses the history of the foundation of research ethics committees by suggesting that ‘the ethics review system’ was created with input from ‘medical and psychological researchers’ but not ‘scholars in other social science and the humanities’ and then references his own book in this area (which explores the development of IRBs in the US, but which does not provide a comparative analysis of ethics review systems in other countries). Yet without additional historical evidence, showing how other ethics review systems were set up and organized, this cannot be regarded as anything more than supposition.
Other examples of US evidence serving as the basis for generalized claims about ethics review systems include the lack of training for committee members (Schrag, 2011: 124–125) – which contrasts with the extensive initial and ongoing training programmes offered to REC members in the UK NHS system – and the suggestion that things would improve ‘if most ethics review were done by individual university departments rather than university-wide committees’ (Schrag, 2011: 128) – an approach which is already fairly common in the UK university sector (Newton, 2009; Tinker and Coomber, 2004: 41).
Failure to engage with empirical evidence
While Schrag is correct to note that, given that there is very limited research on the internal processes of ethics review bodies, ‘researchers must guess at what happens to their applications behind closed doors’ (Schrag, 2011: 125), it is unclear why he chose to misrepresent the conclusions of some work that has been done, and ignore other research in this area that he is perfectly well aware of.
The misrepresentation comes in his discussion of Laura Stark’s ethnographic observation of IRB decision-making. Referring to James Gordon Frazer’s classic, The Golden Bough, Schrag suggests that committees’ willingness to judge ‘proposals based on the proportion of spelling and typographical errors in the proposal’ (Schrag, 2011: 125) is an example of ‘homeopathic magic’ – a form of magical thinking where ‘like affects like’ and a magician, by imitating a particular effect, can achieve the same outcome, with effect resembling cause. Frazer gives a good example when he claims that ‘The savage commonly believes that by eating the flesh of an animal or man he acquires not only the physical, but even the moral and intellectual qualities which were characteristic of that animal or man’ (Frazer, 2008 [1922]: 504). In the case of IRBs and the attention they pay to spelling and grammar in informed consent documents, the homeopathic magic comparison would make sense if, when they looked at, commented on and corrected typographical errors, they thought that by doing this they would have an impact on the applicant – that by making his or her documents ‘better’, the applicant would also become better. In essence, the information and influence flows from the IRB to the applicant, via the corrected document.
Yet this is not what Stark claims in her work. Indeed, the information flow she documents is in exactly the opposite direction: from the applicant to the IRB. The kinds of analysis of spelling and grammar – what she calls ‘housekeeping issues’ – were ‘indispensable for IRBs because the apparent degree of care taken in submitting a tidy proposal served as a proxy for a researcher’s ability, allowing board members to make judgments about people’s reliability as investigators. In this way, ink and paper serve as character witnesses’ (Stark, 2006: 172–173; see also Stark, forthcoming). Stark is clearly not claiming that IRB members believe ‘a tidy proposal guarantees an ethical research project’ (Schrag, 2011: 125). Rather, members believe that a tidy proposal tells them something about the applicant and how they might treat people enrolled in their research.
Obviously there is a legitimate debate to be had over whether an applicant’s inability to use a spell-checker genuinely tells us something about how seriously he or she takes the ethics review process or research subject protection, but this is a rational and reasonable position for IRB members to take. Someone who cannot be bothered to spell-check a document may (note may) cut corners elsewhere, or may be under time pressure – relevant points for an body charged with protecting human research subjects to consider. But Schrag is not particularly interested in representing ethics review bodies’ decisions as rational and reasonable, but rather in using Stark’s work to claim that the IRBs she observed engaged in ‘misbehaviour’ (not a term Stark uses) and ‘lack[ed] the expertise to judge proposals on their merits’ (Schrag, 2011: 125).
For my second objection to Schrag’s use of empirical data, we might point out that it is a bit rich for him to complain on the one hand that ‘Supporters of committee review tend to assume or assert its benefits, rather than offering specific examples’ and then, on the other, for him to ignore examples of research that does just that. The work Schrag seems to have ignored is my own. 3 In 2008 I published an article presenting some results from my own ethnographic work on NHS RECs which, while looking at the full range of applications such committees process, included data on the way in which social science was dealt with. My main aim was, through showing the kinds of things that these committees thought were important and the way in which they made their decisions, to argue against the prevailing sociological wisdom and rather to suggest that NHS RECs were capable of reviewing qualitative social science in a reasonable and supportive manner. I suspect that one reason Schrag did not discuss my work is that an article that argues that REC members are supportive of researchers (even qualitative social scientists) by helping them work round bureaucratic problems (Hedgecoe, 2008: 879), reasonable in highlighting the possible risks of power imbalances in workplace research carried out by managers (p. 880), and flexible around the issue of consent in ethnographic research (p. 881), does not fit particularly well with his own antagonistic attitude towards ethics review of social science. Assumed isomorphism is, I think, the other reason for this oversight on his part. He discusses Stark’s work on IRBs, and from an isomorphic position there is no need to review research carried out on other ethics review systems. After all, anything true of the US system must be true of all.
The paradox is that there are examples of UK based ethnographic work that he could draw on to support his position –Dyer’s being the most obvious case (e.g. Dyer and Demeritt, 2009), and this in turn might lead onto a comparison of the conclusions of those who adopt a traditional thick description approach to explore ethics review (such as Stark and myself) to those who provide a ‘thin and wide’ analysis of individual observations at a series of different committees (such as Dyer and Fitzgerald, to whom Schrag does refer). Yet this sort of detailed, nuanced, discussion which acknowledges that ethics review bodies in different countries are not necessarily isomorphic is not the kind of analysis that Schrag is trying to offer.
Conclusions
Schrag may be a good historian of the US IRB system – and his work in this area (Schrag, 2010) suggests that he is – but as a broader critic of research ethics review in the social sciences he offers little that is new. Unlike the best recent scholarship in this area which explores the contextual and structural elements that shape decision-making within specific national contexts (e.g. Van Den Hoonaard, 2011), Schrag adopts the assumed isomorphism of so many critics before him, believing that his criticisms of one particular ethics review system must be true of all – that he is offering a critique of the ethical review of social science in general, rather than a list of complaints relevant to a specific jurisdiction.