Scientific integrity and the market for lemons

Abstract

Scientific integrity cannot be adequately ensured by appeals to the ethical principles of individual researchers. Research fraud has become a public scandal, exacerbated by our inability accurately to judge its extent. Current reliance on peer review of articles ready for publication as the sole means to control the quality and integrity of the majority of research has been shown to be inadequate, partly because faults in the research process may be concealed and partly because anonymous peer review is itself imperfect. Consequently, the scientific literature is mixed, with the reader unable always to distinguish the good articles from the bad. Scientific research is subject to market forces that will always provide a motivation for a range of misdemeanours. This has led to a ‘market for lemons’. Regulations, and sanctions against miscreants, need to be modelled on those historically found necessary to limit financial fraud. Practical and effective systems of process control and audit have already been devised to ensure the integrity of clinical and pre-clinical research. These should be adapted for use in a much wider range of research activities.

An untenable situation

Science is currently struggling with a crisis of confidence in its integrity (Ball, 2012; Ioannidis, 2005; Sarewitz, 2012). The number of high profile cases of scientific publications that have had to be withdrawn have shocked both the scientific community (Godlee et al., 2011; Pashler and Wagenmakers, 2012) and the public (Jha, 2012). In some of these cases professional misconduct has been alleged, offenders have been dismissed and real harm is thought to have resulted. In others the consequence has been wasted resources and damaged personal and institutional reputations (Cyranowski, 2012; White, 2005). Science itself has become an unknown quantity, with uncertainty as to the extent of malfeasance undermining confidence generally.

Possible approaches to improving the reliability and quality of scientific research have been widely discussed and will be addressed again at a 3rd World Conference on Research Integrity in Montreal Canada on 5–8 May 2013. Whether practicable solutions will emerge remains to be seen. Current guidelines (Singapore Statement on Research Integrity) are aspirational and lack any practicable means of enforcement.

Quality control of scientific research

The reliability of current quality control mechanisms has been brought under scrutiny, with numerous authors questioning the historical reliance on peer review as the only guarantee of the quality or honesty of reported research activity (Chubin and Hackett, 1990; Mahony, 1977; Smith, 1997; Young and Carr, 2011). A number of solutions have been proposed, ranging from improvement in the peer review process itself (Sense about Science, 2004) or improvement in ‘research ethics’ (Cyranowski, 2012) to complete third party re-analysis of all data generated by a study (Young and Carr, 2011). Many research institutions have instituted stronger procedures intended to secure higher standards of research integrity. However, most of these proposals focus on improvements only in the quality assessment of the final research output in the form of an article. Few have addressed the need to improve the quality control of each step in the entire process from research proposal to publication.

One important initiative is the requirement to register details of clinical trials, before commencement, and subsequently to publish all data, not merely a selection, when the trial is finished (British Medical Journal, 2012).This approach currently has limited scope, because the vast bulk of research is not conducted in a manner that records all the details of every study comprehensively and accurately, as demanded of drug trials conducted under ‘Good Clinical Practice’ regulations. As Fanelli (2013) recently observed, distorted reporting of scientific results, especially failure to reveal results that are unwanted, is a fundamental weakness in the integrity of current scientific practice.

Detecting misconduct

Attention has also been drawn to the distinction between the two main types of misconduct – falsification of data and incompetence. The importance of each type has been estimated by examination of the articles whose publication has been withdrawn, either by the authors themselves or by the journal that had published them (Fang et al., 2012; Grieneisen and Zhang, 2012).

However, the use of the small proportion of published research articles that have been withdrawn as an index of the quality of the entire scientific enterprise – from research proposal, to funding, to conduct, and finally publication – underestimates the degree of poor practice (De Vries et al., 2006). It is analogous to measuring the reliability of investment banking decisions by the sparse number of ‘rogue traders’ who have been prosecuted. This approach leaves aside the vast number of bad decisions, both large and small, that have brought so many financial institutions to bankruptcy – decisions that were made with due authority of senior management, and were therefore legal, but which were none the less indicative of inadequate quality control, and particularly of over-reliance on ill-conceived mathematical models. Furthermore, few articles will have been withdrawn because of data selection – arguably the most common form of dishonesty (Fanelli, 2013) but the most difficult to detect.

This article will argue that scientific research is subject to market mechanisms that have been described elsewhere in the context of other activities that might be more readily recognized as markets in the conventional sense. It will also argue that the poor quality of much published research is damaging the entire enterprise and creating a ‘market for lemons’ (Akerlof, 1970; Lauer, 2012) in research that is leading to market failure. This market failure is already apparent with the advent of ‘free’ scientific journal publications. The increasing competition among publishers has led to the adoption of a new business model. Instead of relying on income from the sale of a publication, some publishers are offering the product free to the reader. The costs of publication, and a profit for the publisher, may now come from the researchers themselves, rather than from the income from the sale of the scientific journals. It will be further proposed that this form of publication may deepen, rather than relieve, distrust in the integrity of science.

A market for lemons in publishing and information asymmetry

Akerlof (1970) examined the damaging consequences of differences in the information on the quality of goods that might be available to the seller compared to that available to the buyer. He argued that such ‘information asymmetry’ not only reduces the market value of goods but may actually cause the market to collapse. He exemplified the general principle that bad quality goods drive the good out of the market by discussing, among others, the market for second-hand cars. Here, the seller has a relatively clear idea of the quality of the car, whereas the buyer may have very limited information on its faults, and therefore on its real value. As a result, the buyer will be unwilling to pay the appropriate price for a car in good condition, in case there are hidden faults. The seller is only likely to accept a lower offer if the car is, in fact, worth less. Consequently, the owner of a car in good condition will hold onto it until it develops a fault, when he will try to sell it for more than it is really worth. Thus a market for ‘lemons’ (a colloquial American term for cars with faults known to the sellers) is created. Wary buyers will offer lower and lower prices until the market freezes. This is an example of a market where there is an incentive for sellers to offer poor quality merchandise. Their reward is financial: they achieve a selling price above a fair market valuation of their car. But the consequence is a reduction in the average quality of the goods on offer and a contraction of the market as better cars are not offered for sale at all, because they cannot achieve a fair price.

A similar situation arises with scientific publication. Conscientious researchers have little incentive only to offer fully authenticated research results for publication when it is easier to publish preliminary findings. Research articles are offered for sale by commercial publishers. The publisher may have better information as to the quality of the article than the purchaser, because the publisher has the benefit of seeing the whole article and the opinions of peer reviewers on its quality. The purchaser will only see an abstract (sometimes only the title) before he has to decide whether to buy. As a result, it is relatively easy to sell ‘lemons’ into this market.

In a further analogy to the used car market, the publisher has the opportunity to use advertising and flashy presentation of the product as a means to make a sale. Frequently these techniques are deliberately mobilized in such a way as to attract a sale before the quality of the product can be examined by the buyer at all, as in the judicious use of pre-publication press releases. The research article may be presented in such a way as to imply an importance beyond its real scientific merit, and demonstrably unreliable articles are easily disguised as reliable. The common practice of failing to submit press releases or abstracts to peer review exacerbates the risks of misrepresentation.

A research project that yields startling results is more likely to be published than a dull one, irrespective of its intrinsic worth (Nosek et al., 2012). Journal publishers are, after all, competing with each other and aggressively seeking wider readership. And the historic reputation of a journal is no longer a reliable guide to the quality of all its current content. A journal with a solid reputation can take the chance of an occasional bad article if it is sufficiently newsworthy and thus sells well. The trend for reputable journals to include some headline-grabbing articles that have not been subject to peer review at all must raise questions as to the motivation for their publication.

‘Free’ publications

One symptom of a growing unease with commercial scientific publishing has been the advent of ‘free’ or ‘open access’ publications. These rely on payments from researchers to cover the costs (and sometimes also to provide the publisher with profits). Leaving aside the devastating effect this form of publishing may have on the many learned societies that rely on the profit from their journals to survive, this change in the source of revenue may not be as beneficial to quality of scientific publications as some may wish.

The ‘free’ journals may retain a degree of peer review-based quality control, but the fact that the researchers pay the bulk of the costs of publication, rather than the journal purchaser, must weaken the publisher’s responsibility for the quality of the articles and pass this back to the researchers. The consequence will be a weakening of the only form of quality control to which much scientific research is subject – the peer review of published output. This will do little to address the systemic inadequacies of current quality assurance mechanisms for the whole process of scientific research.

The market for lemons in research

Scientific research itself is a market, not merely in its publication phase but also in each of its other stages. Researchers bid for funds in a highly competitive market. Most research is funded by governments. The vast majority of research applications to government funding agencies are rejected. Those making the application may well have a much better idea of the quality of their proposal than those responsible for deciding whether to fund it. Thus information asymmetry exists in this market too, notwithstanding the inherent uncertainty in any proposal that genuinely seeks to advance knowledge.

The likely success of a research application rests at least as much in the quality of its presentation as its intrinsic scientific merit (which may well be virtually impossible to gauge). Thus, the likelihood of a successful application may well rest on the applicant’s salesmanship. Having an established track record of successful research certainly helps, but it should be noted that ‘success’ in this context is largely measured by the number of publications rather than their real quality.

As with the car salesman, the applicant for research money has an incentive to dishonesty, albeit in a form that may be considered minor. The importance of the topic to be researched may be exaggerated; the likelihood of success will always be claimed to be high; and the reliability and honesty of those engaged in the research will always be assumed to be beyond question (despite the fact that, in many cases, they will not have been recruited until after the research funds have been secured). As with any salesman, a researcher’s career success is founded principally on the ability to sell. The quality of the proposal on offer is secondary and the value of subsequent research output even less important. Academic advancement is heavily influenced by the ability to secure research funding, further increasing the temptations to finesse proposals.

This downward pressure on the quality of research proposals may be worsened by two common external factors. The first is a sudden increase in the amounts of money available, and the second is an over-dominance of one particular research hypothesis or group. When money is freely available, researchers will naturally scrabble to obtain a slice of the cake. If, as will usually be the case, the timing of brilliant new ideas does not coincide with the latest government ‘push for progress’, then the quality of projects that are funded will be diluted. If one particular research idea dominates thinking (and the membership of the research funding committees), then a disproportionate amount of money will be allocated to research in a field that may be sterile. This is precisely what happened with President Nixon’s grandiose plans to defeat cancer.

Research integrity

Most recent attention has been directed towards the honesty of the conduct of the research phase itself (although detection still rests on assessment of the final published articles). Here the incentives for dishonesty abound, perhaps more than in any other phase. These can take a number of forms, ranging from the extreme case of fabrication of data (Trager, 2012; White, 2005) to selection of data, including the failure to publish unwelcome results at all (Fanelli, 2013; Ross et al., 2012), to manipulation of calculations (Young and Carr, 2011), misinterpretation of own or others’ data (Ioannidis, 2005), misquotation of earlier research results (Cope and Allison, 2010), and selection of what to publish (Ferguson and Heene, 2012; Sarewitz, 2012).

A number of attempts have been made to address one or more of these issues, but none seems to contemplate the root and branch reform currently needed. For example, some journals have announced that they will only accept articles describing clinical trials if all the data generated in the trial are published at the same time (Godlee, 2012). This addresses the problem of data selection in this form of research but relies on the fact that such trials will have been conducted under a regulatory regime that requires all data to be recorded securely and verified independently. Such detailed scrutiny of the research process has been found to be necessary where drug trials are involved but has yet to be applied to the vast majority of research (Bohan Brown et al., 2013).

Here a further analogy with commercial activity applies. Few would argue that business activity can be relied upon to be conducted honestly without the rigid controls imposed by the existence of internal financial accounting procedures and external audit. Both of these are required by law and their conduct is specified in detail by the relevant legislation. Despite these measures, cases of financial fraud are not unknown. These invariably involve the temporary evasion of the process control mechanisms set in place as a result of hundreds of years of business experience, but these same mechanisms can be expected to expose the misconduct, if not immediately, then at least in the course of time. Crucially, criminal sanctions apply to miscreants.

The existence of such regulatory frameworks as Good Laboratory Practice and Good Clinical Practice (Medicines and Healthcare Products Regulatory Agency, 2012) has long recognized that in some fields of research the incentives to dishonesty are so great that these detailed controls are necessary, despite the additional costs incurred. They do not, of course, focus on financial matters but on the data generated by the research process, and how it is produced, recorded and interpreted. The central objective of these controls is the quality assurance of the process of research itself.

The manufacturing industry has long recognized that the quality of a finished product cannot be guaranteed solely by examination at the end of the production line. Every stage of the process must be controlled and examined. The same process control thinking must apply to all stages of scientific research if the quality and integrity of the final published product is to be reliable (Young and Carr, 2011). It is time to recognize that these same frameworks of process control and audit should apply more widely. Government funders of research should insist on them in any work they fund. Others would soon follow suit.

Regulation or self-correction

It has been argued that science has little need of regulation because it is inherently self-correcting (Loscalzo, 2013). Unfortunately, errors, especially deliberate ones, may take root for years before they are corrected, leading to wasted resources, unjustifiably damaged careers and further erosion of trust in science. It has also been argued that over-regulation, especially in clinical research, is wasting researchers’ time without clear benefit (Loscalzo, 2013; The Academy of Medical Sciences, 2011). Neither of these arguments, despite their undoubted merits, obviates the necessity to impose such minimal regulations on the scientific research process itself as are necessary to ensure that researchers are honest.

Financial controls and audit are accepted in business because they have been proved repeatedly to be necessary. They do not in themselves accuse anyone of malpractice; rather, they are a guarantee of integrity that all may enjoy (even if some are undeserving). In light of evidence of financial as well as scientific fraud in some areas of research (Chemistry World, 2013), these two systems of regulation may need to be more, rather than less, rigorously applied.

Peer review and the withdrawal of articles

As will be apparent from the foregoing, it is no longer sufficient to attempt to control the quality or integrity of the entire scientific research enterprise simply by examining the finished product. Traditionally, the only quality control check of most scientific research has been peer review. This continues to be the case despite doubts as to its effectiveness. Certainly, a large number of fraudulent and incompetent articles have successfully passed peer review in recent years (Grieneisen and Zhang, 2012).

The peer review of articles for publication is such an established feature of the scientific world that it is unlikely to be abandoned in the near future. That being the case, the reliability and integrity of peer review should also be open to scrutiny. The peer reviewer is also subject to temptations, especially to superficiality or bias. It is generally recognized, and rightly applauded, that most reviewers conscientiously avoid both these pitfalls, but this should not blind us to the existence of these pitfalls or to the encouragement, provided by the cloak of anonymity, to fall into them.

Inattention must be the dominant form of siren song assailing the overworked senior researcher. Juniors are rarely asked to be peer reviewers. No doubt many articles are sent to reviewers when they are busy, in subjects of limited interest, and with deadlines that are impractical. The temptation to scan the article superficially must be overwhelming, and the regrettable number of articles published with howling errors that should have been picked up by reviewers is testament to the problem.

More serious still is the risk that a reviewer will rubbish an article because it disagrees with her/his personal prejudices, or because s/he does not approve of the source of funding, or because the article trespasses into territory the reviewer considers their own. Worse still is the urge to reject an article because it challenges work in which the reviewer has a personal stake, or to accept a bad article because it supports their personal bias. This is a significant possibility, because conscientious journal editors will try to send articles to reviewers who are knowledgeable in the subject covered by the article. Bias among reviewers can be demonstrated experimentally (Mahony, 1977), and publication bias arising from this source is an acknowledged issue in certain fields (Cope and Allison, 2010; Ferguson and Heen, 2012; Shriger, 2012). In addition, the reviewer knows that their comments, however prejudiced, ill-informed or self-interested, will never be attributed to them by name, even if the article is subsequently withdrawn for reasons that should have been evident at review. The reviewer’s opinion is paramount, and they cannot be held to account for bad judgment or bias, however blatant. This situation is no longer tolerable, simply because it is an open invitation to malpractice.

The circumstances surrounding anonymous peer review are not dissimilar to those enjoyed in the past by expert witnesses in court cases. Although the expert witness does not enjoy anonymity, s/he was, until recently, protected from the consequences of her/his own fallibility, even if negligent or dishonest (Dowell, 2011). In addition, and in contrast to all other witnesses, the opinion of an expert witness was considered evidence. The catastrophic miscarriages of justice that have resulted from such over-protection of experts are now well known. Currently, the peer reviewer is over-protected, and their opinion (unsupported by evidence) carries enormous weight. Few authors who have regularly published research articles will have escaped the tell-tale signs of an article rejected by two reviewers but for completely different, but equally superficial reasons, or, after answering a first set of objections, the reviewer who raises an entirely new set.

The solution to some of these problems is clear. Peer review should no longer be anonymous. It is an historical anachronism. This suggestion has been made many times but has been rejected by journals in the belief that reviewers would be reluctant to participate if their names were made public. The contrary view, that reviewers may wish their names to be associated with good quality articles, has rarely been tested.

At the least, when an article is withdrawn for faults that should have been apparent to the reviewers, their names should be published. If there are legitimate reasons why these faults were not noticed, then the reviewer’s reputation will not suffer, but it is unacceptable that negligent or biased reviewers should escape all culpability. Similarly, where reviewers pass articles of poor quality simply because poor quality has become acceptable in their field of research, there seems no reason to protect them with anonymity. Knowing that their names may be released would encourage an improvement in the quality they demand of the research articles they recommend for publication. Abolishing anonymous reviewing would help reverse this aspect of the market for lemons.

Sanctions

Misconduct in research may take many forms, ranging from minor errors, through incompetence to outright falsification of data or their analysis. Clearly, improvements will not come unless the sanctions against those guilty of misconduct are more appropriate.

Falsification of data is currently difficult to detect but would be largely prevented (or at least apparent) with the application of the type of process controls required by Good Laboratory Practice or Good Clinical Practice regulations. Manipulation of the analysis of data is suspected to be widespread and would also be detectable by these means, provided someone was willing to repeat the analysis. These areas of misconduct should be subject to criminal sanctions in the same way as financial fraud, because they are essentially aimed at financial benefit, either in the form of career advancement or the procurement of research funds (including a contribution to the fraudster’s own salary).

Incompetence is a more nuanced issue, because its assessment has a substantial subjective element. Research is difficult, or it is not worth doing. We should not be too quick to label a researcher as incompetent. It could be argued that we are currently too slow, especially when ignorance of basic skills required for the job, such as statistical literacy, is apparent. But incompetence is not a criminal matter (provided it is not a cloak for something more sinister) and should be dealt with by an institution’s employment policies and noted by research funders when next an application from this researcher is received.

Conclusions

Scientific research is in crisis, and public confidence has, rightly, been damaged. Current methods of quality control of the complete scientific enterprise rest too heavily on end-product scrutiny in the form of peer review of articles at publication stage. This approach is no longer acceptable, because it is demonstrably inadequate. Research quality cannot be ensured solely by appeals to improve researchers’ personal integrity, simply because research is a market where incentives to dishonesty abound. What is needed is a radical reappraisal of controls at each stage of the research enterprise, focusing on process control and audit. Systems, such as Good Laboratory Practice or Good Clinical Practice, should be applied (mutatis mutandis) to all scientific research, with criminal sanctions for deliberate fraud and the vigorous application of appropriate institutional employment policies to counter incompetence.

The problems with research integrity have been known, but concealed, for many years (Altman, 1994). It is time to clean the Augean Stable, however Herculean the task.