Abstract
This is the second in a series of articles on making evidence-based medicine work for you. The series is based on the book ‘Atteni Alle Bufale’ by Tom Jefferson (www.attentiallebufale.it).
Quick Instrument (Figure 1)
Quick pathway to follow to minimize the risk of being taken for a bufala ride when scanning an original research paper.
If the topic interests you, go to the bottom of the Introduction section. If you cannot find a clearly defined objective, bin the study. Think twice about reading the journal again. If the study has a clear objective, go to a results table, add up a column at random and match the results with the total. If they do not match, bin the paper. Remember that if you are adding percentages, the total may be slightly over or under 100% because of rounding. If you have a little more time, read the study backwards, starting with the Discussion. If the bits do not fit logically, bin the study. Alternatively, match the content of the abstract (the shop window of the paper) with the content of the text. Again, bin the study if they do not fit. Crazy? Some wag found errors and discrepancies in 68% of studies in one sample . 1
Full Instrument
There are some very useful full instruments to help in this case. The CONSORT statement is perhaps the most famous (http://www.consort-statement.org/Statement/revised-statement.htm). CONSORT was developed to make reporting of randomized trials clearer, but it is a useful checklist to guide you through the intricacies of trials . There is also a separate statement covering cluster-randomized trials. 3
For cohorts and case-control studies, use the Newcastle-Ottawa scale, or NOS (http://www.ohri.ca/programs/clinical_epidemiology/oxford.htm). The NOS takes a while to get used to but it is a good, relatively quick instrument to use 4 .
To assess the quality of diagnotic studies try using the QUADAS instrument (http://www.biomedcentral.com/1471-2288/3/2S) 5
The rules for their reporting are addressed in the STARD instrument (http://www.consort-statement.org/Initiatives/newstard.htm). 6
Rationale
Research papers should be written according to IMRAD (Introduction, Methods, Results and Discussion). They should accurately reflect what the authors intended doing, what really did happen and what they found. However, the majority of research papers one reads while undertaking a systematic review are unreliable for a variety of reasons, regardless of who wrote them or where they were published. The quick instrument is the fastest way of spotting bufale lurking between the folds of the paper. Also beware of perfectly written, immaculate papers, especially those reporting large trials. It is likely that these have had the input of one or more ghost authors—in other words, that they are marketing instruments of one kind or another.
This apparently conflicting advice (bin what is badly written and beware of what is written too well) demonstrates how difficult it can be to spot these bufale. The golden rule is beware at all times. If something appears too good or too bad to be true, it probably is.