Abstract
Poorly designed and analysed experiments can lead to a waste of scientific resources, and may even reach the wrong conclusions. Surveys of published papers by a number of authors have shown that many experiments are poorly analysed statistically, and one survey suggested that about a third of experiments may be unnecessarily large. Few toxicologists attempted to control variability using blocking or covariance analysis.
In this study experimental design and statistical methods in 3 papers published in toxicological journals were used as case studies and were examined in detail. The first used dogs to study the effects of ethanol on blood and hepatic parameters following chronic alcohol consumption in a 2 × 4 factorial experimental design. However, the authors used mongrel dogs of both sexes and different ages with a wide range of body weights without any attempt to control the variation. They had also attempted to analyse a factorial design using Student's t-test rather than the analysis of variance. Means of 2 blood parameters presented with one decimal place had apparently been rounded to the nearest 5 units. It is suggested that this experiment could equally well have been done in 3 blocks using 24 instead of 46 dogs. The second case study was an investigation of the response of 2 strains of mice to a toxic agent causing bladder injury. The first experiment involved 40 treatment combinations (2 strains × 4 doses × 5 days) with 3-6 mice per combination. There was no explanation of how the experiment involving approximately 180 mice had actually been done, but unequal subclass numbers suggest that the experiment may have been done on an ad hoc basis rather than being properly designed. It is suggested that the experiment could have been done as 2 blocks involving 80 instead of about 180 mice.
The third study again involved a factorial design with 4 dose levels of a compound and 2 sexes, with a total of 80 mice. Open field behaviour was examined. The author incorrectly used the t-test to analyse the data, and concluded that there was no dose effect, when a correct analysis showed this to be highly significant.
In all case studies the scientists presented means α standard deviations or standard errors involving only the animals contributing to that mean, rather than the much better estimates that would be obtained with a pooled estimate of error. This is virtually a universal practice. While it is not in itself a serious error, it may lead scientists to design experiments with group sizes of at least 3 animals, which may result in an unnecessarily large experiment if there are many treatment combinations.
In conclusion, all 3 papers could have been substantially improved, with higher precision and the use of fewer animals if more attention had been paid to better experimental design.