- 26
- JAN
Posted by Emu under blog's category : Animal welfare & alternatives
Critiques of animal research usually focus on issues such as the need for the study, the number of animals used, and how they are treated. Equally important, but often overlooked, are issues such as whether the experiment is properly designed, and whether the results are properly analysed and communicated.
In December 2009 the National Centre for the Replacement, Refinement and Reduction of Animals in Research authored a paper in PLoS One reviewing experimental design and reporting of research involving animals.
They argued that there is a 'minimum amount of relevant information [that] must be included in scientific publications to ensure that the methods and results of a study can be reviewed, analysed and repeated'. This is especially relevant for animal research as 'omitting essential information can raise scientific and ethical concerns.'
The review looked at 271 papers involving the use of live rats, mice and non-human primates in the UK and US. The analysis covered the quality of reporting, experimental design and use of statistics. Problems were found with reporting and statistical analysis in over half the papers surveyed.
Some of the flaws discovered, to quote Ben Goldacre in The Guardian, were ‘bizarre'. For instance:
- One in 20 did not describe the purpose of the study, and almost as many did not state the number of animals used in the study.
- A quarter did not report the weight or age of the animals.
- Less than 1 in 10 presented raw data for individual animals (important for interpretation of the results by others).
- Almost one third of the papers using statistical methods did not describe the methods.
This situation is far from ideal; the lack of sufficient detail has implications for the scientific validity of these studies. But is it a case of bad design or bad reporting?
The UK has stringent controls on animal research. For example, three separate licences must be granted before any research can take place. The project licences are particularly detailed, and can be up to 100 pages long. They must contain full justification for why the project should take place, detailed descriptions of all the procedures to be carried out, and how the 3Rs have been considered in every step of the experiment planning.
Most of the information missing from the papers from the PLoS One study will therefore be contained in the project licence, including detailed information about the animals. This suggests that in some cases at least the findings of the review are likely to result from bad reporting rather than bad design. Nevertheless, scientific papers should contain as much of this information as possible to allow the study to be repeated, even if it is on the web rather than in print. The authors of the review conclude that it is the responsibility of all involved in publication to ensure the situation improves:
'Scientific publication is a powerful and important source of information; the authors of scientific publications therefore have a responsibility to describe their methods and results comprehensively, accurately and transparently, and peer reviewers and journal editors share the responsibility to ensure that published studies fulfill these criteria.'
The NC3Rs are working with scientists, journal editors and research funders to develop a set of reporting guidelines (not yet available) for animal research, based on the findings of the survey:
‘The guidelines detail the information that all scientific publications reporting animal research should include... the NC3Rs guidelines are suitable for reporting any area of research, and are intended to support researchers, journal editors and peer reviewers to optimise the way that bioscience research is reported.'
We welcome the principle behind these guidelines and the PLoS study. We agree that reporting and transparency surrounding animal research should be improved. However, the NC3Rs must ensure that its guidelines are helpful for researchers and journals and not so far-reaching that they are unworkable.
Our partner site animalresearch.info has more information about experimental design.
IMAGE©ISTOCKPHOTO.COM/IDEABUG
Get in-depth information on the contribution of animal research to medicine
Comments
No doubt few studies included raw data from individual animals because at most individual cases can only be illustrative of a pattern seen in the group. Individual raw data is not necessary to interpret group data per se - the descriptive and inferential statistics are.
Also, ages might not have been reported in many studies because some suppliers of laboratory rodents sell them by weight rather than age. Also, it should be noted that in many cases reporting the weight would make replication of the research possible, and that it is possible for many rat and mice strains to infer an animal's stage of development from their weight.
Having said all of that, I have reviewed manuscripts for many scientific journals in my field and some researchers have become very sloppy in their reporting of experiments. I believe some of this complacency results from working in a field for a long time. If a researcher has performed a procedure over and over again, then it becomes so routine that reporting all the details seems repetitive. Indeed, the NC3Rs review in PLoS One does not describe how the authors scored papers in which methodological details are given by reference to previously published sources.
Looking at the missing information they list I would not expect it to be found in most non-clinical or basic research studies whether or not they were animal studies, indeed it is still often missing from clinical studies. The authors of the PLoS study point this out themselves. I'm not sure that I've ever read an in vitro study that included randomisation and blinding. No doubt this reflects a cultural difference where clinical studies are expected to inform clinical practice, whereas the results of basic research will usually be expected to be confuirmed or refuted by independent repitition and follow-up research.
I would agree that in most cases the issue is one of reporting rather than design, and now that far greater detail can be added to papers as supplemental online information there is really no excuse to not demand it from authors.
Randomisation and blinding are another issue, where possible they should be performed, but in animal research they are not always possible. Blinding in particular is difficult when the study is performed by a small research group or even just one scientist, in contrast to clinical trials which usually involve a much larger number of personnel. Publishers, reviewers and readers need to be pragmatic about this.
One area where I would agree that randomisation and blinding - or more likely their reporting - should be demanded is in pre-clinical or translational studies that are intended to inform the decision to take a new treatment into clinical trials. Given the potential risks and the certain expense they should be held to a standard comparable to that of clinical trials. Given the progress made in improving the design and reporting of clinical trials in the past decade this would seem the logical next step to concentrate on.