A while ago I posted about a study by Mathieu et al. in JAMA that appeared to suggest that registering clinical trials was not having the hoped-for benefits on the transparency of journal publications of trial results: http://brodyhooked.blogspot.com/2009/09/registering-clinical-trials-less-than.html. Another paper published shortly after that one by Robert Ewart and colleagues from Southern Illinois U.--
http://www.annfammed.org/cgi/reprint/7/6/542
--adds to our level of concern that trial registries are not having the impact we had sought.
Ewart and friends looked at 158 reports of randomized clinical trials published in high-impact journals in 2008. They were able to analyze 110, including only trials that had been registered. Comparing the trial as described in the registry entry with the final report in the journal, they discovered that 31% of the time, a primary outcome had been changed, and in 70%, a secondary outcome had been changed, without these being mentioned in the publication.
Unlike the earlier paper, Ewart did look at trial sponsorship, but they found no difference related to industry vs. non-commercial sponsorship in whether an outcome had been altered.
Ewart and colleagues suggest that at least some of this problem may result from investigators' lack of familiarity with the registry process, but agree that a serious problem with the interpretation and integrity of trial results arises when changes are made but not explained in the methodology section.
I go back to the same point I made in the previous post--where are the journal editors or reviewers? What is the appropriate division of labor here? Is the reviewer supposed to be the one to check the trial registry, or is the journal editor? Do the instructions to reviewers make clear which is the case? Information is being made readily available now on the internet that is designed to allow us to have greater reassurance that what we read about a clinical trial is kosher--yet people responsible for the journals and their content are ignoring that information. It's ridiculous to say that each journal reader is responsible for going to the web to look up the trial on the registry. If a manuscript is submitted to a medical journal, and something has been altered without explanation from when the trial was first registered, then the editors should demand an explanation or else refuse to publish the paper. What is going to change, and when?
Showing posts with label trial registries; medical journals; suppression of data. Show all posts
Showing posts with label trial registries; medical journals; suppression of data. Show all posts
Wednesday, June 2, 2010
Sunday, September 6, 2009
Registering Clinical Trials--Less than Meets the Eye?
The current JAMA features an article (subscription needed to access) by an international team of authors headed by Dr. Sylvain Mathieu of Paris. The authors decided to study what has happened as a result of the requirements from leading medical journals (initiated in 2005) that all clinical trials should be properly registered--in theory allowing anyone to compare the final, published results of the trial with the original design.
They found 323 trials that were published in 2008 in high-impact medical journals, either general or in one of 3 specialties (cardiology, rheumatology, gastroenterology). They discovered that only 45% of these were adequately registered. Others were not registered until the trial had been completed (14%), and with no or an unclear description of the primary outcome (12%). More than a quarter were not registered at all.
In about half the cases, there were insufficient data to determine what sort of bias would have been introduced by a change in the primary outcome from that originally identified in the study methods to the one featured in the published report. In the other half, in 19 of 23 studies, an outcome for which the results were statistically insignificant was replaced by an outcome whose results were significant.
Naturally a person like me would wonder--and what association did commercial sponsorship have with whether the authors ended up playing fast and loose with the study design? About 56% of the trials in this review were commercially sponsored (and sponsorship was not reported in another 9%). That would seemingly have allowed some comparisons to be made. But these authors reported no data based on associations with commercial sponsorship.
The most worrisome finding was that the authors found evidence of selective outcome reporting in 28% of studies that were properly registered. This suggests that neither the editors nor the reviewers took the time and trouble to use data available in open trial registries to see whether the outcomes reported in the final publication were indeed the outcomes listed in the pre-trial study design. In short, in these instances, the whole reason for trial registration was subverted by the failure of journals to take advantage of the data.
So let's leave aside the question that we wish had been answered, but was not--whether commercially sponsored studies were more likely to be registered incorrectly or incompletely, or to have partial or biased reporting of endpoints. We have long asked--why do articles get published in major medical journals that are ghostwritten or that suppress key data in the interests of marketing drugs? The usual reply from the journal editors is that they are not detectives. If the authors flat-out lie to them about who wrote the article, or what the endpoints were, how is the journal going to smell a rat? In a previous post (http://brodyhooked.blogspot.com/2009/08/read-em-and-weep-wyeth-ghostwriting.html) I noted the criticism that even if journal editors lack the detective facilities to identify ghostwritten articles, that hardly explains why no article has as yet been officially retracted by a journal once ghostwriting was proven by another route. Now, in light of the Mathieu study, journal editors need to explain why they cannot be bothered to cross-check trial reports against registered data that are available in plain sight.
It appears that it's not simply the case that journal editors make rotten detectives. Journal editors don't seem to run a very tight ship even when other people do the detective work for them and hand them the results neatly giftwrapped.
Clinical trial registries were supposed to solve this problem. Do we have here yet another example of Epstein's Law--"If you think the problem is bad now, just wait till you've solved it"?
Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302:977-84, Sept. 2, 2009.
They found 323 trials that were published in 2008 in high-impact medical journals, either general or in one of 3 specialties (cardiology, rheumatology, gastroenterology). They discovered that only 45% of these were adequately registered. Others were not registered until the trial had been completed (14%), and with no or an unclear description of the primary outcome (12%). More than a quarter were not registered at all.
In about half the cases, there were insufficient data to determine what sort of bias would have been introduced by a change in the primary outcome from that originally identified in the study methods to the one featured in the published report. In the other half, in 19 of 23 studies, an outcome for which the results were statistically insignificant was replaced by an outcome whose results were significant.
Naturally a person like me would wonder--and what association did commercial sponsorship have with whether the authors ended up playing fast and loose with the study design? About 56% of the trials in this review were commercially sponsored (and sponsorship was not reported in another 9%). That would seemingly have allowed some comparisons to be made. But these authors reported no data based on associations with commercial sponsorship.
The most worrisome finding was that the authors found evidence of selective outcome reporting in 28% of studies that were properly registered. This suggests that neither the editors nor the reviewers took the time and trouble to use data available in open trial registries to see whether the outcomes reported in the final publication were indeed the outcomes listed in the pre-trial study design. In short, in these instances, the whole reason for trial registration was subverted by the failure of journals to take advantage of the data.
So let's leave aside the question that we wish had been answered, but was not--whether commercially sponsored studies were more likely to be registered incorrectly or incompletely, or to have partial or biased reporting of endpoints. We have long asked--why do articles get published in major medical journals that are ghostwritten or that suppress key data in the interests of marketing drugs? The usual reply from the journal editors is that they are not detectives. If the authors flat-out lie to them about who wrote the article, or what the endpoints were, how is the journal going to smell a rat? In a previous post (http://brodyhooked.blogspot.com/2009/08/read-em-and-weep-wyeth-ghostwriting.html) I noted the criticism that even if journal editors lack the detective facilities to identify ghostwritten articles, that hardly explains why no article has as yet been officially retracted by a journal once ghostwriting was proven by another route. Now, in light of the Mathieu study, journal editors need to explain why they cannot be bothered to cross-check trial reports against registered data that are available in plain sight.
It appears that it's not simply the case that journal editors make rotten detectives. Journal editors don't seem to run a very tight ship even when other people do the detective work for them and hand them the results neatly giftwrapped.
Clinical trial registries were supposed to solve this problem. Do we have here yet another example of Epstein's Law--"If you think the problem is bad now, just wait till you've solved it"?
Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302:977-84, Sept. 2, 2009.
Subscribe to:
Posts (Atom)