An Update on Bias in Industry-Funded Trials

Sorry that this pair of publications went past me when they first appeared last year, but I doubt many of us keep copies of this journal on our bedside tables:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2868984/pdf/Dtsch_Arztebl_Int-107-0279.pdf
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2872821/pdf/Dtsch_Arztebl_Int-107-0295.pdf

This German team led by Dr. Gisela Schott set out to update us about the influence of drug company sponsorship of clinical trials on the reliability of the published outcomes. They reviewed two important systematic reviews both published in 2003, and then surveyed the literature from when those studies left off until 2009, to see what the more recent studies showed. They identified 57 published studies that compared industry funded trials to those funded by other sources, or that were otherwise relevant.

Overall they replicated the results of the earlier systematic reviews, that there's a clear association between industry funding a trial and the results turning out positive for the company's drug. Again they demonstrated that this is unlikely to be due to poorer quality of the methodology of the industry trial.

They then tried to drill down a bit to see why this is the case. They noted that many unfavorable studies (possibly half) are not published, and many positive studies tend to be published multiple times. (I suppose, on average, that makes it all okay.) There is a considerable bias in what results are included in published work, with repeated evidence that adverse drug reactions are simply buried whenever and however possible. Despite the rise of clinical trial registries during the years covered by this new study, and industry group policies promising compliance with the journals' requirements that trials be registered, there are still industry-sponsored trials that are not being registered, and many "registered" trials are registered with key data missing.

The problems do not appear to be completely outside the controls of the journal editors, and the authors note the conflicts of interest that journals face when they depend on reprint and advertising sales to drug companies. The authors make a proposal that seems long overdue--that journals should be required to publish detailed financial statements annually. As I explained in HOOKED, the balance sheets of medical journal publishers were at the time that I wrote one of the hardest-to-locate (actually, in most cases, impossible to locate) data points in the whole business of the medicine-Pharma interface.

The other problem with journals is well illustrated by a separate article by Emerson and colleagues (subscription required; hat tip to Primary Care Medical Abstracts for this citation). These folks did a sneaky number on reviewers for a couple of orthopedics journals (though with prior consent). They concocted a fictitious trial and sent a manuscript for review. They also planted a few small errors in the Methods section of the manuscript, though by most standards the overall quality of the study would be graded as excellent. In half the cases, the results were shown as positive; in the other half everything else about the study was the same, but the results were negative.

Emerson and colleagues found that the positive version was recommended for publication 97% of the time but the negative version only 80% of the time. The reviewers were more likely to pick up on the planted errors if they were sent the negative version of the study, and if sent that version, they rated the overall methodologic quality lower. In short, reviewers of these journals were biased against negative and in favor of poisitive studies, and I doubt very much that this is somehow unique to orthopedics. Bottom line: when we look for reasons that negative drug studies are suppressed, we have to look both at industry manipulation and also at the inherent biases of journal reviewers and editors.

Emerson GB, Warme WJ, Wolf FM, et al. Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Archives of Internal Medicine 170:1934-39, Nov. 22, 2010.