The Depressing Results of the NEJM Antidepressant Study

Do antidepressants work?

If you read the studies submitted by Pharma to the FDA, but never published, the answer is no.

If you read the published studies, the answer is yes.

As I reviewed extensively in HOOKED, it is old news that papers published in medical journals, of research funded by industry, disproportionately favor the company's drug; and that research studies showing the drug in a bad light have routinely been suppressed. What's newsworthy about the paper by Erick Turner and colleagues in the January 17 New England Journal is the breathtaking extent of the resulting bias.

The group had access to the FDA reviews of 74 company-sponsored studies of 12 different antidepressants. The data showed whether the study was viewed as positive or negative in terms of the drug's superiority to placebo; and the size of the positive effect if there was one. They then tracked down whether each study was eventually published in a journal, and whether one reading the article would classify the study as positive or negative.

Of 38 studies deemed positive by the FDA reviewers, 37 were published. Of 36 studies deemed to be negative or questionable, 22 were not published at all, and 11 were published with an interpretation that made them sound positive, so that the true negative outcome was concealed in one way or other in 33/36 studies.

There was also the interesting mystery of the effect size. The effect size reported in the published papers (which, remember, were virtually all positive) exceeded the effect size calculated by the FDA reviewers by an amount ranging from 11 to 69 percent. The median increase in effect size was 32 percent, meaning that on average, published papers made the drugs sound one-third more effective than the scientific data (at least according to the FDA) would support.

In other words, if we assumed that an evil demon was at work, trying to make it impossible for the readers of medical journals to know the truth about antidepressant efficacy, we would expect results precisely matching those actually uncovered by Turner et al.

So, can we give this evil demon a name? Turner et al. demur. They have no way of knowing whether all these papers were submitted to journals, and were turned down by editors because they were negative, and who wants to read a boring ol' negative study; or whether the drug company never submitted the papers for publication, or some of each.

Now, a couple of comments at this point. First, it seems that once we have identified this set of 74 studies, we have an extremely useful database for further inquiry. On patient investigation, it ought to be quite possible to determine the submission-to-journals track record (or lack of same) for each of the 74 studies (or rather, the 23 that were not published according to Turner and company). Since Congress is starting to interest itself in these matters, including such arcane issues as Robert Jarvik's qualifications to shill for Lipitor on TV, it's even conceivable that if all else fails, the 23 PI's of those studies could be subpoenaed to Capitol Hill to testify under oath. Then we would know a lot more.

Next comment--if a paper is not published, it may be the drug company not submitting it, or it may be the editor rejecting it. But if the paper is published, and the effect size has magically grown like Pinocchio's nose, can we really suspect that perhaps the editor personally rewrote the manuscript and made the company's drug sound better than it is? Or do we have to suspect the drug company in-house statisticians of doctoring the manuscript? To me, the company is much more likely to be the culprit. And, while awaiting further evidence, if the company is the culprit in that part of this intrigue, I have to be suspicious that non-submission to journals can be laid at that doorstep also.

Having said that, I still must admit that journal editors have not in the past acted in a way that would make them seem blameless in all this. In both the VIGOR and CLASS trials, as related in HOOKED, we have some strongly suggestive evidence that the editors of major journals (NEJM and JAMA) colluded with the academic authors of drug company sponsored trials, to make the results sound better and the risks sound less than they really were. Richard Smith, former BMJ editor, has written about the millions of dollars major journals can make when drug companies buy up reprints of highly successful clinical trials. So we need to keep in mind the possibility of collusion in the antidepressant trials as well.

Meanwhile, I have another question. One of my jobs is teaching medical students. Up till now, I have always taught them that a good doctor carefully reads the medical journals and decides how to treat patients based on what we read there. How can I go on telling them this, with a straight face?

Turner EH, Matthews AM, Linardatos E, et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 358:252-260, 2008.