Thursday, June 18, 2009

Case Study: Detecting Marketing Bias in Sponsored Research

I would suppose that a newcomer to this debate over Pharma influence, who does not routinely read medical journal articles, would have the following reaction to many of the arguments put forth in previous posts:

Why are you making such a big deal about industry sponsorship of research? It must be fairly easy to read a medical journal article, assess the methods employed, and decide that a study is of good or poor quality--that one can or cannot trust the findings. Isn't that the key variable--that one can or cannot trust the findings--rather than whether or not the study was sponsored by a drug company?

For this reason I feel justified in placing on record a case study, showing just how far one often needs to "drill down" to discover the evidence of bias in a commercially sponsored research publication. For this example I must thank my friends Rick Bukata and Jerry Hoffman and their excellent audio service, "Primary Care Medical Abstracts."

For their May, 2009 issue, Rick and Jerry discussed a paper by Ho et al. in Lancet (a top tier journal, which would lead some to imagine immediately that no possible skullduggery could have made it past the distinguished editors of this journal). The basic idea here is that Merck (who funded the study and whose role is clearly labeled in the paper) has an investigational drug for migraine headache that is in a new class, calcitonin gene-regulated peptide receptor antagonists. Merck obviously hopes that this new medicine, when approved, will quickly become the preferred therapy for migraine in place of the older standbys, the triptan drugs, which are currently going off patent and becoming cheaper.

The study was set up to compare two doses of the new drug, called telcagepant (we assume Merck will think of a melodious brand name instead of this nearly-unpronounceable generic name), the older standard drug, zolmitriptan at maximum recommended dose, and placebo. They looked at a variety of migraine symptoms at both 2 and 24 hours after the initial drug dose.

How did the drugs compare? Here's how the authors of the paper reported their outcomes in the abstract (the only part of the paper that many busy clinicians ever read): "Telcagepant 300 mg is effective as an acute treatment for migraine with efficacy comparable to that of zolmitriptan 5 mg, but with fewer associated adverse effects." In other words the new drug is clearly better than the old drug.

The abstract prepared by the Primary Care Medical Abstracts team expressed the same data but with a more cautious spin: "In this study, a 300mg dose of...telcagepant was not more effective that zolmitriptan for the treatment of acute migraine."

But to get the full story we needed to wait till Jerry Hoffman had his say on the audio disk. Jerry pointed out a feature of the study design that was not mentioned in the abstract at all, and that was only discussed in the methods section and in Figure 1. The study design called for the option for any subject to receive a second dose of medication at 2 hours past the initial dose, if needed. This feature makes sense because migraine headaches often rebound within the first 24 hours or are not completely relieved by the first medication dose, or both. If the subject was assigned to the placebo group or the zolmitriptan group, the second dose would be placebo. If the subject was assigned to either of the telcagepant groups, the second dose was randomized, half the subjects getting a second dose of telcagepant, the other half getting placebo.

Now, stop and think. Can you imagine any legitimate scientific question that would be answered by having the study design as described? I cannot imagine one. I have to conclude that this study design must have been selected purely for its marketing impact, and not as a way of doing science.

So you can see that the playing field was not level--as Jerry pointed out. A single dose of zolmitriptan was being compared to the possibility of repeated doses of telcagepant. (In actual fact, about half of all the subjects required a second dose, so a total of a quarter of the telcagepant subjects got a second dose of an "active" drug instead of placebo.)

This understanding of the true study design puts quite a different spin on the reported outcomes. What if half the zolmitriptan subjects, who needed a second dose, had been given a second dose of zolmitriptan? Would telcagepant still have been "as effective as" zolmitriptan--or would the older standby drug, and the soon to be the probably much cheaper drug, have proven superior?

You can pretty much answer this question for yourself by studying the finer details of Table 3. Efficacy for each study endpoint was expressed as a percentage of subjects who achieved the target outcome. In almost every single category, at 2 hours, the percentages reaching target were higher for zolmitriptan than for either dose of telcagepant, with the single exception for phonophobia (being bothered by loud noises). In photophobia (being bothered by bright light), the higher dose of telcagepant was slightly better than zolmitriptan (51% vs. 50%). For example, a secondary endpoint, "total migraine freedom [at] 2 hr" was achieved by 22.9 % of the higher-dose telcagepant group, and by 27.2% of the zolmitriptan group. The only endpoints for which there was any superiority for telcagepant were those endpoints that looked at sustained effects over 24 hours--where, as we saw, the playing field was not level and a quarter of the subjects in the telcagepant groups got an extra medicine dose. Even there, the differences were nothing to write home about--for "2-24 h sustained pain freedom," 20.2% of the higher dose telcagepant group, vs. 18.2% in the zolmitriptan group.

Exactly how did the methods section of the paper explain the strange methods, by which one active medication group got a second dose that was denied to the other? Here's what Ho et al say: "The efficacy of the optional second dose is not discussed here since the sample sizes were limited. A prespecified meta-analysis across all of the telcagepant phase III studies is planned to address the therapeutic effect of the second dose." Pardon my cynicism but I interpret this statement as: "We won't report the role of the second dose in skewing the study findings now, because then the reader might actually put two and two together. Instead we'll report them (if we ever get around to it at all) separately at a later time, when we hope you'll have forgotten about the issue, and when we can put a favorable spin on whatever we find one way or another." As I have demonstrated, even without an explicit report from the authors, the raw data in the tables, which the average reader would never bother to look at, hint very strongly that the optional dose at 2 hours skewed the results in favor of telcagepant--and without that skew, the expensive new drug would in all probability not have performed as well as the older and cheaper drug.

I know I put you to sleep a long time ago. My take home message is very simple--this is often how carefully you have to read a published article in a medical journal to be sure you have detected any bias that apparently was introduced by commercial sponsorship. And that, in turn, is why commercial bias and conflict of interest is such a plague on scientific medicine.

Ho TW, Ferrari MD, Dodick DW, et al. Efficacy and tolerability of MK-0974 (telcagepant), a new oral antagonist of calcitonin gene-related peptide receptor, compared with zolmitriptan for acute migraine: a randomized, placebo-controlled, parallel-treatment trial. Lancet 372:2115, Dec 20/27, 2008.


Anonymous said...

Looks like you caught them with their telcagepants down. Well done.

TSC said...

As a matter of fact, Merck announced a delay of its NDA submission for telcagepant because of the cases of liver toxicity. So telcagepant may never happen, thank goodness. Besides, the FDA people are experts in reading through the statistical crap, so Merck would have had hard time with it.

Anonymous said...

There is a very basic problem with this critique of the study by Ho et al.: a single dose of zolmitriptan is the current standard dose. Although a second dose can be administered, there is no evidence that it improves efficacy. To quote the prescribing information: “Controlled trials have not adequately established the effectiveness of a second dose if the initial dose is ineffective.” It may “make sense” that an additional dose would improve efficacy, but that hypothesis is not supported by data.

Therefore, the selection of single-dose zolmitriptan in the active comparator arm was justified.

Indeed, allowing patients to take a second dose of zolmitriptan would have raised much more serious questions about the trial design. The results of this trial demonstrated that telcagepant has similar efficacy to zolmitriptan but is less toxic. If patients had been allowed to take a second dose of zolmitriptan they would likely have experienced more adverse events and the investigators would then have been open to the criticism that they allowed a non-standard dosing regimen in order to make zolmitriptan appear less safe.

The investigators should also get some credit for designing their trials to investigate whether the second dose of telcagepant improved efficacy rather than being berated for it. Determining the efficacy of a second dose in a single clinical trial will likely be difficult. Because the effect on outcomes of the second dose is probably quite small and because it is not possible to know whether patients will require a second dose or not, to get a patient population large enough to investigate this question it would be necessary to run a very very large trial. That would be prohibitively expensive and ethically dubious (since the data from the ~50% of patients who required only one dose would have to be discarded). Randomizing the second dose across several phase III trials so that patients received either placebo or telcagepant and then performing a meta-analysis to determine the efficacy (or lack thereof) of a second dose seems a quite clever solution to this problem. In fact they should be applauded for conducting the research rather than simply leaving the question unanswered (as it is for zolmitriptan). There certainly doesn’t seem to be any reason to disbelieve their statement regarding the prespecified metaanalysis.

In future, it would be appropriate to consider alternative explanations (particularly those that are more straightforward) before claiming to have identified "marketing bias".

Disclosure: I am currently employed as a consultant to the Biotech industry. I have never worked for Merck or for any company that produces a therapy for migraine (or for any other CNS disease for that matter). I am a former member of the faculty of an academic medical center in the United States.

Howard Brody said...

First off let me thank "Anonymous" for leaving a detailed comment. He is quite correct-- we must be careful in assessing the methods of company-sponsored trials, and not be too quick to charge commercial bias if there indeed could be a legitimate scientific basis for the design features. I welcome calling attention to the finer points for discussion.

Now, having said that, let me point out some considerations that would support the assessment I offered in my original post (with thanks again to my esteemed EBM colleague Jerry Hoffman of UCLA):

1. Anonymous says that the "current standard dose" of zolmitriptan is a single dose and that "controlled trials have not established the effectiveness of a second dose..." The common clinical usage is to repeat doses of triptans given the frequent refractory and/or recurring feature of migraine. There may be many reasons why controlled trials have not demonstrated the efficacy of a second dose, for example, maybe no one did the trials; but the fact that the efficacy of a second dose has not been established does not equate with "we know that a second dose is ineffective." If there is a true scientific need to document the effectiveness of a second dose of zolmitriptan, the study could have been designed with the zolmitriptan arms exactly the same as the experimental drug arms--that is, half the folks get a placebo second dose and the other half get an "active" drug second dose.

2. If the study design as it stands is scientifically valid, there remains the question of why one has to dig so deep into the fine print, as it were, to find out that half the telcagepant subjects got a second dose compared to none of the zolmitriptan subjects.

3. Even with the nonlevel playing field favoring telcagepant over zolmitriptan, the investigators (according to my friend Jerry) "could only manage an underpowered non-inferiority design." The combination of these two features suggests strongly to me that marketing and not science was at the root of the design decisions.