Saturday, March 15, 2014

Over-Enthusiasm in Research Results: How Pharma Finds a Seam

Every so often I comment on a research study that has nothing directly to do with pharmaceutical matters, but which illustrates some basic points about the medical literature and how apparently authoritative studies can be misleading. As is often the case, I am prompted here by Rick Bukata and Jerry Hoffman in their monthly Primary Care Medical Abstracts. Here I'll rely especially on Jerry's commentary on this paper, utilizing his skills as an evidence-based medicine mayvin.

The paper is:

The study was funded by NIH and the authors declare no conflicts of interest, so at first blush this would appear to be a poster child for reliable, unbiased research.

The research question is: If you take a group of folks who have had a stroke of one specific type (lacunar strokes, located in the lower portion of the brain and comprising about a quarter of all strokes caused by blood clots), will they do better in the future if you set a lower (less than 130) rather than a higher (130-149) target for controlling their blood pressure? To study this the research group enrolled about 1900 folks at 81 sites in North America, Latin America, and Spain and followed them for nearly 4 years. The local physicians used whatever medicines they wanted to try to reach the blood pressure goals; and the lower-pressure group did in fact achieve a lower blood pressure overall than the higher-target group (11 points difference).

If you listen to the authors of the paper, they'll tell you:
  • While none of the results of the study were statistically significant, the so-called "trends" were in the direction of fewer strokes of all sorts, and fewer bad vascular events, in the lower-pressure-target group.
  • The lower-pressure target group did not seem to have any more serious side effects from treatment.
  • Other previous studies have tended to show benefits from lower blood pressure in similar populations
  • Therefore, we should recommend setting the blood pressure targets lower for patients who have had a lacunar stroke.
If on the other hand you listen to Dr. Jerry Hoffman's critique (subscription required), he'll tell you:
  • Since no results were statistically significant, that means the study failed to show any real benefit from the lower blood pressure target in these patients.
  • Let's for a minute pretend that all the so-called "trends" were in fact statistically significant. We'd still have to ask how large the resulting differences were. It turns out that the differences are really tiny anyway--often in the fractions of a percentage point.
  • If you are going to look at slight, nonsignificant trends, how about the slightly higher overall mortality rate in the lower blood pressure target arm of the study? Funny how the authors get so excited about "trends" that confirm their hypothesis, and yet ignore similar "trends" that go the wrong way.
So what's going on here? A large group of enthusiastic and dedicated investigators spent years and tons of NIH money at 81 different sites all over the world. You can't blame them if they were desperate to rescue something of value from  their disappointing numbers. Sadly, that doesn't make it good science.

If good scientists like this can publish results that are so little supported by their data, and a supposedly excellent journal lets them get away with it-- then what happens when to the natural enthusiasm of the research team, we add the incentives as well as the incredible monetary resources of the drug industry? You start to get a sense of how much misleading information can appear on the pages of the "best" medical journals, and how long it could take us to sort through it all and decide what's really best for patients.


  1. Please correct me if I am wrong, but their results aren't disappointing, are they? Aren't results just what they are and they provide an increase in knowledge regardless of whatever agenda someone had? (Note, I am unclear on who or what the agenda was).


  2. A saddening observation is that “enthusiastic” investigators like these seem driven by the adage: The name of the game is to keep the game going. That applies equally in academic medicine as in corporate medicine. So we see spinning of data reporting in service of the next grant application just as in service to the next corporate quarterly report and the next advertising campaign. A corollary is that professional and academic organizations develop a sense of entitlement about government and commercial funding, along with a failure to recognize the poor performance.

    There is a discussion of this issue over on the 1Boringoldman blog today that dovetails nicely with this post from Howard.

  3. AnnB, your are right of course, in the ideal world. In the real world two things are the case-- first, scientists don't just report their results, they also offer recommendations; and in this case the scientists recommended using the lower blood pressure target. That would mean more patients taking more drugs and being at risk for more adverse reactions, in order to obtain a benefit that might not exist and that if it did exist seems very small. The second real-world problem is that physicians are very busy and if they read journals at all, they tend to skim and try to pick up the highlights, and so don't critically evaluate the validity of recommendations like this one but are inclined simply to believe them. Thanks a lot for commenting, Howard

  4. Dr. Brody,

    Thank you for another thoughtful and eloquent post.

    One wonders how the Affordable Care Act (ACA) will affect the long-standing trend to over-diagnose and over-medicate chronic diseases -- diseases that were called "lifestyle diseases" back in the days when therapeutic life style change (TLC) was the primary focus of prevention.

    In crude terms, Big Pharma profits most when people are unwell, while private health insurers profit most when people are well. One wonders, then, whether the ascendant economic interests of private insurers will eventually conflict with, and prevail over, the economic interests of Big Pharma.

    Might it prove more profitable for private health insurers to endorse lifestyle interventions that promote a level of overall health and well-being which would limit the cost of treating an anticipated 'epidemic' of chronic disease? In other words, might we see a return to prevention -- in the original sense of the word?

    Do you have any personal knowledge as to whether this is an issue that is being debated, behind the scenes, of the current health care debate?