In recent months, several important books and articles have appeared that jointly help us understand much better how we can be misled about the value of new pharmaceuticals from reports in the medical literature. In a later post I'll try to pull all the strands together to give a big picture. Here I want to get on record a very nice article by a major expert in research analysis, that contributes some of the key threads. (Hat tip to Rick Bukata and Jerry Hoffman at Primary Care Medical Abstracts for citing this paper.)
Our expert of the day is John Ioannidis from Greece, whose work on debunking the claims of the research literature have even made it into the popular press. The article in question appeared in the BMJ (subscription required).
We have focused a lot in previous posts on one way a drug company can mislead us--suppress negative research data and spin neutral data to make it seem positive. Ioannidis asks this question: how can we be misled even if the company is scrupulously honest in reporting the data?
There are two major ways, the authors report, and they can be illustrated by a specific case study, the research history of tumor necrosis factor blocker drugs for cancer and rheumatoid arthritis. (I'll here summarize the general points and you can read the paper if you want the details of the TNF story.)
First, drug companies typically try out a drug on numerous conditions, hoping to expand the sales potential. Typically, for each condition, the drug is tested against an array of outcomes, as many as 10-20 per study. (For example, a cancer drug might be reported in terms of outcomes such as survival at 3, 6, 9, 12, 15, and 18 months, as well as quality of life measures, time to first metastasis, etc.) Statistician Ioannidis reminds us (the reminder really shouldn't be needed) that we can do the math and calculate how many of these outcome measures will be positive simply by chance, assuming that the drug is actually no better than placebo--or in the more common case, is a little bit better than placebo, while maybe also having some significant adverse reactions and a high cost. If you looked at 20 outcomes per trial, and conducted trials for the drug in 6 different medical conditions, the odds are that for each condition, at least one outcome will be statistically significant in favor of the drug. If the company plays its cards right, it can get regulatory approval to market the drug for all 6 conditions, even though the results, so far, occur purely at random and indicate no real benefit.
The second mechanism is one we've previous looked at, early stopping of trials. In those earlier posts (such as http://brodyhooked.blogspot.com/2009/11/no-fair-peeking-more-questions-about.html), I erred in focusing on the question of whether the company inappropriately pressured the data safety and monitoring committees to end the trials early in ways that benefitted marketing. Ioannidis shrewdly reminds us that we don't have to assume any skullduggery to see how stopping trials early could exaggerate the drug's efficacy. Suppose we simply do what DSM committes are routinely told to do--for ethical reasons so that research subjects are not put at unnecessary risk. If a treatment reaches a pre-specified level of statistical significance showing superiority, then the trial is stopped, on the belief that you have the answer and that continuing the trial longer would not change things. But that's surely wrong, the authors say, because of the well-known phenomenon of regression to the mean. If at any given stage in the research, the drug is beating the placebo by let's say 20%, if you quit then, you report that the drug is better than placebo by 20%. Yet if you'd continued the trial longer, the odds are excellent either that the drug would have turned out to be no better at all, or else that the true degree of superiority is 5% or 10%, not 20%. The authors cite a previous paper that analyzed 91 early-stopped trials and demonstrated these effects clearly.
On this topic I like to use the analogy of a horse race. We all know that the right way to run a race is to run a given distance, and the first horse across the finish line is the winner. Supposing that we decided that a statistically significant lead is 1-1/2 lengths. So we develop a new rule, that as soon as one horse is out in front by at least 1-1/2 lengths, then you stop the race and declare that horse the winner, even if the field has only gone a quarter of the distance. How often do you think the winner by these new rules would be the same horse as would have won by the old rules?
Ioannidis adds a further wrinkle. If you stop a trial early, the results are also published early. If you let a trial run its normal course, the results will be published much later. A trial that's stopped early is stopped because the treatment looks very good early on. A trial that is not stopped is therefore a trial where the treatment does not look so much better for most of the trial duration. That means almost certainly that the first data published about a new drug will be unrepresentatively positive, and that the more negative data will come rolling in much more slowly. (Or not at all if we add the common tactic of data suppression to the company's bag of tricks.)
Bottom line: if companies were scrupulously honest in reporting data, you could still end up concluding that a new drug is much more effective than it really is. But we know, as documented here ad nauseam, that all too often this scrupulous honesty is honored in the breach rather than the observance. So if you add a little sprinkling of dishonesty or spin to the factors Ioannidis cites, then you have an even more misleading picture.
Ioannidis proceeds to explain how the authors of systematic literature reviews and meta-analyses can try to correct for these sources of bias. But for us the main lesson is to understand these sources of bias and how they operate. Later I'll try to connect the dots between these concepts and other recent analyses of sources of bias in the research literature and its interpretation.
Ioannidis J, Karassa F. The need to consider the wider agenda in systematic reviews and meta-analyses. BMJ 341:761-64, 9 October 2010.
Wednesday, March 16, 2011
Monday, March 14, 2011
PR, Sock Puppets, and the Ways of Corporations
This rather incoherent post arises from a conjunction of two events--first, seeing the following post--http://www.cbo.gov/ftpdocs/88xx/doc8891/MainText.3.1.shtml-- on our fellow blog, Health Care Renewal; and second, being in the process of reading Wendell Potter's book, Deadly Spin, as recommended by HCR's Dr. Roy Poses in his comment to my March 7 post (http://brodyhooked.blogspot.com/2011/03/and-another-defense-of-medicine-pharma.html).
HCR's Dr. Scot Silverstein, their regular blogger on matters relating to health information technology and electronic records, using technology that is far beyond my poor powers, was able to track down an anonymous person who regularly left disparaging, ad hominem comments about any criticism of the HIT industry or its products. He found that the messages originated in a computer located in the headquarters of an HIT firm in Massachusetts. As soon as he "outed" the source, the comments from that anonymous individual ceased.
Dr. Silverstein thereby introduced me to the useful term "sock puppet"--a shill who is in the employ of or sympathetic to a moneyed interest, and who attacks opponents of that interest with misdirection, obfuscation, or ad hominem invective, all the while concealing the link between the attacks and the moneyed interest.
Back to Wendell Potter. His book reviews how the health insurance industry has responded to all attempts at health reform in the US with well-financed and highly effective PR campaigns either to defeat reform outright (as with Clinton), or else to be sure that reforms take whatever shape will best preserve the profits of the private insurance industry (as with "Obamacare," that supposedly socialist program). The primary tool of these PR compaigns is the creation of phony organizations fully funded by the insurance industry but supposedly made up of grass roots supporters ("astroturf"), that can parrot the talking points that the insurance poobahs have refined and field-tested, but while making it appear that the statements come from anywhere except the insurance companies. The goal is to make it seem as if "everyone is saying that" when "that" in fact was deliberately invented and promulgated by the insurance folks.
It is instructive to compare the standard PR procedures of the insurance industry with those of the drug industry. While there are some differences, the basic approach seems to be the same. As anthropologist Kalman Applbaum (see previous post, http://brodyhooked.blogspot.com/2010/06/how-does-drug-industry-exert-power.html) shows, the special insight that Pharma has added is the notion of a drug "channel," the entire collection of events that must occur between the time a new drug is discovered and when it is sold to patients. The drug industry has become adept in creating marketing strategies that manage to control an entire channel. This serves the same purpose as insurance industry PR. To the average spectator, it seems simply inconceivable that the drug companies could control all these disparate players--scientists, physicians, the FDA, celebrities who mention a new drug on TV, etc. Therefore the sheer audacity of the industry strategy renders it invisible, in a sense--it simply does not seem possible that all those inputs could be deliberately orchestrated.
Potter W. Deadly Spin: An Insurance Company Insider Speaks Out on How Corporate PR Is Killing Health Care and Deceiving Americans. New York: Bloomsbury Press, 2010.
HCR's Dr. Scot Silverstein, their regular blogger on matters relating to health information technology and electronic records, using technology that is far beyond my poor powers, was able to track down an anonymous person who regularly left disparaging, ad hominem comments about any criticism of the HIT industry or its products. He found that the messages originated in a computer located in the headquarters of an HIT firm in Massachusetts. As soon as he "outed" the source, the comments from that anonymous individual ceased.
Dr. Silverstein thereby introduced me to the useful term "sock puppet"--a shill who is in the employ of or sympathetic to a moneyed interest, and who attacks opponents of that interest with misdirection, obfuscation, or ad hominem invective, all the while concealing the link between the attacks and the moneyed interest.
Back to Wendell Potter. His book reviews how the health insurance industry has responded to all attempts at health reform in the US with well-financed and highly effective PR campaigns either to defeat reform outright (as with Clinton), or else to be sure that reforms take whatever shape will best preserve the profits of the private insurance industry (as with "Obamacare," that supposedly socialist program). The primary tool of these PR compaigns is the creation of phony organizations fully funded by the insurance industry but supposedly made up of grass roots supporters ("astroturf"), that can parrot the talking points that the insurance poobahs have refined and field-tested, but while making it appear that the statements come from anywhere except the insurance companies. The goal is to make it seem as if "everyone is saying that" when "that" in fact was deliberately invented and promulgated by the insurance folks.
It is instructive to compare the standard PR procedures of the insurance industry with those of the drug industry. While there are some differences, the basic approach seems to be the same. As anthropologist Kalman Applbaum (see previous post, http://brodyhooked.blogspot.com/2010/06/how-does-drug-industry-exert-power.html) shows, the special insight that Pharma has added is the notion of a drug "channel," the entire collection of events that must occur between the time a new drug is discovered and when it is sold to patients. The drug industry has become adept in creating marketing strategies that manage to control an entire channel. This serves the same purpose as insurance industry PR. To the average spectator, it seems simply inconceivable that the drug companies could control all these disparate players--scientists, physicians, the FDA, celebrities who mention a new drug on TV, etc. Therefore the sheer audacity of the industry strategy renders it invisible, in a sense--it simply does not seem possible that all those inputs could be deliberately orchestrated.
Potter W. Deadly Spin: An Insurance Company Insider Speaks Out on How Corporate PR Is Killing Health Care and Deceiving Americans. New York: Bloomsbury Press, 2010.
Monday, March 7, 2011
And Another Defense of the Medicine-Pharma Status Quo...
Back over to Health Care Renewal, this time from Dr. Roy Poses, commenting on a Medscape piece by a surgeon, Dr. Frank J. Veith. As you need to be a Medscape subscriber to access the original, see Dr. Poses' post at http://hcrenewal.blogspot.com/2011/03/once-more-with-feeling-another-defense.html.
Dr. Veith is highly exercised about busybodies like us pharmascolds who would mess up the cozy financial relationships between the drug industry and docs. So he writes a piece that's virtually a Xerox copy (says Dr. Poses) of a bunch of pieces that have been published over the past 5 or so years. All, he says, contain the same logical fallacies and the same insistence on the huge benefits of taking cash from Pharma, and the dire dangers of ceasing to do so, with nary a hint of evidence to back up the claims.
Given what we know about the prevalence of industry ghostwriting in the scientific literature, one has to wonder--is it just that these pharmapologists are an unimaginative bunch and can think of nothing new or fresh to say? Or is it that the industry hacks are writing this stuff for them, and the same draft keeps circulating?
Dr. Veith is highly exercised about busybodies like us pharmascolds who would mess up the cozy financial relationships between the drug industry and docs. So he writes a piece that's virtually a Xerox copy (says Dr. Poses) of a bunch of pieces that have been published over the past 5 or so years. All, he says, contain the same logical fallacies and the same insistence on the huge benefits of taking cash from Pharma, and the dire dangers of ceasing to do so, with nary a hint of evidence to back up the claims.
Given what we know about the prevalence of industry ghostwriting in the scientific literature, one has to wonder--is it just that these pharmapologists are an unimaginative bunch and can think of nothing new or fresh to say? Or is it that the industry hacks are writing this stuff for them, and the same draft keeps circulating?
Industry Doublespeak, This Time in IT
Our friends over at the Health Care Renewal blog have been providing us much useful material as of late. This time Dr. Scot Silverstein, their persistent critic of electronic health records that have not been properly field-tested before the software companies unleash them on a helpless public, looks at a recent report from the health information technology industry consortium group. He finds it full of gobbledygook, as he puts it, perhaps most notably the idea of "usability maturity model." The industry poobahs admit that their confreres might be hesitant to worry about "usability" of their products, as demanding that docs and nurses actually might be able to use the dang thing before you sell it could cut into sales and profits. But they reassure doubters among them that "usability" is actually a good thing to worry about, because it has a positive ROI (return on investment). Once again, the idea that you might actually make more money if you sell a product that's useable, as opposed to one that's not, seems a revolutionary idea.
Dr. Silverstein is right to be appalled at this tone when we see that the IT industry basically subsumes the notion of patient safety under their idea of "usability." The very idea that the IT industry has to be convinced that worrying about whether your electronic record will cause patients to die is only a good idea if it leads to greater profits gives him the willies.
From our standpoint, we can wonder what will happen when Pharma comes upon this highly promising notion of the "usability maturity model." We will then hear that Vioxx, for instance, did not really cause the estimated 144,000 cases of excess heart disease in the US before it was yanked from the market. The only problem was that its usability matured a little bit too slowly. (Enjoy Dr. Silverstein's blog posting at http://hcrenewal.blogspot.com/2011/03/himss-and-health-it-we-dont-need.html.)
Dr. Silverstein is right to be appalled at this tone when we see that the IT industry basically subsumes the notion of patient safety under their idea of "usability." The very idea that the IT industry has to be convinced that worrying about whether your electronic record will cause patients to die is only a good idea if it leads to greater profits gives him the willies.
From our standpoint, we can wonder what will happen when Pharma comes upon this highly promising notion of the "usability maturity model." We will then hear that Vioxx, for instance, did not really cause the estimated 144,000 cases of excess heart disease in the US before it was yanked from the market. The only problem was that its usability matured a little bit too slowly. (Enjoy Dr. Silverstein's blog posting at http://hcrenewal.blogspot.com/2011/03/himss-and-health-it-we-dont-need.html.)
AMSA: Tide Has Turned in Medical Schools
AMSA, the American Medical Student Association, can justly claim pride of place in the near-revolution that has occurred in academic medicine over the past half-decade. AMSA developed the astoundingly successful propaganda ploy, their "report card" on institutional COI policies. Deans that had never been bothered about how many of their faculty were in bed with Pharma (just keep those lucrative grants coming in, boys and girls) suddenly developed an abiding concern about COI when it was splashed across the local news that their medical school had been given an F on AMSA's report card. My own stomping ground, UTMB-Galveston, went from an F to an A pretty promptly when spurred on by AMSA (though personally I would not place my own Dean in the "never been bothered" category).
AMSA now reports a new survey showing that now a majority of US medical schools have policies that they consider "strong" in policing drug industry COI. They also add that, "Nearly one-third of medical schools now teach medical students to understand institutional conflict of interest policies, to recognize how industry promotion and marketing can influence clinical judgment and to consider the ethics around conflict of interest." See the entire AMSA press release at http://www.amsa.org/AMSA/Homepage/About/News/121510.aspx.
AMSA now reports a new survey showing that now a majority of US medical schools have policies that they consider "strong" in policing drug industry COI. They also add that, "Nearly one-third of medical schools now teach medical students to understand institutional conflict of interest policies, to recognize how industry promotion and marketing can influence clinical judgment and to consider the ethics around conflict of interest." See the entire AMSA press release at http://www.amsa.org/AMSA/Homepage/About/News/121510.aspx.
Just How Does One Find New Drugs?
Daniel Cressy, in an informative brief news item in Nature (subscription required), tells us something that most readers of this blog either knew or suspected--that the current drug industry research model for discovering new medications is largely a bust. The industry apparently realizes it too, as Pfizer announces a $1.5B cut in its 2012 R&D budget, and closes a major research facility in the UK.
But what's the answer? After apparently working hard to shift research funding out of academia and into the private sector, the industry may be deciding that academic centers are in fact the very best place to discover promising new molecules. Presumably, according to Cressy, the industry is quite right that it can do very effectively and efficiently what the academics can do only slowly and ploddingly--clinical trials to demonstrate the effectiveness of drugs that have shown promise in Phase I trials. But it should stop trying to do those Phase I trials themselves, or the painstaking basic work leading up to them, and let the academic centers do that heavy lifting, with the companies later swooping in to buy up rights to the promising results.
Cressy reports that this is what the companies now do with drugs for so-called orphan diseases, and basically it is a matter of extending the orphan-disease model to their operations as a whole. The big problem now looming, Cressy concludes, is that ideally everyone involved wants somebody else to pay for this whole process; and academic research is unlikely to produce the hoped-for gains unless both governments and industry are willing to pony up.
Cressy may have offered us a good analysis of cutting-edge thinking about drug R&D, but he totally ignores any issues relating to conflicts of interest. He talks about the possibilities of "long-term partnerships" between drug companies and academic centers, without indicating what these partnherships would look like and how academic values are to be maintained if the industry piper is calling the tune. One promising development, however, is the apparent realization among some in industry that proprietary secrecy works against scientific discovery. Cressy cites Patrick Vallance, senior VP for medicines development at GlaxoSmithKline in London, as a proponent of "open innovation," putting potential molecular structures into the public domain and inviting academics to have a go at taking them to the next level.
Cressy D. "Traditional Drug-Discovery Model Ripe for Reform." Nature 471:17-18, 3 March 2011.
But what's the answer? After apparently working hard to shift research funding out of academia and into the private sector, the industry may be deciding that academic centers are in fact the very best place to discover promising new molecules. Presumably, according to Cressy, the industry is quite right that it can do very effectively and efficiently what the academics can do only slowly and ploddingly--clinical trials to demonstrate the effectiveness of drugs that have shown promise in Phase I trials. But it should stop trying to do those Phase I trials themselves, or the painstaking basic work leading up to them, and let the academic centers do that heavy lifting, with the companies later swooping in to buy up rights to the promising results.
Cressy reports that this is what the companies now do with drugs for so-called orphan diseases, and basically it is a matter of extending the orphan-disease model to their operations as a whole. The big problem now looming, Cressy concludes, is that ideally everyone involved wants somebody else to pay for this whole process; and academic research is unlikely to produce the hoped-for gains unless both governments and industry are willing to pony up.
Cressy may have offered us a good analysis of cutting-edge thinking about drug R&D, but he totally ignores any issues relating to conflicts of interest. He talks about the possibilities of "long-term partnerships" between drug companies and academic centers, without indicating what these partnherships would look like and how academic values are to be maintained if the industry piper is calling the tune. One promising development, however, is the apparent realization among some in industry that proprietary secrecy works against scientific discovery. Cressy cites Patrick Vallance, senior VP for medicines development at GlaxoSmithKline in London, as a proponent of "open innovation," putting potential molecular structures into the public domain and inviting academics to have a go at taking them to the next level.
Cressy D. "Traditional Drug-Discovery Model Ripe for Reform." Nature 471:17-18, 3 March 2011.
Thursday, March 3, 2011
Good Summary of Markingson Case at U-Minn
I've commented several times previously on the Markingson case, most recently at http://brodyhooked.blogspot.com/2010/12/u-minn-bioethicists-call-for.html. Dr. Carl Elliott, the U-Minn bioethicist who's done the most to publicize this case and keep the heat on the University, offers a recent blog post--
http://www.thehastingscenter.org/Bioethicsforum/Post.aspx?id=5147&blogid=140
--on the Hastings Center's bioethics blog. The focus of this latest post is to rebut the University's claims that there's nothing more to investigate about the case since the FDA already looked into it and gave the U. a clean bill. Dr. Elliott analyzes the FDA "investigation" and details its shortcomings. However, I'd recommend this post primarily as a very concise review of the facts of the case for anyone looking for same. There are so many inconsistencies remaining that U-Minn still has a lot to answer for.
http://www.thehastingscenter.org/Bioethicsforum/Post.aspx?id=5147&blogid=140
--on the Hastings Center's bioethics blog. The focus of this latest post is to rebut the University's claims that there's nothing more to investigate about the case since the FDA already looked into it and gave the U. a clean bill. Dr. Elliott analyzes the FDA "investigation" and details its shortcomings. However, I'd recommend this post primarily as a very concise review of the facts of the case for anyone looking for same. There are so many inconsistencies remaining that U-Minn still has a lot to answer for.
Subscribe to:
Posts (Atom)