How to Tell Good from Bad Guidelines, and More on Conflicts of Interest

A recently released paper in the BMJ (subscription will soon be required but apparently it’s openly available for a limited time):
http://www.bmj.com/content/347/bmj.f5535
--features an all-star cast of authors, all of whom we’ve had occasion to praise in earlier posts: Jeanne Lenzer, Jerome Hoffman, Curt Furberg, and John Ioannidis.

The gang tells us that it would be really nice if the medical profession would police the guideline-writing process so that practitioners could be sure that any published guideline was a high-quality review of solid evidence and as free as possible from distorting biases. This is what the Institute of Medicine and others have called for, and so far we’ve done a rotten job of responding. So the burden is on the docs to be critical readers of guidelines, and editors of journal to be pickier about which ones they publish and how much disclosure to demand.

The authors then address the counter-attack we’ve considered here in the past, that financial conflicts are not really important because they are just another form of bias, and bias is everywhere and cannot be eliminated from science. Their reply to this assertion is two-fold. First, they note that other forms of bias tend to be multidirectional; some docs for instance are biased in favor of surgery for a condition while others are biased in favor of drug therapy, and so forth. But when there’s money in play, the bias tends to be unidirectional; all the bias tends to line up on the side that has the deep pocket.

Their second reply is a version of something I wrote in an earlier post:

http://brodyhooked.blogspot.com/2013/06/the-pharmapologists-ballad-same-song.html
Since that was a longish post, the specific point could have gotten buried, so I now take the liberty of reprinting my own comment on the subject:

As an individual academic-medicine investigator, I may have a strongly held belief that the earth is flat. This gives me a strong and pervasive bias. When I do a study, I will selectively seek out evidence showing that the earth is flat and tend to dismiss data showing that it isn't. Given a choice of things to study, my interests will gravitate toward flat-earth issues. My published papers, if the editors allow them to get through, will contain a variety of statements tending in a flat-earth direction. And so on.

There are also many things that won't happen. I cannot assure anyone will give me funding for flat-earth research. I cannot assure that if I get funding, I won't be the only one; there will be a whole stable of investigators all studying flat-earth matters. I cannot hire expert science writers to draft all of my published papers for me to aid them through the editorial review process and to assure that things about flat-earth are slipped in at every possible point in the manuscript. I cannot afford to buy tens of thousands of dollars of reprints of my published articles, making it more likely that journal editors will be swayed to print them. I cannot afford to hold international "consensus" conferences at glitzy five-star hotels, in which all the high rollers in the field are paid big bucks to come and proclaim how flat-earth thinking is the new best thing in medicine. I cannot donate millions of dollars to medical specialty societies to assure that when they write their guidelines on a clinical subject, they'll be sure that paid consultants to flat-earth thinking dominate the panel and write flat-earth-friendly clinical practice guidelines, which will then be imposed on practitioners as "evidence-based." And so on and so on.

There is a big difference, in short, between investigator bias, and well-funded investigator bias.

The authors of the BMJ paper then go on to address two other ways that guideline panels can be inappropriately biased. The first seems initially counterintuitive: filling a guideline panel with experts in that medical specialty. What could possibly be wrong with having heart surgeons, for example, write a guideline on heart surgery? The problem is that there is a peculiar overlap here between intellectual/professional bias and financial conflict. Heart surgeons as a group have one way of looking at the world, that tends to involve a scalpel. In addition to looking at the world that way, they tend to make more money when more patients seek out their particular ways of solving problems. These combined intellectual and financial biases are very hard to self-detect and eliminate.

Who else could be on guideline panels if not content area experts? This is where we need methodologists who understand study design and critical appraisal of the medical literature, who often gravitate toward primary care specialties. An ideal guideline panel would have a majority of method experts and a minority of content area experts.

The final “red flag” to warn of a badly constituted guideline panel is panel stacking. This is likely to happen when the chair of the committee, or many members of the committee, have financial conflicts of interest, or when the sponsoring organization itself as a huge financial stake in industry funding. It’s relatively easy for a few key panelists to cherry-pick other panelists whom they know will agree with their favorite viewpoint, whether or not those individuals personally have financial conflicts.

As a final touch, the authors review several recent guidelines and rate them according to their red-flag list. Not surprisingly, they rate recent guidelines on cholesterol drugs, treatment of depression, and heart stents as pretty crummy, while giving the U.S. Preventive Services Task Force high marks for their stop-using-PSA guidelines.

Lenzer J, Hoffman J, Furberg C, Ioannidis J. Ensuring the integrity of clinical practice guidelines: a tool for protecting patients. BMJ 2013; 347:f5535 doi: 10.1136/bmj.f5535 (published 17 September 2013)