Defending the Hiding of Negative Clinical Trial Data

In the wake of the New England Journal of Medicine study that revealed a sizable discrepancy between the raw data on antidepressants and the data published in medical journals, a few people have jumped to the defense of drug companies. In this post, I will examine their arguments and compare them with evidence. As an extra feature, the DrugWonks' disconcerting love of Vioxx will be discussed toward the end of the post.

"Nathan", who posts frequently on the excellent Pharmalot blog, made several comments. Here's one of them:

For those of you that haven’t written up a scientific article for publication, I’ll make you aware of a few things:
1) Journals don’t accept an article just because it is written. Journal editors scrutinize it and submit it to reviewers who further scrutinize it — very frequently rejecting articles that aren’t conclusive or are poorly written.
2) Scientists write scientific articles — not public relations representatives.
3) Scientists generally don’t waste weeks (or months) of their time writing up articles that don’t prove anything and probably won’t be accepted by the journal. A negative result is generally meaningless in science. Do you think Thomas Edison wrote up articles about the 900 filaments that failed to light a light bulb?

Now I’ll ask again: Is it any surprise that the negative clinical trials were not written up as publications?

Here’s one example: The Paxil trial may have tried doses of 10, 25, and 50 mg. The 10 and 25 mg dose failed to show an effect. The 50 mg dose did show an effect. Why should anyone waste their time writing up a journal article explaining why the 10 and 25 mg dose failed? It’s obvious that drug levels just weren’t high enough to observe an effect.

The problems with the above post:

1. Well, let's face it, peer review does not exactly catch everything. In fact, peer review often results in the publication of very poor studies, like the infamous Paxil Study 329. But it actually appears that Nathan's point is that studies that "aren't conclusive or are poorly written" don't get published. So is he saying that the positive studies of SSRIs were more likely to written better than studies that found negative results? Not sure that makes much sense. Perhaps he is implying that a negative study is not "conclusive" -- well, neither is a positive study, for that matter. When one considers the small benefit of antidepressants over placebo, then even positive studies have not exactly yielded conclusive evidence of anything particularly impressive.

2. Nathan is either naive on this point or is being dishonest. It is well known that ghostwriters are frequently used in the publication process (1, 2, 3, 4, 5). This is not a problem when they are just helping a poor writer piece together a readable manuscript, but this is often a huge problem when one considers that ghostwriters often have a huge conflict of interest in that they are hired by drug companies to pen papers that will paint their product in a favorable light.

3. "A negative result is meaningless in science." That is an amazing comment. Truly amazing. So if 10 trials are conducted on a drug, and one turns up positive, then I suppose everyone should prescribe that drug because the nine negative results were "meaningless." Yeah, that makes a lot of sense.

3.5. The point about dosing would make sense except that it is irrelevant to the case at hand. Remember, Nathan is commenting specifically on the antidepressant studies that did not get published, and he is hypothesizing that the authors of that study included doses that were not approved as effective. Perhaps he missed page 253 of the article, which states:

We included data pertaining only to doses later approved as safe and effective; data pertaining to unapproved dosages were excluded.

Oh, so the authors did not include data for unapproved doses. So much for that critique. Reading a study before offering critiques of it: A good idea.

Nathan in another comment:

I understand how this APPEARS to be a smoking gun. But I’ll state again why I believe it is not: Let’s say I do a clinical trial with Paxil at 10 mg (once per day), 20 mg (once per day), 20 mg (twice per day), and 50 mg (twice per day). That’s a total of 4 trials. Only the 50 mg dose works. Are the other three trials counted as “negative” evidence that the drug doesn’t work? Of course not.

See above for why this critique holds no water.

Nathan:

As Bob pointed out, editors like to accept articles that make their journal look good. NEJM is probably not going to accept a bunch of articles about clinical trials that didn’t work. That doesn’t mean the editors are in some sort of under-the-table deal with the company. It simply means that they are looking out for the best interest of the JOURNAL, not the public.

I think we’re all missing the point. Of course scientists like to publish things that work, not things that fail. The point is that ALL clinical trial data (the good, the bad, and the ugly) is available to the public, the FDA, and anyone else interested. Whether or not the data makes it into a journal is irrelevant.

All clinical trial data is available to the public? Cue the laugh track. Turn it up a little. No, a little more still. Ah, there we go. Who is he kidding? There is absolutely no such thing as a repository for clinical trial data. That is part of the reason why the problem with ghosted science exists in the first place. There are clinical trial registries, but go ahead and check out clinicaltrials.gov. One can outline a study on such a registry then report the results in whatever manner one deems fit, or not at all.

Enter the DrugWonks: And, as you would have guessed, the dependable, fair and balanced DrugWonks weighed in as well. Here are pieces of Robert Goldberg's post:

The NEJM of medicine recycles the old story that many of negative studies about antidepressants were not published. That doesn't affect whether the drugs work or not. It does add to the distortion of what a negative study is and why they are negative. Most of the time they are negative because they simply confirm the hypothesis. Other times they are poorly designed or small studies of little statistical power. They don't prove that the drugs fail. There is a difference. Taken together they can often help guide who responds to what medicines or why not...which again is why we need the Critical Path. To suggest that the failure to publish negative studies is part of a coverup is wrong and leads to fearmongering once again. We have been down this road. And journalists are once again raising unfounded fears about the safety and efficacy of drugs...leading people to die because they stop taking medicines because of the fearmongering the media has engaged in regarding vaccines, SSRIs, Avandia, Vioxx and Vytorin

As for the argument that studies were negative simply because their sample sizes were small, please read the following from the article (page 254):

The difference between the sample sizes for the published studies (median, 153 patients) and the unpublished studies (median, 146 patients) was neither large nor significant.

In other words, Goldberg's analysis is wrong. As I stated above: Reading a study before offering critiques of it: A good idea. As for "Most of the time they are negative because they simply confirm the hypothesis." -- What the hell does that mean? Confirm what hypothesis? This is the sort of seemingly random statement that I have run across frequently on the DrugWonks site. As for the negative studies being poorly designed (so the positive studies are thus designed better?), this seems wrong as well. These studies aren't particularly complex. One group gets drug, the other gets placebo. Participants are assigned to groups randomly. In theory, participants are unaware of if they are receiving drug or placebo. It appears that there were not large differences in how these studies were designed. In any case, Goldberg provides not one single shred of evidence to support his claims. Nathan, mentioned above, also provided no evidence to support his critiques of the NEJM study.

Bring on the Vioxx Love. Goldberg then goes to a new level. He wrote, in part:

And journalists are once again raising unfounded fears about the safety and efficacy of drugs...leading people to die because they stop taking medicines because of the fearmongering the media has engaged in regarding vaccines, SSRIs, Avandia, Vioxx and Vytorin

Let's just focus on one of these medicines: Vioxx. I should mention that as I type this, I am quaking with rage. Goldberg is either incredibly ignorant or incredibly dishonest. For a person who claims expertise in matters of drugs, he should know much better than this. He is claiming that people died because they stopped taking Vioxx. He is apparently serious. Lord Jesus, tell me that this man is joking... First of all, how many people have died because they switched from one minor painkiller to another? Bob, please provide the data. PLEASE.

But (and this is where the rage is coming from), Bob, since you are a Drug Wonk, which would imply at the very least a firm understanding of important and highly publicized findings from the medical literature, how many people died because they did take Vioxx? According to a study by David Graham and colleagues published in the Lancet:

Using the relative risks from the abovementioned randomised clinical trials and the background rates seen in NSAID risk studies, an estimated 88000–140000 excess cases of serious coronary heart disease probably occurred in the USA over the market-life of rofecoxib. [If even a third of these people died, well, you do the math...]

But taking this drug off the market resulted in people dying? Are you *!@! kidding me? Because Graham is a "fearmonger," one of the favorite epithets thrown around on the DrugWonks site, how about we not take his word for it... What have other researchers found? One meta-analysis found an increased risk for heart attacks with COX-2 inhibitors relative to placebo and to some other drugs. Another meta-analysis also found elevated heart attack risk for Vioxx. How about this meta-analysis, which found increased risk for both cardiovascular and renal events on Vioxx? Another meta-analysis found cardiovascular events were related to Vioxx use.

So, Bob, there's my evidence on Vioxx. Where's yours? It's clear that Bob and I differ on a number of points. I think reasonable people can disagree on many items, but Vioxx is not one of them. A drug that has been clearly and repeatedly linked to serious health problems -- and Bob is defending it, while calling those who criticize it "fearmongers." Nice game plan.

A More Reasonable Critique. The Last Psychiatrist offers a more reasonable analysis, suggesting that academic authors did not want to publish the negative studies because publishing negative studies would make them look incompetent -- the mainstream culture of medicine would ask who runs a trial and finds negative results? And even if the studies were submitted to a journal, then they were likely to be rejected by peer reviewers and editors.

This may be true to an extent, but I tend to believe that some negative studies would have been published. There have always been journal editors who have some ability to think critically and publish material that runs counter to that of mainstream medicine. Sure, some of the studies would have been rejected a time or two, but I think they would have been published at some point. While I think his analysis is intelligent (and it thankfully does not involve the term "fearmonger"), I think it only offers a partial explanation.

Missing the Boat. I could certainly have missed something, but I have not even seen a lame attempt to critique part of the study that is quite damning. Remember, the study found that drug companies changed their primary outcome measures and statistical analyses in between submitting to the FDA and submitting for journal publication. This resulted in inflated effect sizes for every antidepressant. Kind of a big deal, as the medical literature ends up suggesting the drugs are more beneficial than they actually are. Not even DrugWonks has mentioned this rather major point.

Oh, and one more thing for Bob Goldberg: Before engaging in a variety of tactics that border on slander regarding Roy Poses of Health Care Renewal (in a recent often nonsensical post that I won't honor by linking to it), you might want to improve your own credibility.

If you missed it, read my post about the NEJM study that some folks are now critiquing.