Evidence Biased Medicine (To the Core)

The Last Psychiatrist wrote an excellent comment regarding a post describing how "scientific" data and/or its analysis and interpretation are often cooked by ghostwriters and/or friendly academics. He discussed how the whole process of publishing research is biased, a point that will be discussed in depth throughout this lengthy post. The Last Psychiatrist said:

Sure Pharma puts pressure on doctors, and forces through studies that are helpful to them (and suppresses those that hurt them).

But the real problem in medicine is the academic centers. Their bias is dangerous because it's so subtle and pervasive.

If Astra Zeneca does a Seroquel study, I think we can guess the bias. But when Assistant Professor Jones does a Seroquel study-- funded by the NIH-- is that study magically free of bias? What about Jones's beliefs on medications (he thinks pharmacotherapy is a gold mine, or is he anti-drugs and pro therapy?; maybe he's pro-seizure drugs (Depakote, Lamictal) and anti-antipsychotics (or the other way around?) Maybe his mentor gets AZ money (which is used to pay his salary through the university?) Maybe NIH has a stake in getting expensive drugs like Seroquel to look bad (e.g. CATIE?)

And journals are worse: think that the editors of a journal don't have biases-- even direct pharma ones?

And the three peer reviewers?

Ever wondered what articles don't get accepted for publication, and why (and I say this as someone who has a pretty high rate of publication success).

And why do those journals-- which publish public data-- cost $1000/yr and can't be accessed by the public?

The first and most important step to fixing medicine is abandoning the journal system. All articles, including the raw data that generated them, photos, scientific notebooks, etc, should go online. Let the world vet the data.

I agree with the great majority of what he said. I find it hard to believe that NIH is against expensive medications, since many of the NIH folks have ties with drug companies which are, of course, pushing newer and more expensive meds.

We have to keep in mind that the whole "scientific" peer-review process includes a lot of bias.

Let's review how studies go from a set of numbers into a published manuscript. It may sound like a dull process, but this is the foundation of our so-called evidence base in medicine, so it is actually very important to understand.

How is it biased? Let me count the ways...

Step 1. Analyzing data and writing the paper. Researchers transform a bunch of numbers into a paper.

1) It's anyone's guess as to whether the researchers have actually seen the data upon which they are to base their writings. In some cases, it is unlikely that they have. So the company could have already made some alterations to the data -- it's unclear how often this happens, but it is certainly a possibility in some (hopefully rare?) instances.

2) The company can analyze the data in any way it sees fit. Go to Aubrey Blumsohn's site for an excellent example of why this can be problematic. Company statisticians can cook the books either overtly or in a more subtle manner (like they did with the Seroquel data -- here and here).

3) The company can interpret the numbers in any way it wants. For example, if someone committed suicide while taking a drug, the drug couldn't have caused it, but if the patient committed suicide on a placebo, then the placebo caused it. Even when the data are not favorable, positive conclusions are reached in most instances (here, for example).

4) The company can bury any unfavorable data. Suppose that depression was measured in five different ways. If a couple of those measures yielded unfavorable results, toss them aside and act as if they never existed. Don't even mention that they existed in the article.

5) When all else fails, deep-six the study. If the data still fail to prove favorable, just bury the entire thing -- don't publish it. When lawyers and/or researchers get their hands on unpublished data, it quite often shows unfavorable results which the sponsoring company thought best to bury.

Step 2. Peer Review. The paper is then sent off to "experts" for peer review. As the LP said earlier, these folks (including me) have their biases. Indeed, one of my peers has called the peer review process "a Rorscach test of the reviewers," meaning that you can easily see their biases through the reviews. Most reviewers of psychiatry journals have ties to industry which have likely shaped their beliefs to roughly the following: "Drugs are safe and effective," though biases will vary.

The comments of these expert reviewers are quite important in determining whether the study will get published.

Here's what one former journal editor, Richard Smith (British Medical Journal) had to say about peer review (with my emphasis):

The problem with peer review is that we have good evidence on its deficiencies and poor evidence on its benefits. We know that it is expensive, slow, prone to bias, open to abuse, possibly anti-innovatory, and unable to detect fraud. We also know that the published papers that emerge from the process are often grossly deficient.

Hmmm. No, this is not sour grapes on my part -- I've little to complain about in terms of being published. But myself and many other researchers are often befuddled by the whole process -- it often seems that reviewers are unhelpful.

Does peer review help, at least a little bit? I think so. Does it solve the problem of low-quality papers hitting journals, which are then turned into marketing copy by the drug and device industry? Obviously not.

Step 3. Editoral Decision. The editor chooses whether to accept the paper (usually after some revisions are made).

Journal editors frequently have huge ties to industry. Just google the names of many editors and you'll find that they have received funding from a lot of different sources. We also know that sometimes peer reviewers make good comments, yet the editor chooses to ignore them.

Note that nearly all journals are a for-profit entity. How can they make money? Advertising, subscriptions, and reprints. If a journal runs an article favorable to industry (saying that vagus nerve stimulation is great for depression, for example), then it is likely that the company will buy thousands of reprints for dissemination to physicians. The journal is making good money from each reprint and can make tens of thousands or even up to a million dollars from reprints of a study. A study that is unfavorable or irrelevant to industry is not going to generate revenue for the journal. So from a business standpoint, it makes more sense to print studies (like this or this) that are written from a slant of favoring a product than to run something less industry-friendly.

What to do?

Start by making all trial information publicly available. The Last Psychiatrist said it. Richard Smith said it, and I agree with it. I don't think we should abolish journals altogether -- seems extreme, but making data publicly available -- that's an excellent idea.

Penalize those who engage in misconduct
As Fiona Godlee (editor of the British Medical Journal) stated recently:

So what can we do to change the blind-eye culture of medicine? In the interests of patients and professional integrity I suggest intolerance and exposure.

--SNIP--

And if journals discover authors who are guests on their own papers, they should report them to their institution, admonish them in the journal and probably retract the paper.

Reputations for sale are reputations at risk. We need to make that risk so high it's not worth taking.

Other thoughts?

Update (2-9-07): Quite a few people have been reading this via reddit. To see comments regarding this post at reddit, click the following link.