The
Last Psychiatrist wrote an excellent
comment regarding a
post describing how "scientific" data and/or its analysis and interpretation are often cooked by ghostwriters and/or friendly academics. He discussed how the
whole process of publishing research is biased, a point that will be discussed in depth throughout this lengthy post. The Last Psychiatrist said:
Sure Pharma puts pressure on doctors, and forces through studies that are helpful to them (and suppresses those that hurt them).
But the real problem in medicine is the academic centers. Their bias is dangerous because it's so subtle and pervasive.
If Astra Zeneca does a Seroquel study, I think we can guess the bias. But when Assistant Professor Jones does a Seroquel study-- funded by the NIH-- is that study magically free of bias? What about Jones's beliefs on medications (he thinks pharmacotherapy is a gold mine, or is he anti-drugs and pro therapy?; maybe he's pro-seizure drugs (Depakote, Lamictal) and anti-antipsychotics (or the other way around?) Maybe his mentor gets AZ money (which is used to pay his salary through the university?) Maybe NIH has a stake in getting expensive drugs like Seroquel to look bad (e.g. CATIE?)
And journals are worse: think that the editors of a journal don't have biases-- even direct pharma ones?
And the three peer reviewers?
Ever wondered what articles don't get accepted for publication, and why (and I say this as someone who has a pretty high rate of publication success).
And why do those journals-- which publish public data-- cost $1000/yr and can't be accessed by the public?
The first and most important step to fixing medicine is abandoning the journal system. All articles, including the raw data that generated them, photos, scientific notebooks, etc, should go online. Let the world vet the data.
I agree with the great majority of what he said. I find it hard to believe that NIH is against expensive medications, since many of the NIH folks have ties with drug companies which are, of course, pushing newer and more expensive meds.
We have to keep in mind that the whole "scientific" peer-review process includes a lot of bias.
Let's review how studies go from a set of numbers into a published manuscript. It may sound like a dull process, but this is the foundation of our so-called evidence base in medicine, so it is actually very important to understand.How is it biased? Let me count the ways...
Step 1. Analyzing data and writing the paper. Researchers transform a bunch of numbers into a paper.
1) It's anyone's guess as to whether the researchers have actually seen the data upon which they are to base their writings. In some cases, it is unlikely that they have. So the company could have already made some alterations to the data -- it's unclear how often this happens, but it is certainly a possibility in some (hopefully rare?) instances.
2) The company can analyze the data in any way it sees fit. Go to
Aubrey Blumsohn's site for an excellent example of why this can be problematic. Company statisticians can cook the books either
overtly or in a more subtle manner (like they did with the Seroquel data --
here and
here).
3) The company can interpret the numbers in any way it wants. For example, if someone committed suicide while taking a drug, the drug
couldn't have caused it, but if the patient committed suicide on a placebo, then the placebo
caused it. Even when the data are not favorable, positive conclusions are reached in most instances (
here, for example).
4) The company can bury any unfavorable data. Suppose that depression was measured in five different ways. If a couple of those measures yielded unfavorable results, toss them aside and act as if they never existed.
Don't even mention that they existed in the article.
5) When all else fails, deep-six the study. If the data still fail to prove favorable, just bury the entire thing --
don't publish it. When lawyers and/or researchers get their hands on unpublished data, it quite often shows
unfavorable results which the sponsoring company thought best to bury.
Step 2. Peer Review. The paper is then sent off to "experts" for peer review. As the LP said earlier, these folks (including me) have their biases. Indeed, one of my peers has called the peer review process "a Rorscach test of the reviewers," meaning that you can easily see their biases through the reviews. Most reviewers of psychiatry journals have ties to industry which have likely shaped their beliefs to roughly the following: "Drugs are safe and effective," though biases will vary.
The comments of these expert reviewers are quite important in determining whether the study will get published.
Here's what one former
journal editor, Richard Smith (British Medical Journal) had to say about peer review (with my emphasis):
The problem with peer review is that we have good evidence on its deficiencies and poor evidence on its benefits. We know that it is expensive, slow, prone to bias, open to abuse, possibly anti-innovatory, and unable to detect fraud. We also know that the published papers that emerge from the process are often grossly deficient.
Hmmm. No, this is not sour grapes on my part -- I've little to complain about in terms of being published. But myself and many other researchers are often befuddled by the whole process -- it often seems that reviewers are unhelpful.
Does peer review help, at least a little bit? I think so. Does it solve the problem of low-quality papers hitting journals, which are then turned into marketing copy by the drug and device industry? Obviously not.
Step 3. Editoral Decision. The editor chooses whether to accept the paper (usually after some revisions are made).
Journal editors frequently have huge ties to industry. Just google the names of many editors and you'll find that they have received funding from a lot of different sources. We also know that sometimes peer reviewers make good comments, yet the editor chooses to
ignore them.
Note that nearly all journals are a for-profit entity.
How can they make money? Advertising, subscriptions, and reprints. If a journal runs an article favorable to industry (saying that vagus nerve stimulation is great for depression, for example), then it is likely that the company will buy thousands of reprints for dissemination to physicians. The journal is making good money from each reprint and can make tens of thousands or even up to a million dollars from reprints of a study. A study that is unfavorable or irrelevant to industry is not going to generate revenue for the journal.
So from a business standpoint, it makes more sense to print studies (like this or this)
that are written from a slant of favoring a product than to run something less industry-friendly. What to do?
Start by making all trial information publicly available. The Last Psychiatrist said it.
Richard Smith said it, and I agree with it. I don't think we should abolish journals altogether -- seems extreme, but making data publicly available -- that's an excellent idea.
Penalize those who engage in misconductAs Fiona Godlee (editor of the British Medical Journal) stated recently:
So what can we do to change the blind-eye culture of medicine? In the interests of patients and professional integrity I suggest intolerance and exposure.
--SNIP--
And if journals discover authors who are guests on their own papers, they should report them to their institution, admonish them in the journal and probably retract the paper.
Reputations for sale are reputations at risk. We need to make that risk so high it's not worth taking.
Other thoughts?
Update (2-9-07): Quite a few people have been reading this via reddit. To see comments regarding this post at reddit, click the following
link.
The next logical step from Healy's accurate description:
For all of those patients who have been betrayed - directly or indirectly - or any of us who are on the "outside" should/do not really care whether this was complicity or whether many were (as I think) hoodwinked.
Accountability must be demanded from the entire system (academic - research - clinical): we don't care how you do it, you just need to not have these results. If you are a clinician and you get fooled, then I don't feel you to be gaining personally, but I would like to see the scrutiny that people like you are demanding, or else, how do we know this sort of thing won't keep happening.
That trust is fragile, and that is why (a) I truly appreciate the outrage we can from you - CL Psych and the likes of Healy, Avorn, etc., and (b) am completely frustrated and disturbed by the general lack of such a response from the mainstrean medical community.
For a profession that demands - and is mostly granted - autonomy in its decision-making, I don't really care - to a large extent - how or why bad knowledge was propagated. Obviously those directly responsible should individually be held to account, but at the institutional or professional level, these are examples of a systemic failure.
I feel that many doctors reach Healy's conclusion, shrug their shoulders, and say, "damn, those bastards fooled us, but I didn't do anything personally wrong so whatayagonnado?" Why do we not hear more outrage? Where is the outrage about Vioxx for example?
Regardless, I know Healy means well, but it is quite an indictment of the entire system - not just the pharmaco. or specific academics and clinicians involved. I feel that organized medicine (and doctors generally), love to point the finger at the insurance companies, the pharmaceutical companies, hospitals, anyone besides themselves.
Unfortunately, this causes them to ignore the obvious fact that the practicing doctors on the frontline and the honest and honorable academics researchers have the most power and could be the most effective at remedying these problems. And they ignore this at their own peril because if they don't demand accountability then ultimately soneone has to.