Tuesday, October 02, 2007

Peer Review Is Mediocre at Best

Regarding bias in "science" and the utter balderdash that passes for peer-reviewed science, I sometimes feel like a lone voice in the wilderness. Well, thank God -- another blogger has thrown down the gauntlet on the topic. The Last Psychiatrist has a great post on the topic in which he notes a few huge problems with medical journals, of which I'll highlight a few in upcoming posts. Let's start with peer review...
"Most people think peer review is some infallible system for evaluating knowledge. It's not. Here's what peer review does not do: it does not try to verify the accuracy of the content. They do not have access to the raw data. They don't re-run the statistical calculations to see if they're correct. They don't look up references to see if they are appropriate/accurate."
Couldn't agree more with the Last Psychiatrist. We just assume the raw data are accurate. Every study likely contains some small data entry or calculation errors, but what if the whole paper is based on a significant misrepresentation of the raw data? Wouldn't that be a, large problem? What is reported and what is not reported? To put it in layman's terms, anybody can make up whatever the hell they want, and the peer reviewers are under the assumption that it is true. We're working on the honor system here and who knows how often the final paper reflects the real data, or if we are dealing with undisclosed errors due to sloppiness, an accident, greed, or just wanting to cover up the bad news, like in the following...

There is no way that even the world's greatest peer reviewer would catch this, as without access to raw data, we're trusting that relevant information is presented in the manuscript. Reviewers might catch an obvious statistical error, but they sometimes miss the most blatant errors, such as a paper that makes an important conclusion based on no evidence whatsoever.

What do peer reviewers do?

Again, quoting from The Last Psychiatrist...
They look for study "importance" and "relevance." You know what that means? Nothing. It means four people think the article is important. Imagine your four doctor friends were exclusively responsible for deciding what you read in journals. Better example: imagine the four members of the current Administration "peer reviewed" news stories for the NY Times.
And what determines if they think an article is important? Their bias, hopefully mixed with some basic understanding of research methodology and statistics. In maintream psychiatry, you are raised to believe that meds are safe and effective and that newer meds are better than older ones. Is this based on evidence? Sometimes, but quite often it is based on marketing that would make a user car dealer proud. Peer reviewers are often somewhat of an echo chamber than reflects whatever hooey marketing is going on lately. For example, consider that many poorly done studies on second generation antipsychotics were featured prominently in journals -- the mainstream psychiatry culture was overflowing with excitement about heavily promoted new antipsychotic meds, to the point where peer reviewers were willing to gloss over the flawed studies (for example, often using a control group receiving an unreasonable dose of an older antipsychotic) -- and now we have these drugs selling at a clip of well over $10 billion annually in the U.S.

No, I'm not claiming I don't have my own bias. Duh. You can see the cards I'm holding pretty clearly if you read this site with much regularity. The point is that peer reviewers need to realize their bias and take a better, more objective look at research they are reviewing. Too many industry-cheerleading pieces in journals leads to uncritical acceptance of treatments that nearly always fail to live up to their initial hype. After all, after a few trials have been published (even if poorly done and/or overstating efficacy, understating risks), the drugs are now based on "science," which leads to yet more marketing. Check the actual track record of benzos, SSRIs, Depakote, and atypical antipsychotics if you doubt me. Each treatment "revolution" is closely linked to peer review. So if you are pleased as punch with the current state of affairs in mental health treatments, then please make sure to send letters to your favorite medical journal editors thanking them for the present system. Don't let it change.

Or maybe the whole system needs a fundamental overhaul. More on that later.

Promo Time. I'll take yet another lead from the LP and humbly suggest that you promote this post via Digg, Reddit, or any other favorite service. I'd even more strongly suggest you hit up the LP's post and promote it. While you're sharing posts with the world, you should you read my take on SSRI's, Suicide and Dunce Journalism and send it to all of your friends. And while I'm in promotion mode, give your money to Philip Dawdy if you like good journalism on mental health issues. If you want to pledge money to support the operations of this unpaid anonymous blogger (and you know you do!), thanks, but take it and give it to Philip. Now!

6 comments:

Herb said...

Dear Doc,

I’ve already sent my comments to “The Last Psychiatrist” regarding his recent posting so I might as well also share my thoughts with you.


“No, I'm not claiming I don't have my own bias. Duh. You can see the cards I'm holding pretty clearly if you read this site with much regularity.” --- CL PSYCH


Wow, do I enjoy the irreverence especially coming from both of you professionals. Sadly though I’m somewhat disturbed from the standpoint of a support person and care giver. Is there nothing sacred upon which I can hang my hat to try and make relatively informed medical decisions for my charge? As I stated I do enjoy the irreverence as it helps to open my eyes and challenges my thoughts and biases as well as fuels my ever increasing skepticism especially as it relates to the field of Psychiatry, Pharmaceuticals and the infant science of the brain.

With all that I’ve come away with from your site and several others I still feel despite our individual biases that it still is the responsibility of you professionals to somehow self-regulate and correct these problems and improprieties before we end up with more wasteful interventions.

You (medical) professionals fouled up and were responsible for the poorer quality health care we now endure as we are burdened with these crap HMOs and/or managed care system; better known as managed profits for management and less care for patients and grumbling professionals.

While I do acknowledge, appreciate and enjoy your writings; how about doing something about that which you write?

Warmly,
Herb
VNSdepression.com

Romeo Vitelli said...

The real mechanism that is meant to catch bad research isn't peer review, it's replication. Peer review is just meant to ensure that research meets minimum standards of quality (in theory). The idea is that replication of research findings will weed out the fraudulent and substandard researchers. Unfortunately, most research studies don't get replicated. There's no money or glory in doing what some other scientist has already done.

CL Psych said...

Herb,

"With all that I’ve come away with from your site and several others I still feel despite our individual biases that it still is the responsibility of you professionals to somehow self-regulate and correct these problems and improprieties before we end up with more wasteful interventions."

I've tried contacting people in reasonably high places regarding some of the information posted on my site. While I know that people with whom I have shared information have then examined my site, these authority figures have then never responded to me directly. When people such as David Healy and Aubrey Blumsohn have attempted to blow the whistle, they have lost their jobs. It is difficult to regulate the profession from the inside when those who speak out are forced from their positions and otherwise intimidated. I will continue to occasionally pass along info from my site to persons in power and see if anything comes of it.

CL Psych said...

Romeo,

Excellent thought regarding replication. I agree to a point. Consider the rather vast literature, including clinical trials and review articles, regarding atypical antipsychotics. They were hailed across the board as a huge advance. Many of the clinical trials used similar designs; thus, replication occurred. The design of these clinical trials was often shabby:

1. high doses of Haldol used as a comparator
2. other conventional antipsychotics with a better side effect profile than Haldol were not included
3. concomitant benztropine was sometimes not allowed
4. trials that showed improvement when switching from a conventional antipsychotic to a newer antipsychotic did not have comparison groups that switched from an atypical antipsychotic to a conventional antipsychotic, and this limitation was strangely missing from the discussion in some such studies.

I agree that replication will root out some cases of shabby science, but in many cases replication occurs and peer reviewers miss important points again and again. See what I mean?

Herb said...

Doc,

“I've tried contacting people in reasonably high places regarding some of the information posted on my site. While I know that people with whom I have shared information have then examined my site, these authority figures have then never responded to me directly.” --- CL PSYCH

I am appreciative of the fact that I’ve learned that you are one of many trying to make an effort to change and improve the situation.

We may be butting our heads against a brick wall from our respective vantage points but brick walls have been known to eventually tumble when chipped away piece by piece. While some unfortunately having to pay the price of financial security and careers and others like patients being challenged by the status quo and the system and my spouse and me being subjected to some professionals with tunnel vision who simply tow the line and accept the doctrine lacking in thought or investigation or truly concerned with patient wellness that we must continue in our persistence and own ways of achieving ideals simply because that is the right thing to do.

Once again, your efforts and this forum are much appreciated by me. Please keep up the good work and the professional irreverence. It is extremely enlightening for me as a layperson and care giver.

Warmly,
Herb
VNSdepression.com

Anonymous said...

CP: Organized psychiatry has a code akin to omerta. You may as well charge windmills.