Assay Sensitivity or Efficacy?

I was sure glad to see David Healy mentioning the concept of “assay sensitivity” in his latest article. I discussed his article earlier regarding issues of safety. Now it's time to move on to efficacy. A few years ago, I remember Donald Klein mentioning that some trials that failed to show superiority of medication over placebo were not failed trials; they simply lacked “assay sensitivity.” So apparently the trials that showed superiority of medication were sufficiently sensitive and trials which did not show superiority lacked sensitivity. To me, this sounded a lot more like a way to justify the generally meager to moderate (at best) effects of antidepressants over placebo rather than a realistic scientific appraisal of the evidence.

Here’s what Healy says: “… many, including the regulators who approve the drugs on the basis of such trials, regard antidepressant trials as assay systems aimed at demonstrating a treatment signal from which a presumption of efficacy can be drawn, rather [than] as efficacy trials… If these trials are simply assay systems, it can be reasonable to discount and leave unpublished evidence from failed trials in which an active treatment fails to distinguish from placebo, on the basis that the trial lacked assay sensitivity... Alternatively, if we regard antidepressant trials as efficacy trials then both those demonstrating a treatment effect and those not demonstrating a treatment effect should be thrown into the meta-analytic hopper, and if this is done the degree of superiority of active treatment may be little more than 5%, or a mean of 2 points on the Hamilton Rating Scale for Depression scores, or no greater than it was shown to be in paediatric antidepressant trials (Khan et al., 2000; Kirsch et al, 2002).”

Counting trials as “assays” is ridiculous. Patently absurd, in fact. To do so is to endorse the idea that only one’s successes count. This is akin to boxer A fighting boxer B 10 times. In 7 fights, boxer A and boxer B fight to a draw – the judges cannot reach a decision. In two fights, boxer A wins by knockout and in a third, boxer A wins via a split decision of the judges. We would logically conclude that boxer A is a bit better than boxer B, but is clearly not superior by a large margin. However, under the “assay sensitivity rule,” we’d only count the times when boxer A won, and conclude that he is a far superior fighter than boxer B, which is clearly a mockery of the evidence based on their ten bouts.

In fact, a meta-analysis reached the same conclusion – that including only studies with “assay sensitivity” results in biased conclusions. In fact, the authors state, “Unless evidence is gathered to support the hypothesis that using the AS method reduces bias [of which there is none currently], meta-analysts should make quality judgments that are based on study methods, and that are independent of outcome.”

This essentially means that existing meta-analyses of antidepressant efficacy are biased, because, with the exception of the few meta-analyses that included unpublished studies, meta-analyses have relied on only published studies (which, almost by definition, yielded at least some significant advantage for the drug).

More from Healy:
“From the RCT data cited above, it appears that when people improve during antidepressant trials, 80-90% of the response can be attributed to the natural history of the disorder, or to the effect of seeking help, or to the benefit of any lifestyle advice or problem-solving offered by the clinician, or to what has been called countertransference or related aspects of the therapeutic encounter.”

I believe what Dr. Healy is referring to is what we in psychology call the “common factors,” including the therapeutic relationship. I recall that 80-82% of the antidepressant effect was accounted for by placebo in Kirsch’s work, and that when one looks at active placebos (placebos with side effects similar to antidepressants, which then keeps participants blind to their treatment condition more consistently), the number gets closer to 90% or above. Moncrieff, Wessely, & Hardy conducted a meta-analysis on the active placebo topic that bears this out.

So if the active effect of the drug (i.e., the effect we can attribute to the drug over placebo and other factors) is very small, what should we do?

Healy opines as follows:

"But what should happen if the combined non-drug components contribute four times more of the eventual response to treatment in standard cases than does the active drug? If the money and culture are to follow the evidence in this scenario, where should they go? One possibility is to modify the APA statement to say that psychiatrists rather than antidepressants can save lives. For example, we might expect lives to be saved in the case of clinical practice, informed
by the evidence, that restricts antidepressant use to cases in which it is clear that the condition has not resolved of its own accord, efforts at problem-solving have not led to a resolution, and hazards such as suicide arising from the severity of the condition have shifted the risk–benefit ratio in favour of a closely monitored drug intervention with informed patients, rather than non-intervention. Aside from the scientific and clinical merits of this position, there is a political case for reading the data this way, in that if there is no evidence that antidepressants pose risks [which is of course untrue but is often stated by "opinion leaders"] and if antidepressants rather than physicians save lives [likewise untrue but believed by many], then in a brave new world in which healthcare is being segmented, it is not difficult to foresee a future in which depression screening and treatment might be undertaken by non-medical personnel."

If I'm understanding him correctly, then I don't think he could be more correct. The psychiatrist him/herself accounts for a significant part of treatment outcome. Poor interpersonal skills? Can't form solid relationships with your patients? My bet is that your outcomes are poor, regardless of the pharmacological regimens you employ. In both psychotherapy and pharmacotherapy, the therapist/physician influences outcome regardless of the treatment provided. Instead of focusing on the type of antidepressant, which we know does not make much of a difference in influencing outcomes, we should be figuring out what therapist behaviors and/or personality traits are related to good outcomes -- there's more action there than sifting through a bunch of antidepressants which yield little benefit over a placebo in any case.

But figuring out what types of therapists are effective does not reward shareholders and is thus research that will take decades to conduct (who's going to fund it?), while clinical trials of medication will plug along in a direction where we can be assured that the sponsor's desired outcomes will be found (except when the pesky government gets involved in research, such as with CUtLASS 1) while patients in the real world benefit little to not at all from these trial results. Creating yet more drugs which serve to benefit shareholders, corporate executives, and allied academic researchers yet fail to yield any benefit to patients over existing regimens -- The cycle continues.