Efficacy: Here’s what the authors said…
Apripiprazole 10 mg/day was efficacious, and safe for psychosis associated with AD, significantly improving psychotic symptoms, agitation, and clinical global impression.
Quickie stats lesson. In determining if there is a “statistically significant difference” between two groups, a large factor is the size of the sample. Just because a difference is “statistically significant” does not imply that the difference actually means anything of value. With a large sample size (as was seen in the present study), very small effects can become “statistically significant.”
Next, let's discuss the size of these treatment effects -- how helpful was Abilify? Mean change on the Neuopsychiatric Inventory – Nursing Home Version was 17.6 points for the 10 mg/day Abilify group and 13.0 points for the placebo group. With the knowledge of the mean change and the standard deviations from both groups, one can easily calculate the effect size, which indicates the magnitude of the treatment effect (i.e., did people get a little better or a lot better?). The effect size (using Cohen’s d) was .14. The general guideline is that d = .20 is a small effect, so less than small – I guess you could call it miniscule.
How about other measures used in the study? Here are a few...
- Neuropsychiatric Inventory – Nursing Home Version Psychosis subscale: Effect size = .19
- Clinical Global Impressions – Severity: Effect size = .12
- Clinical Global Impressions – Improvement: Effect size = .10
- Brief Psychiatric Rating Scale: Effect size = .15
- Cohen-Mansfield Agitation Inventory: Effect size = .17
No matter how you slice it, the effects were all very small. Let’s also do a common sense test. The Clinical Global Impressions – Improvement Scale asks one question. The wording may vary slightly across studies, but one example is:
Rate total improvement whether or not in your judgment it is due entirely to drug treatment. Compared to his/her condition at admission to the project, how much has he/she changed?
In the current Abilify study, the average advantage for people taking Abilify 10 mg/day versus placebo was 0.3 points. There are seven points on the rating scale. For example, at one point on the rating scale is minimally improved and the next point is moderately improved. So the average patient on Abilify was 0.3 points closer to moderately improved compared to minimally improved relative to someone on a placebo. Can you say ooooh, big deal?
Anyone who has taken a basic course in statistics should be either laughing or crying (your choice) at this point. What makes this particularly egregious is that the first two authors are academic researchers who absolutely must have received this lesson dozens of times during their research training, so there is no pleading ignorance on their part.
Statistical Significance: Okay, maybe this is a smaller point. But when I plug their numbers (means, SD, N) into a t-test, I find that some of the differences they labeled as statistically significant are no longer statistically significant. Yes, they used a different statistic, which was apparently useful to them, but I thought I’d at least mention that if someone chose to use a different analysis, the data may have turned out as not statistically significant. In any case, the benefits of Abilify as shown in this study are minimal at best.
Safety: Okay, so Abilify does not work very well, but at least it won’t kill you, right? 18 people died across the course of the study, including 3% of the placebo group and 7% of the Abilify 10 mg/day group. Cerebrovascular adverse events were reported for four patients in the 10/mg Abilify group and zero people in the placebo group. As has become nearly an essential disclaimer in studies these days, the authors write: “Eighteen deaths occurred during the study, none of which were considered to be related to study medication. [my emphasis]” Of course not. The study is funded by a Bristol-Myers Squibb, and the company and its “independent” academic investigators determine if the drug could have caused any deaths. The fox is policing the henhouse.
Discussion: The best part of any article is the discussion, because it is a marketing exhibit.
The results of this placebo-controlled fixed-dose study indicate that aripiprazole impacts beneficially on psychotic symptoms in institutionalized patients suffering from psychosis associated with AD; this is the first such study to report improvement in both primary and secondary outcome measures in this patient population...
Aripiprazole also produced significant improvement in other aspects of psychological and behavioral symptoms, as evident from changes in secondary outcome scores...
Thus, the results indicate that the use of aripiprazole cannot only alleviate the specific symptoms of psychosis, but can also reduce the overall psychological and behavioral burden of AD...
Interestingly, the 5 and 10 mg/day doses of aripiprazole show robust efficacy to improve the CMAI scores from baseline to endpoint, reinforcing the evidence that atypical antipsychotics are efficacious in the treatment of agitation. [Apparently “robust efficacy” means having a miniscule impact relative to placebo.]
Again, see above commentary about the extremely small extent to which Abilify “impacts beneficially” upon participants in this study.
To the authors’ credit, they mention that
Elderly patients with dementia-related psychosis treated with atypical antipsychotic drugs are at an increased risk of death compared to placebo.
Right, kind of like what happened in this study, at least for the patients who took 10 mg of Abilify. I realize it may have been a chance occurrence that the Abilify 10 mg patients died at a higher rate, but when you combine minimal treatment benefit with doubling the risk of death, why the hell would anyone choose to prescribe Abilify based on results from this study?
Believe it or not, there are a couple of other points about the study that I could prattle on about, but I think the point has been made.
Journal Policies: If you wander over to the American Journal of Geriatric Psychiatry’s Instructions for Authors, you can see the following:
Provide measures of effect size liberally. Give cautions when statistical significance has doubtful clinical or practical significance.
What I have done in this post is provide the measures of effect size since the authors did not provide a single measure of effect size. Not a single one. But wait, if this paper clearly violates one of the main principles listed under “Statistic Guidelines” for the Instructions to Authors, how did it get published? Good question. And that takes me right back to my earlier posts on peer review and how the “experts” who review papers sometimes do so in a slipshod, lazy, biased, and/or incompetent fashion (1, 2, 3, 4). And the editor? Did he even skim this manuscript to see if it conformed to the guidelines of the journal?
So a paper that flouts the journal’s own policies is published, a paper in which statements regarding a product’s efficacy are far overblown. Perhaps drug reps will be disseminating this wonderful piece of science nationwide or even internationally to help get patients onto Abilify because now their marketing is based on "science." Will physicians see through this sort of ruse and laugh the reps out of the office, or will they briefly scan the abstract, conclude the drug is effective, and then whip out their prescription pads? You tell me.
Last, but not least, I present a Golden Goblet Award to the academic authors of the study as well as to the coauthors at Bristol-Myers Squibb. Your ability to present teeny treatment effects as "robust" and get away with it is notable. Great work. You deserve a good cut of however many $$$ are added to BMS's coffers from unneeded prescriptions of Abilify for dementia.
Excellent comment. The whole point of the article is to provide pseudo-scientific support for the indiscriminate use of Abilify on the nursing home population with the AD. Apparently, Abilify hasn't been selling well. There's been a lot of TV and print ads about Ability recently. It turns out that BMS has been concurrently targeting targeting the nursing homes. The Geriatrics Journal is simply a shill for BMS, and the "Pimp" award was most fitting.
ReplyDeleteGood work, CP. This article is a fine example of what Bernard Carroll has called an experimercial.
ReplyDeleteI think here we also have an excellent illustration of the difference between statistical significance and clinical significance. Oy.
ReplyDeleteIn defense of the authors, it was probably ghost-written anyway.
ReplyDeleteGreat comments across the board. I don't think I saw any acknowledgment of "editorial assistance," but a good ghostwriting job would include hiding any traces of outside involvement.
ReplyDeletePlease feel free to use this very study as an example of the difference between clinical and statistical significance.
And it is indeed a poster study for the "experimercial," a term that deserves to be used more often.
I read this because we are caregivers and Abilify was just prescribed to my family member with dementia, not AD, who has periods of rage and great agitation. As consumer and a caregiver I feel duped by the doctor. Thank you for this information.
ReplyDeleteok, talk about bias. what exactly did the pharma. companies and physician community do to this idiot? one thing you didn't mention, oh jaded one....is that psychiatric studies often have a robust placebo effect, because the placebo group still have interventions associated with the measurement of results. sooooo...the statistical significance in a study, though it may seem to the "expert" (or not) statistician to not be particularly relevent, ARE actually more clinically relevant than the average, jilted, pointy-head may otherwise expect.
ReplyDeleteAnon,
ReplyDeleteRight, there is a placebo effect. And the drug was every-so-slightly better than placebo. If you're saying that Abilify was actually highly effective for dementia based on this study, then you must be practicing a whole new breed of statistics. But what do I know -- I'm just a "jilted, pointy-head."
This is obviously a statistician with no first hand experience with this medication. My son has dementia due to a brain injury and has been in the deepest depths of depression for 20 years. Four serious suicide attempts and no medication would bring him up to a acceptable level depression. Until Abilify came along. It has been a godsend. And this idiot "author" has some sort of vandetta against the product. This isn't a statistics game here. Lives depend on drugs like this. Butt out, jerk!
ReplyDeletePat: I'm glad your son had a good response. But keep in mind that we have to evaluate products using some other method than personal stories. Look around on the internet and you'll also find plenty of stories of people who have not enjoyed their time on Abilify. Are we supposed to believe the positive stories about a drug or the negative ones?
ReplyDeleteThat's why we conduct research. If that makes me an idiot or a jerk, then so be it. If you want to only rely on people's personal stories, then go for it.
And in the article I critiqued in this post, more people died on Abilify 10 mg than on placebo. Do we simply not worry about the people that died because they don't match your son's story?