Positive Antidepressant Study “Misleading” and “Erroneous”

New research by the Nordic Cochrane Center has challenged last year’s positive study about antidepressant efficacy. The authors write:

“The evidence does not support definitive conclusions regarding the benefits of antidepressants for depression in adults. It is unclear whether antidepressants are more efficacious than placebo.”

Many analyses of the research on antidepressant drugs have found that they are no better than placebo for the treatment of depression. These results have been especially reliable for mild to moderate depression, and especially when unpublished trials of the drug are included. At best, the drugs show an incremental improvement over placebo, which is not clinically detectable,

However, a study in The Lancet last year, led by Andrea Cipriani and conducted by authors with strong ties to numerous pharmaceutical companies, suggested that previous research was mistaken and that the drugs were effective. Nonetheless, researchers have suggested that the Cipriani study was flawed in multiple ways.

Now, a new study using the rigorous procedures of the Cochrane Collaboration has challenged that controversial positive conclusion put forth by Cipriani and his colleagues.

The research was conducted by three members of the Nordic Cochrane Center in Denmark, led by Dr. Klaus Munkholm. It was published open-access in the journal of the British Medical Association, BMJ Open.

Munkholm and the other researchers found several errors and misrepresented data in the Cipriani analysis. They write that “Several methodological limitations in the evidence base of antidepressants were either unrecognized or underestimated in the systematic review.”

For instance, certain types of antidepressant research were far more likely to find a positive result:

1. Research using the controversial “placebo-run in” design. This type of study eliminates people who respond to the placebo effect–which removes it as a fair comparison. The study essentially compares people on the drug to a smaller group of people who didn’t respond to the placebo effect.
2. Published versus unpublished clinical trials. Antidepressants were far more likely to have failed in the unpublished trials, and even when they succeeded, they were likely to have a much smaller effect size.
3. Studies with a high risk of bias. Studies that were graded as having unclear methods, risk of unblinding, and other limitations were more likely to find a positive antidepressant effect.

Cipriani and his colleagues claimed to have followed the Cochrane handbook’s guidance on how to report the risk of bias in clinical trials. However, the researchers from the Nordic Cochrane Center found that Cipriani had misrepresented this data in several ways.

Munkholm and the other authors write, “The outcome data reported by Cipriani et al. differed from the clinical study reports in 12 (63%) of 19 trials.”

In some situations, the Nordic Cochrane authors could not determine whether Cipriani and colleagues had followed the Cochrane rules because Cipriani and his co-authors failed to describe their methods adequately. In other situations, however, Cipriani and co-authors defied those rules:

Cipriani and colleagues didn’t analyze study drop-outs, even though that is one of the regulations in the Cochrane handbook. Researchers must analyze why participants drop out in the placebo versus the active antidepressant groups. If it is due to adverse events, suicidal ideation, or ineffectiveness of the drug, that changes the risk/benefit ratio. But according to Munkholm and co-authors:

“The authors [Cipriani et al.] did not consider the reasons for dropout, although this is also recommended by the Cochrane Handbook.”

Additionally, they write that “Cipriani et al. did not use the standard Cochrane categorization of low, unclear or high risk of bias due to a lack of blinding […] While this implied the presence of a blinding issue, their categorization did not affect the overall risk of bias assessment, and it seemed that the stated-not tested domains were counted as ‘low risk of bias.’”

This means that Cipriani and colleagues:

1. Didn’t use the Cochrane categorization, despite claiming to in their methods.
2. Didn’t count the “not-tested” or “unclear” domains as being at risk of bias, even though unclear methods are a way of biasing results.

Further, many of the studies were at risk of bias due to selective outcome reporting. The researchers chose only to report favorable outcomes, despite the presence of primary findings that showed no effect of the antidepressant. This is one of the most egregious ways to bias research studies, and again, Cipriani and his co-authors failed to correctly follow the Cochrane Handbook in their treatment of the issue:

“A trial was only rated at high risk of bias in case both outcomes were missing. This is not in accordance with the Cochrane Handbook.”

Even discounting the bias issue, the evidence was less strong than the Cipriani analysis suggested. The average improvement over placebo (on the 52-point Hamilton Depression Rating Scale) was less than two points–a clinically insignificant difference. At least three points, and perhaps as much as seven points, is the minimum clinically noticeable change on the Hamilton scale.

All of this evidence confirms previous studies. Although Cipriani and his co-authors obfuscated this data in their summary, writing that the benefits of antidepressants outweigh the risks, the evidence itself tells a different story. The raw data in the Cipriani analysis suggests that antidepressants do not outperform placebo by a significant degree.

According to Munkholm and the other researchers, “It seems misleading to rank the antidepressants when we have very low confidence in the evidence.”

“In the light of our findings, the [Cipriani et al.] review should not inform clinical practice. Second, our reanalysis has highlighted the need for a radical change in the way antidepressant trials are being conducted, reported, and interpreted.”

 

****

Munkholm, K., Paludan-Miller, A. S., & Boesen K. (2019). Considering the methodological limitations in the evidence base of antidepressants for depression: A reanalysis of a network meta-analysis. BMJ Open, 9(e024886). http://dx.doi.org/10.1136/bmjopen-2018-024886 (Link)