How Do We Know When a Treatment Works? A Primer on the Scientific Method

The layperson often has some difficulty understanding the nuance of what “science” is. Science, in its broadest sense, is not the objective truth—rather, it is a method for determining the truth. There are many ways of finding truth, including personal experience and anecdotal reports, but the scientific method is, at its best, the most rigorous method, one which others can replicate in order to confirm the legitimacy of findings.

In this discussion I am limiting the topic to how the application of science can determine the truth about the efficacy and safety of health treatments. Scientific procedures to determine treatment efficacy are in sharp contrast to what might be called clinical opinion or everyday clinical evidence. 

It is not that clinical opinion is necessarily wrong, but if it is not backed up by the scientific method, then there is no way of verifying that it is correct. Indeed, one of the major issues with clinical opinion is that it does not provide a “control group”—a group of people who are not given the drug or who are given a placebo.

If a doctor gives a drug and then notices that the patient has improved, the doctor’s clinical opinion is likely to be that the drug works. However, the patient might also have improved if he did not take the drug, or if he simply believed he was taking an effective drug (the placebo effect). Clinical opinion has no way to “control” for this potential factor—but the scientific method does. It’s called the “randomized, placebo-controlled trial,” or RCT.

The best way to control for researcher bias is with the double-blind RCT. This means that neither the trial subjects nor the researchers know who is getting the drug and who is getting the placebo. Without the double-blind, patients who know they are getting the placebo might not experience the full effect, and researchers might subconsciously rate subjective criteria as more improved if they know the patient is on the real drug.

Unfortunately, the blind is often compromised because the researchers—and often, the subjects—are aware of the drug’s side effects. Thus, the people who experience side effects know they are on the real drug. This problem can be avoided by using active placebos, a drug with side effects but which does not treat the health problem in question. But this is rarely done. For instance, I don’t believe an active placebo was ever used in a clinical trial for an FDA-approved depression drug.

Though there are many issues with the US FDA’s drug-approval process—it does not always work as intended—ideally, it should synthesize the results of RCTs so that only drugs that prove to be better than placebo (ie, better than nothing) get approval.

Off-Label Prescribing

When your physician prescribes you a drug off-label—one not FDA-approved for the condition being treated—you and your physician are both in the dark about the drug’s true efficacy and safety. This is not a minor problem, because it is estimated that 20% of the drugs on the market are prescribed off-label, meaning the FDA did not approve this drug for this specific health problem.

At the very least, the patient should be informed that the drug is being prescribe off-label (so that they know there is not enough evidence that the drug is better than nothing), but I doubt physicians rarely inform the patient of such.

I suspect physicians justify this by never wanting to cast any doubt on their drug recommendations, which might limit the drug’s efficacy! Physicians are very aware of the powerful placebo effect, and they know that if they let the patient know that the treatment may not work, the patient may not benefit as much.

Another concern is the risk/benefit ratio. All treatments have potential harms, and most have potential benefits. For example, Depakote is an anticonvulsive medication that has very severe side effects. When used for seizure prevention, the risk/benefit ratio may be appropriate, because in some cases, seizures are life-threatening. But Depakote is also offered off-label for pain relief, which may not be worth it because the harms of the medication have the potential to be more severe than the condition it’s treating!

A neurologist friend offered me a prescription for Depakote for my left leg pain, but I declined by saying, “I like my brain too much”!

Patient Satisfaction

The “snake oil” salesman used to report that 90% of their users were very satisfied with the results of treating dozens of health problems with their phony remedies. How can we account for the popularity of these treatments? There were two factors: the placebo effect and, often, the effect of drugs like alcohol or cocaine. The alcohol/cocaine in the treatment certainly made the patient feel better quickly, which contributed largely to the patient’s satisfaction with the treatment.

The charlatan’s drug gave the patient hope of improvement, which reduced the patient’s stress, which in turn perhaps strengthened the immune system to fight the health problem. Perhaps more importantly, many health problems are self-limiting—that is, the patient gets better with or without treatment (eg, the flu, upper respiratory infections, and, often, “depression”).

In short, patients are often satisfied with placebos because their symptoms improve naturally. But if a medical treatment only works due to the placebo effect, it exposes patients to the harms of that treatment unnecessarily.

What Reduction in Symptoms Is Scientifically Acceptable?

Physicians and patients would both prefer if the treatment resulted in “remission”—the complete absence of the health problem. However, such a goal is unobtainable for many health problems in the real world. In the real world, patients and their prescribing doctors both have a right to be pleased if the prescribed treatment substantially reduces the magnitude of distressing symptoms.

But what percentage of reduction is appropriate and clinically meaningful? As one trained in science and statistics I must depart both from the FDA and many physicians who generally have accepted a lower standard of efficacy than can be justified by a reasonable risk/benefit ratio.

The appropriate level of efficacy depends on many factors, including the severity of the health problem as well as the adverse effect profile of the prescribed treatment. With a life-threatening health condition, a lower standard of efficacy would be appropriate. Not so for a less-serious health problem, which should demand a substantially higher standard.

Take, for example, antipsychotic drug treatment of “schizophrenia” and psychosis. One large meta-analysis found that all antipsychotic drugs were superior to placebo, with moderate effect sizes in reducing symptoms, which may sound pretty good. However, 51% of the antipsychotic group experienced only a “minimal” response as opposed to 30% in the placebo group. Only 23% of the drug-treated group had a “good” response (compared with 14% of the placebo group).

But I believe this is the most important finding in the study: Since the 1960s, in these antipsychotic trials, the average improvement in those receiving the placebo has increased, while the efficacy has decreased for those receiving the antipsychotic medication.

Why this change? The investigators had some hypotheses to explain the decline of antipsychotic drug efficacy over time, including the notion that the longer people take the drugs, the worse the outcome. The researchers intended this to reassure us about the drugs’ efficacy, but this actually means that antipsychotic drugs have very negative effects after long-term use—in fact, they cause irreversible structural damage to the patient’s brain!

For a first-episode psychotic break, the antipsychotic drug’s brain-impairing effects does seem to make it difficult for the patient to focus on the psychotic ideation, and there is often very clear short-term improvement, which I observed in my early experience treating people with acute schizophrenia. However, studies have also found that psychotherapy, such as cognitive-behavioral case management, is just as effective with or without antipsychotic use.

I concur with the minority of psychiatrists who believe in using these drugs only when the patient is dangerous to self or others or too agitated to function in the family and even on a psychiatric unit.

In the above I have taken the position that many factors must be taken into account when considering the risks and benefits of a treatment, such as the severity of the health problem and certainly the seriousness of the treatment’s adverse effects. As a rule of thumb, I would argue as a general standard that the efficacy of the drug or non-drug therapy be at least one standard deviation or more over the average improvement of the placebo group.

I realize what I am recommending would reduce the pharmaceuticals on the market by at least 50% and probably more!

Biases in the Clinical Trials

If one reviews the RCTs for a treatment, it is very important to read more than just the abstract and conclusion, which are full of researcher “spin.” However, even the results section may not tell the full story. It is also important to look at the section on how the researchers chose their subjects for the clinical trial.

For example, clinical trials for drugs to treat depression almost always include a 10-day placebo wash-out period to exclude those from the study who get better on placebo alone! (If so many depressed patients remit their depression on placebo alone in these trials, why do psychiatrists continue to call depression a brain disease?) In the real world, then, the placebo effect is likely to be even more substantial than in the clinical trials—which is significant, since in 49% of the trials for antidepressants, the drug didn’t even beat the placebo.

One perfect example of how patient selection can substantially bias RCTs is Chantix, the popular smoking cessation drug. In the clinical trials for this drug, the researchers cherry-picked the subjects who are more likely to benefit from the drug than other smokers. In these trials, certain smokers are eliminated from the study—namely those with any psychiatric diagnosis, cardiovascular disease, obese patients, adolescents, pregnant women, and light smokers—so, almost everyone.

This means that the study results are only pertinent to a very small proportion of smokers. No wonder Chantix does so much better in the clinical trials than other smoking cessation drugs! Yet do doctors tell people (those with psychiatric diagnoses, for example, or light smokers, or people with heart problems) that this drug has never been tested for them, and that in fact people like them were deliberately excluded from the trial, likely because the drug wouldn’t work for them? Doubtful.

There are many other practices used to ensure that researchers get the results they want, and they all combine to help ensure that the drug is “superior” to the placebo effect.

But anyone trained in statistics and how to conduct sound scientific research, reviewing the FDA depression clinical trials, must come to the inescapable conclusion that these drugs should never have been approved by the FDA for treating clinical depression. Since the FDA did not insist on the use of sound science in the design of these clinical trials, the FDA must be responsible for the many negative consequences following its approval of these drugs.

Beside exposing a very large section of the population to the serious side effects of these drugs, and the additional cost of drugs to treat the adverse side effects, a major negative effect was the creation of a large industry called Treatment Resistant Depression (TRD), a mythical but major new “disease” when the patient fails to respond adequately to the primary drug.

In the newest TV ads for treating depression, we are told that two-thirds of depressed persons on these drugs continue to have depressive symptoms! The add-on drug these ads suggest to take care of this problem is one of several antipsychotic drugs. However, these add-on drugs also don’t seem to reliably get these depressed patents to a permanent remission. This depression roller coaster does not stop here with these add-on antipsychotic drugs (and their adverse side effects), because when these fail to do the job, there are about 8 to 10 FDA-cleared bio-mechanical treatments (mainly zapping the brain) to treat so called TRD!

Actually, psychiatry and the pharmaceutical industry may be better off financially when drugs fail to treat a health problem, or if they create adverse effects that also must be treated with further drugs, than when drugs effectively treat a health problem and treatment can end! To complete the TRD story, the latest pharmaceutical trend has been the introduction of street and hallucinogenic drugs into the pharmacy mainstream (ketamine, for example). If and when these fail, what’s next?

Despite the above, I certainly concur that there are many drugs and non-drug therapies that do meet both a reasonable efficacy and safety standard (if it were not for blood pressure medications, I would probably be dead by now). But in order to achieve better healthcare at lower cost, we must eliminate those drug and non-drug therapies that do not measure up to an appropriate scientific standard of proven efficacy and safety for the condition being treated.

I am not at all hopeful that the FDA and the medical profession will initiate, on their own, a higher efficacy and safety standard than the present one. If a higher efficacy and safety standard is ever to occur, it will have to be instituted by an entity that only allows payment for health services that meet that higher standard.

 

Dr. Ruthven’s website is http://www.ruthvenassessments.com/ and he can be reached by email at [email protected]. His recent book on healthcare can be found here.