The previous science lesson explained why clinical trials are not useful for assessing the effectiveness of vaping for quitting smoking. While clinical trials (medical experiments) are often a useful study method, they do not replicate the real-world experience of switching to vaping. Clinical trials can measure the effect of assigning vaping to random smokers. But we really want to know how helpful vaping is for smokers who choose to try it. How do so many people confuse these obviously very different questions?
Here’s the problem: Those who conduct such studies do not realize they are implicitly assuming everything is as simple as medical treatment. Anything more complicated than that is ignored. This mistake also causes other errors.
Clinical trials are good for determining which medical treatment is better. If Drug X and Drug Y are each pretty good treatments for a disease, it is useful to know which one works better on average. Usually patients should not take X and Y at the same time, so one must be chosen. Thus, figuring out the average effectiveness of each drug, when given to everyone with the disease, lets us identify which to assign to everyone. It may be that X would work better for some patients and Y for others, but we typically do not know in advance who is who, so need to go with the averages.
For this simple scenario, clinical trials show which works best on average, and they replicate the real-world situation (clinicians assigning a particular drug to patients with the disease). Unfortunately, this scenario creates a simplistic mindset that all health interventions are just easy to analyze. This results in nonsensical claims when the greater complexities of social science are ignored.
One problem is insisting that randomized trials are necessary before assessing the effectiveness of a particular intervention, like recommending vaping. Sometimes experiments are informative though certainly not necessary (if Drug X works ten times as often as Drug Y, casual observation will make the superiority of X clear). In other cases – such as with switching to vaping – trials are not even very informative.
Other problems are more subtle: It is wrong to claim that a study shows that vaping (or unaided quitting, or some other method) is the “most effective” way to quit smoking.
Some such claims are based on counting. Most studies find that the majority of smoking cessation is “unaided,” done without any substitute product, drugs, or formal intervention. But this tells us nothing about what method is most helpful. People who choose to quit smoking unaided believe that they are ready to just stop. They are often wrong, of course, but if someone believes that, it is reasonably likely to be true. If she does not believe it, she is almost certainly correct. Thus if someone is not inclined to try to quit unaided, but instead tried to switch to vaping or another substitute, it would be insane to tell her, “No, stop! Most people quit unaided, so just try that.”
More often, claims of “most effective” are based on the rate of success among people who tried a particular method. This is the medical treatment model of smoking cessation, and is equally wrong. Consider a study that compares the rates of success, in a particular population at a particular time.
The first thing to note is that last phrase is critically important. For medical treatments, human biology does not vary all that much, so if a clinical trial shows Drug X works better, there is a reasonable chance this is true for all humans. But for consumer choice – and everything else in social science – there are many differences across populations. Prayer might aid smoking cessation in some populations, but it is easy to predict it would have no effect in others. Trying to switch to vaping or another product is not going to work so well in a population where supply of the products is likely to be interrupted.
What if the hypothetical study found that those who tried to switch to vaping did not quit smoking as often than those trying unaided quitting? A naive interpretation would be that vaping does not help, or even hinders, smoking cessation. This is, of course, exactly the interpretation anti-vaping activists have offered for some such results. But for most populations for most of the time vaping has existed, switching has not been a common approach. Smokers who tried vaping had usually already considered – and tried – quitting unaided and with various government-recommended methods. They had a much harder time quitting than average, and so still did when they tried vaping.
But now consider a population where vaping has become popular and many smokers have learned it is a good way to quit. The hypothetical study might show that trying vaping is more often successful than any other aid. Does this mean it is the most effective cessation aid, as claimed in some interpretations of a recent study? No. It could just be that a lot of those would-be unaided quitters tried vaping because it was popular, and they would have quit with or without vaping.
Additionally, in any population, those who try to quit with vaping are self-selected. They have probably tried vaping and did not hate it, and they are happy using a substitute rather than being completely abstinent. That is, of course, great for them that they realize they have that option and like it. Unlike with the drug trial, people have some idea whether X or Y will work better for them; the fact that Y works better for those who try it does not mean we can say it is “more effective” in general.
Eliminating complications like these are exactly what clinical trials are good for. But in this case we cannot – and should not – eliminate them. Ironically, the error comes from interpreting the data as if it were from a useless clinical trial.
It is a bit subtle and tricky, but it is often useful to think about whether a particular interpretation of a study result is based on inaccurately pretending it was similar to a clinical trial, with exposures randomly assigned rather than being individual choices. If so, chances are that confounding and self-selection are being ignored and the interpretation is wrong.