When randomized trials don’t help

The Herald reports on a study of weight gain after quitting smoking, which is based on analyzing the results of 62 randomized trials of treatments to help quitting.  Ordinarily, data from randomized trials is what we want, so why is Professor Simon Chapman quoted as complaining the results are unreliable?

Well, it’s partly because he doesn’t like anything that might be construed as favorable about smoking, but he has a good point in this case.  Randomized trials give us fair and trustworthy comparisons between two treatments: in this case the 62 trials tell us something about which ways to quit actually lead to the most quitting.   The information on weight gain, on the other hand, isn’t a comparison of two randomized treatments, it’s a before-after comparison on the people who managed to quit.  The fact that the treatments were randomized is of no help at all, since the analysis lumps all quitters together.

In fact, it’s worse than that. A lot of smokers manage to quit with only moderate difficulty.  They tend not to end up in randomized trials.  Some smokers find quitting much harder, and they are much more likely to end up in randomized trials.  So the research actually has found that a group of people who probably found quitting hard have gained 4-5kg after quitting.   It’s quite likely that people who find quitting hard are also going to gain more weight than those who quit without major difficulties, so we may well be overestimating the impact of quitting on weight.