Posts filed under Correlation vs Causation (68)

New vaccines only get approved after randomized controlled trials showing that they are beneficial, but if you want to estimate the benefit of expanding vaccination to a new group of people it’s hard to do a randomized trial.  Just comparing vaccinated and unvaccinated people doesn’t help, since there are many reasons why these groups are different, and the comparison gives completely unreasonable estimates. There’s a new study from Canada, reported by Reuters, that takes advantage of a ‘natural experiment’ to estimate the benefit of vaccinating children aged 2-4.

In the US, the guidelines on vaccination changed in 2006 to include kids in this age range, in Canada the guidelines didn’t change until last year.  This allowed the researchers to compare hospital emergency room visits in Montreal and Boston and estimate the impact of the change.  Just looking at US hospitals wouldn’t be enough, since there is a lot of year-to-year variation in the severity of the current flu strains, and just comparing the US to Canada wouldn’t work, since there are a whole lot of differences between the countries (starting with a national health insurance system).  But looking at how the US:Canada difference changed from before 2006 to after 2006 gives a reasonable estimate of the effect of vaccination.

Looking at over 100,000 emergency-room visits for flu-like illness, researchers found a 34% decrease in risk for the 2-4 year age group affected by the change in guidelines. This wasn’t just for kids who were actually vaccinated — it also includes the reduction in risk from having your playmates vaccinated.  There was a smaller reduction in risk, 10-20%, for older children — either because the additional reminders made them more likely to get vaccinated, or because they were less likely to catch flu from younger siblings.

Natural experiments get used a lot in economics. In medicine, we tend to prefer real experiments, but sometimes these are impractical or unethical, and natural experiments are the best we can do.

Last week’s Stat of the Week nomination was a story on the “J-curve” for disease risk and alcohol consumption.  Yet another research paper, this time from the Nurses’ Health Study, had found that people who drink small amounts of alcohol regularly are healthier than those who drink none and those who drink larger amounts.    This sort of result is unpopular with doctors, as the Herald story reported, and for good reasons, but that doesn’t mean it’s untrue.  On the other hand, the fact that it’s true doesn’t mean that it’s news.

The obvious difficulty in comparing drinkers to non-drinkers is that some of the strictest non-drinkers are actually ex-drinkers, people who you would expect to be in worse health.  Since epidemiologists are not completely stupid, they know about this problem and many studies have addressed it. Excluding ex-drinkers doesn’t make the effect go away, waiting for a long time between the drinking assessment and the health assessment (as in this paper) doesn’t make it go away, and splitting up light drinking into finer categories shows that there is lower risk for people who drink occasionally than for those who regularly drink a small amount (again, as in this paper).  For some of the claimed benefit there are even plausible mechanisms (eg, alcohol consumption does definitely raise HDL cholesterol levels in short-term experimental studies).   This is just observational research, so the results could be just as wrong as the apparent protective effect of beta-carotene in cancer, or of raising HDL cholesterol with niacin in heart disease, which fell apart when subjected to randomised trials, but it’s carefully-done observational research.

As doctors will tell you, the problem with announcing a health benefit of moderate alcohol consumption is that most people interpret “moderate” to mean “a bit more than I currently drink”.  As a scientific result, it’s fine; as a public-health intervention, it’s badly off-target.   The American Heart Association guidelines on alcohol and heart disease, for example, basically say that regular consumption of small amounts of alcohol probably is protective against heart disease, but that you shouldn’t go around advocating it.

The problem for medical researchers is that funding bodies and universities (and their own egos) want press coverage of research results, but that this sort of marketing of incremental medical research as if it was ground-breaking health advice is unhelpful to the public. It’s very rare that you should change your behavior based on the results of a single medical study, but that’s the model that a lot of medical reporting is based around.

In this week’s Stat of the Week the misinterpretation is not primarily the fault of the individual media outlet, since it was present in the original source.  Still, if a press release or a wire service story told you that the Wallabies had a new training regimen that would improve their game without making them fitter, faster, tougher in the scrum, more accurate with kicking, or better at putting in the elbow, you’d ask questions.  We’d like to see science journalism eventually get up to the standard of sports journalism.

The Herald reported “a new study suggests [TV’s] damaging effects may even rank alongside those from smoking and obesity”. If you look at the British Journal of Sports Medicine, that’s what the authors actually say. They go on to say “TV viewing time may have adverse health consequences that rival those of lack of physical activity, obesity and smoking; every single hour of TV viewed may shorten life by as much as 22 min”. The implication that TV has an effect separate from physical activity and obesity, just as it is separate from the effect of smoking, is reinforced when they say that the associations were adjusted for a whole bunch of cardiovascular risk factors: cholesterol, blood pressure, age, gender, weight, blood glucose, etc.   The implied claim is that TV kills in a way that isn’t explained by any of these risk factors: it’s not that TV-watching uses up fewer calories, or that you are more likely to snack while watching.  Perhaps the mechanism is that watching too much TV makes you believe all the health-related advertising and medical news? (more…)

As we all know, mean people suck. But do they earn more?

A US study presented at a management conference today looked at measurements of agreeableness, and found that people (or, at least, men) who rated themselves as less agreeable, cooperative, and flexible earned more money.  This isn’t precisely about `mean’ people, but headline writers around the world spontaneously went for the four-letter word (or just copied each other). (more…)

There’s a new genetic study (stuff.co.nz has the Associated Press article) claiming that variation in intelligence is about 50% genetic.  That claim is not new, but previous studies got estimates like that by studying the IQ of close relatives,  and this study actually measures genes.  The researchers studied 3500 unrelated people in Scotland and England,   measuring half a million genetic variants on each person and relating them to two different types of intelligence test, and found that while they couldn’t identify specific genetic variants that affected intelligence, they did find evidence that there were hundreds of variants with some effect.  (more…)

From the NZ Herald:

“The survey found almost 65 per cent of women believed they were paid less because of their gender. Just under 43 per cent of men agreed but 47 per cent didn’t.”

Unfortunately the Herald doesn’t tell us what the actual question was. Were people asked whether they, personally, were paid differently because of their gender, or whether women, on average, were paid less because of their gender?   In either case, my sympathies are with Women’s Affairs Minister Hekia Parata, who refused to offer her own answer to the “simplistic” poll question.

There are two statistical problems here. The first is what we mean by “because of their gender”.  After that’s settled, we have the problem of finding data to answer the question.  Inference about cause and effect from non-experimental data will always be hard because of both problems, but that’s what we’re here for.

Usually, when we say that income or health is worse because of some factor, we mean that if you could experimentally change the factor you would change health or income. We say high blood pressure causes strokes, and we mean that if you lower the blood pressure of a bunch of people, fewer of them will get strokes.  This isn’t possible for gender — not only can we not assign gender at random, we can’t even say what it would mean to do that.  Would a female Dan Carter be his sister Sarah, or Irene van Dyk? (more…)

Q: So, have you seen the article about Vitamin D and diabetes?

A: Of course. The tireless staff of StatsChat read even West Island newspapers. It’s a good report, too.

Q: What did the researchers do?

A: They studied 5200 people without diabetes, following them up for five years. 199 of them developed diabetes. The people who ended up with diabetes started off with lower vitamin D levels in their blood.

Q: Where did you get those details?

A: The abstract for the study publication (you can also get the full text there free if you’re at a university or if you wait until next year).

Q: Isn’t it annoying that newspaper websites don’t provide any links to that sort of information?

A: It’s like you’re reading my mind.

Q: One of the study authors is quoted as saying “”It’s hard to underestimate how important this might be.” What do you think?

A: I think he meant “overestimate”.

Q: So, how important is this finding?

A: If it really is an effect of vitamin D, it would be really important.  A simple supplement would be able to dramatically reduce the risk of diabetes.

Q: How can we tell?

A: Someone needs to do a randomized trial, where half the participants get vitamin D and half get a dummy pill. If the effect is real, fewer people getting vitamin D will end up with diabetes.

Q: That sounds like a good idea. Is someone doing a trial?

A: Yes, Professor Peter Ebeling, of the the University of Melbourne.

Q: Is there some useful website where I can find more information about the trial?

A: Indeed.

Q: Will it work?

A: No.

Q: Are you sure?

A: No, that’s why we need the trial.  But it’s a trial of vitamin supplementation, which almost always has disappointing results, and it’s a trial  in adult-onset diabetes, which almost always has disappointing results.

Stats.org has published an excellent article on the research surrounding risk factors for Sudden Infant Death Syndrome (SIDS), in particular breastfeeding:

Let there be no doubt: not breastfeeding and SIDS are correlated. The problem is that breastfeeding is correlated with many other factors as well, any of which could be the “cause” (or “causes”) behind an increased SIDS rate among people who use formula instead of mothers’ milk. These include a variety of social and cultural differences, differences in care, differences in other feeding patterns, differences in sleeping patterns, differences in genetic makeup, differences in home environment, differences in medical care, etc. The question is whether the evidence points to breastfeeding (or mother’s milk) as a preventative factor by itself and independent of all the other factors with which breastfeeding tends to go hand in hand.

It continues to unravel the statistics and evidence and concludes by saying:

Without the science, the claims of cost due to not breastfeeding – 447 babies and almost 5 billion dollars in economic loss — are like an empty bottle: wanting for real substance.

Read more »