Posts filed under Correlation vs Causation (68)

There’s an interesting piece on cannabis risks at Project Syndicate. One of the things they look at is the correlation between frequent cannabis use and psychosis.  Many people are, quite rightly, unimpressed with the sort of correlation, since it isn’t hard to come up with explanations for psychosis causing cannabis use or for other factors causing both.

However, there is also some genetic data.  The added risk of psychosis seems to be confined to people with two copies of a particular genetic variant in a gene called AKT1. This is harder to explain as confounding (assuming the genetics has been done right), and is one of the things genetics is useful for. This isn’t just a one-off finding; it was found in one study and replicated in another.

On the other hand, the gene AKT1 doesn’t seem to be very active in brain cells, making it more likely that the finding is just a coincidence.  This is one of the things bioinformatics is good for.

In times like these it’s good to remember Ben Goldacre’s slogan “I think you’ll find it’s a bit more complicated than that.”

TV3 has tried to stir up the issue of the death penalty in New Zealand.  They have a poll showing majority opposition by the country as a whole, and by supporters of every party except NZ First.  Even the Sensible Sentencing Trust isn’t in favor.

The lead-in to the story is that the murder rate has never ‘recovered’ from the abolition of the death penalty.  They have a graph showing homicides per capita rising and then falling again, but not to the earlier levels.

Using the term ‘recovered’ comes very close to asserting a causal connection; but is there even a reliable correlation? International comparisons are useful here.  I don’t have long time series for homicide, but Kieran Healy has produced a graph of international trends in deaths due to assault. This isn’t the same as homicide, but is close enough to be relevant.

Here’s the New Zealand panel, with the arrow indicating the abolition of the death penalty. The details are slightly different from those for homicide, but the basic trend is the same that TV3 reports.

and here’s the international comparison, with NZ second from the bottom, on the left. As usual, click to embiggen

The NZ pattern is very similar to other countries, including Australia (where abolition didn’t happen until about 10 years later), Finland (where it was abolished in 1949 for crimes committed in peacetime), and Switzerland (1942).

If you look at the countries that still have the death penalty, murder rates are low and falling in Japan, South Korea had the same sort of rise and fall that NZ has had (over a shorter time scale), and of course there’s the USA.

It doesn’t look as though the death penalty is a major driving force in these patterns.

Two hundred and sixty six years ago today, James Lind began what is regarded as the first proper controlled clinical trial

On the 20th May, 1747, I took twelve patients in the scurvy on board the Salisbury at sea. Their cases were as similar as I could have them. They all in general had putrid gums, the spots and lassitude, with weakness of their knees. They lay together in one place, being a proper apartment for the sick in the fore-hold; and had one diet in common to all, viz., water gruel sweetened with sugar in the morning; fresh mutton broth often times for dinner; at other times puddings, boiled biscuit with sugar etc.; and for supper barley, raisins, rice and currants, sago and wine, or the like. Two of these were ordered each a quart of cyder a day. Two others took twenty five gutts of elixir vitriol three times a day upon an empty stomach, using a gargle strongly acidulated with it for their mouths. Two others took two spoonfuls of vinegar three times a day upon an empty stomach, having their gruels and their other food well acidulated with it, as also the gargle for the mouth. Two of the worst patients, with the tendons in the ham rigid (a symptom none the rest had) were put under a course of sea water. Of this they drank half a pint every day and sometimes more or less as it operated by way of gentle physic. Two others had each two oranges and one lemon given them every day. These they eat with greediness at different times upon an empty stomach. They continued but six days under this course, having consumed the quantity that could be spared. The two remaining patients took the bigness of a nutmeg three times a day of an electuray recommended by an hospital surgeon made of garlic, mustard seed, rad. raphan., balsam of Peru and gum myrrh, using for common drink narley water well acidulated with tamarinds, by a decoction of wich, with the addition of cremor tartar, they were gently purged three or four times during the course.

The consequence was that the most sudden and visible good effects were perceived from the use of the oranges and lemons; one of those who had taken them being at the end of six days fit four duty. The spots were not indeed at that time quite off his body, nor his gums sound; but without any other medicine than a gargarism or elixir of vitriol he became quite healthy before we came into Plymouth, which was on the 16th June. The other was the best recovered of any in his condition, and being now deemed pretty well was appointed nurse to the rest of the sick …

Lind knew very little about scurvy apart from the typical progress of the disease, and he had no real idea of how the treatments might work.  That’s a handicap in coming up with ideas for treatment, but not in doing fair tests of whether treatments work.

The trial didn’t have an untreated group: all the patients got one of the treatments recommended by experts.  There’s no need for a controlled trial to have an untreated group — if there is an existing treatment, you want to compare to that treatment; if there is none, you may want to compare immediate vs delayed treatment.

Despite the dramatic success of fruit juice in the trial, it wasn’t adopted as a treatment. That, sadly, can still be the case today.  New drugs or surgical techniques are taken up enthusiastically, but boring interventions like nurse home visits or surgery checklists get less attention. Still, things are much better than they were even twenty years ago. Nearly all of medicine accepts the idea of randomized controlled comparison, and it is spreading to other areas such as development economics.

There are two excellent, free books about clinical trials and health choices: Testing Treatments, from the James Lind Initiative, and Smart Health Choices, from Les Irwig (at Sydney Uni) and coworkers.

Much of clinical trials development is unapologetically technical, but there are important areas where public participation can help:

• The James Lind Alliance asks patients and clinicians to say what questions matter most.  Clinical trials still tend to answer questions that are scientifically interesting or commercially important, not what actually matters most to patients
• The Cochrane Collaboration is attempting to collect and summarise all randomized trials.  The Cochrane Consumers Network is for non-specialist participation — in particular as Consumer Referees to help ensure that summaries of research address the questions that are important to consumers and are presented in language that consumers can understand.
• Ethics Committees that review clinical trials and other human research in New Zealand are required to have non-specialist community members. This is a substantial commitment, but one that is important if ethics committees are to be more than just red tape.
• And if you haven’t yet signed the AllTrials petition, calling for the results of all clinical trials to be published so we can know what treatments work, now would be an excellent time.

• From Felix Salmon: US population is increasing, and people are moving to the cities, so why is (sufficiently fine-scale) population density going down? Because rich people take up more space and fight for stricter zoning.  You’ve heard of NIMBYs, but perhaps not of BANANAs
• From the New York Times.  One of the big credit-rating companies is no longer using debts referred for collection as an indicator, as long as they end up paid.  This isn’t a new spark of moral feeling, it’s just for better prediction.
• And from Felix Salmon again: Firstly, Americans are bad at statistics. When it comes to breast cancer, they massively overestimate the probability that early diagnosis and treatment will lead to a cure, while they also massively underestimate the probability that an undetected cancer will turn out to be harmless.

There’s a good piece in one of the New Yorker‘s blogs  about the Human Connectome Project and the proposed Brain Activity Map.  The Connectome project has produced two terabytes of functional and structural data on the brains of 68 volunteers, and the Brain Activity Map is more or less what it says on the tin.

Almost certainly people will be able to do something useful with all this data, but some of the claims for what it means to our understanding of the brain are a bit much. As the RealClearScience blog points out (in a slightly different context), we know the complete nervous system of the nematode C. elegans.  We know every cell and all its connnections to other cells.  We still can’t use this knowledge to reliably predict the nematode’s behaviour even by brute-force simulation, let alone by sophisticated analysis.

Simply using Big Data to work out how a complex system functions requires a lot of simplifying assumptions to be true.  This isn’t because we’re not smart enough to build more complex models (though we’re not), and it isn’t because the computation is beyond us (though it is), it’s a fundamental limitation on learning without an underlying theory to help you.

The way Amazon or Netflix does prediction with all its data is to look for a large group of people who are similar to you, in relevant ways, and see what they bought or watched. That sounds easy, but the weasel words are ‘in relevant ways’.  If you have a moderately large number of variables, there are far too many ways in which people, or nerve signals, or protein concentrations could be similar, and you need to decide which ones are relevant.   This is critical, because finding the true relationships in large numbers of associations is only possible if nearly all the associations are zero; in current jargon, the model is ‘sparse’.

In order to see sparseness, you need to know how to look.    Consider the economy: if you just look at associations between measurements, everything is correlated: you see inflation, you see population growth,  you see seasonal variation.  These patterns need to be removed to get a sparse model where you’ve got some hope of disentangling cause and effect.

In really complex fields like brain activity, we don’t know enough about how to pose the problem so that Big Data will have a hope of solving it.

Remember the correlation between current chocolate consumption and all-time Nobel Prizes?

Two British researchers now have done the same exercise for current milk consumption. Their letter, in the journal Practical Neurology suggests (I hope not seriously) that vitamin D might be responsible. They used Messerli’s data on Nobel Prizes, and don’t seem to have noticed any of the problems with it.

As you will remember, we showed length of country name (per capita) was rather more strongly correlated with Nobel Prizes (per capita) than chocolate consumption, and it also beats milk consumption. It’s also much more convincing as a causal relationship: the country names are much more constant over the time the Nobel Prize data were accumulated than milk or chocolate consumption, and since there’s no plausible mechanism for wealthy countries to have longer names than poor countries we avoid economic confounding.

From Columbia Journalism Review (via Tony Cooper), a good long piece on science journalism by David H. Freedman (whom Google seems to confuse with statistician David A. Freedman)

What is a science journalist’s responsibility to openly question findings from highly credentialed scientists and trusted journals? There can only be one answer: The responsibility is large, and it clearly has been neglected. It’s not nearly enough to include in news reports the few mild qualifications attached to any study (“the study wasn’t large,” “the effect was modest,” “some subjects withdrew from the study partway through it”). Readers ought to be alerted, as a matter of course, to the fact that wrongness is embedded in the entire research system, and that few medical research findings ought to be considered completely reliable, regardless of the type of study, who conducted it, where it was published, or who says it’s a good study.

Worse still, health journalists are taking advantage of the wrongness problem. Presented with a range of conflicting findings for almost any interesting question, reporters are free to pick those that back up their preferred thesis—typically the exciting, controversial idea that their editors are counting on. When a reporter, for whatever reasons, wants to demonstrate that a particular type of diet works better than others—or that diets never work—there is a wealth of studies that will back him or her up, never mind all those other studies that have found exactly the opposite (or the studies can be mentioned, then explained away as “flawed”). For “balance,” just throw in a quote or two from a scientist whose opinion strays a bit from the thesis, then drown those quotes out with supportive quotes and more study findings.

I think the author is unduly negative about medical science — part of the problem is that published claims of associations are expected to have a fairly high false positive rate, and there’s not necessarily anything wrong with that as long as everyone understand the situation.  Lowering the false positive rate would either require much higher sample sizes or a much higher false  negative rate, and the coordination problems needed to get a sample size that will make the error rate low are prohibitive in most settings (with phase III clinical trials and modern genome-wide association studies as two partial exceptions).    It’s still true that most interesting or controversial findings about nutrition are wrong, and that journalists should know they are mostly wrong, and should write as if they know this.   Not reprinting Daily Mail stories would probably help, too.

You may have heard some under-rehearsed radio wittering from me on this topic, so I thought I should write something more coherent.

Last August, researchers from the Dunedin Cohort Study published a paper showing that people who had been heavy cannabis users as teenagers performed worse on cognitive function tests later in life, where this wasn’t true of people who started using cannabis as adults. One natural interpretation of these associations is that cannabis has toxic effects during brain development.  As I pointed out at the time, the evidence isn’t overwhelming (since it’s a relatively small study and we know environmental factors can lead to differences as big as those observed) but was somewhat persuasive and is probably better than other studies on this topic.

Now, a Norwegian economist has argued that the results could be explained by purely sociological factors: that people from low-socioeconomic status backgrounds are more likely to use cannabis, and to perform worse on cognitive function tests, and that the difference in cognitive function tests tends to increase over time after school.  He is correct; this could explain the published results.  However, the Dunedin Cohort Study researchers have done further analyses in response, and while the socioeconomic explanation was reasonable, it seems to not be true.  Both the relationship between socioeconomic status and cannabis use, and the relationship between low socioeconomic status and change over time in cognitive function test results were weak in this particular data set.

Even if the association is real and causal there could still be explanations that don’t involve brain toxicity.  For example, imagine that people who enjoy being stoned are less likely to choose jobs and recreational activities that are cognitively demanding.  They would then to some extent tend to end up scoring lower on cognitive function tests in later life.  This, if it were true, would be an explanation that does depend on the properties of cannabis, but not on toxic effects.

Over all, this result doesn’t have huge implications for drug policy.  It doesn’t change the basic fact that cannabis is far from innocuous but is also much safer than alcohol or tobacco.  It doesn’t affect the relevant international treaties and probably won’t shift domestic public opinion.  Differences of opinion on cannabis policy questions depend mostly on different preferences, and partly on other uncertainties. For example, would legal cannabis lead to more or less alcohol consumption?

I’d recommend the book “Marijuana legalization: what everyone needs to know”.  This is written by a group of public-policy academics, who have varying policy preferences, but looked at what evidence they could agree on. It gives a series of questions and what is known about their answers. Unfortunately it’s not (yet?) in the Auckland city library.

Links: Stuff, Science Media Centre

• A zombie story:  Stuff has an opinion piece about chocolate and intelligence, based on the joke article in the New England Journal of Medicine back in October. We covered what was wrong with it then, and showed that you get better correlation with the number of letters in the country’s name than with chocolate consumption.  [Update: the piece is really from an Australian publication, with a light makeover for the Kiwi audience.]
• A better joke.  An article in the Canadian Medical Association Journal looks at the impact on clinical trials if the world ends on December 21, as the Mayan calendar does not give the slightest suggestion will happen.
• Language Log examines the inability of journalists around the world to get the basic numbers right in reporting a study on water chlorination and allergies (a story that the NZ media seem to have had the good sense not to pick up).
• “Good data-driven journalism both publishes as much data as possible, and uses the data to drive conclusions, rather than simply dropping numbers into a foreordained article.”  Felix Salmon, complaining about a New York Times story.
• The American Statistical Association has a new prize for “Causality in Statistics Education”, aimed at encouraging the teaching of basic causal inference in introductory statistics courses.

Stuff is reporting “Food and learning connection shot down”,based on a local study

Researchers at Auckland University’s School of Population Health studied 423 children at decile one to four schools in Auckland, Waikato and Wellington for the 2010 school year.

They were given a free daily breakfast – Weet-Bix, bread with honey, jam or Marmite, and Milo – by either the Red Cross or a private sector provider.

My first reaction on reading this was: why didn’t they take this opportunity to do a randomised trial, so we could actually get reliable data.  So I went to the Cochrane Library to see what randomised trials had been done in the past. These have mostly been in developing countries and have found improvements in growth, but smaller differences in school performance.

Then I tried asking the Google, and its second link was a paper by Dr Ni Mhurchu, the researcher mentioned in the story, detailing the plans for a randomised trial of school breakfasts in Auckland.  At that point it was easy to find the results, and see that in fact Stuff is talking about a randomized trial. They just didn’t think it was important enough to mention that detail.

To the extent that one can trust the Stuff story at this point, there seem to be three reactions:

• I don’t believe it because my opinions are more reliable than this research
• Lunch would work even if breakfast didn’t
•  We should be making sure kids have breakfast even if it doesn’t improve school performance.

The latter two responses are perfectly reasonable positions to take (though they’re more convincing where they were taken before the results came out).  School lunches might be more effective than breakfasts, and the US (hardly a hotbed of socialism) has had a huge school nutrition program for 60 years.

Still, if we’re going to supply subsidised meals to school kids, we do need to know why we’re doing it and what we expect to gain.    This study is one of the first to go beyond just saying that the benefits are obvious.