Is epidemiology 90% wrong?

There’s been a recent recurrence of the factoid that 90% of results in epidemiology are wrong. For example, @StatFact on Twitter posted ‘Empirical evidence is that 80-90% of the claims made by epidemiologists are false.’ with a link to a talk by Stanley Young at the National Institute of Statistical Sciences.  I replied “For suitable values of ‘claim’ and ‘false'”, and if you don’t want to read further, that’s a good summary.

The claim is obviously wrong if interpreted at face value: firstly, epidemiologists mostly make claims of the form “in this data set, processed meat consumption was higher in people who ended up dying during follow-up”. These claims are usually true, modulo occasional fraud or error. Certainly 90% of them aren’t false. Secondly, if some papers say people who ate more processed meat were more likely to die and other papers say they weren’t, one or other claim must be true and the whole 90% thing makes no sense.

Interpreted in its strongest sense, then, the claim that 90% of epidemiological results are false is uninterestingly false.  Using this sort of interpretation would obviously be unreasonable.  Unfortunately, it’s this sort of interpretation of results in epidemiology that you need in order to make the 90% statistic true and interesting.

Let’s look at the results and conclusion from the abstract of the processed-meat paper I wrote about last week

Results: As of June 2009, 26,344 deaths were observed. After multivariate adjustment, a high consumption of red meat was related to higher all-cause mortality (hazard ratio (HR) = 1.14, 95% confidence interval (CI) 1.01 to 1.28, 160+ versus 10 to 19.9 g/day), and the association was stronger for processed meat (HR = 1.44, 95% CI 1.24 to 1.66, 160+ versus 10 to 19.9 g/day). After correction for measurement error, higher all-cause mortality remained significant only for processed meat (HR = 1.18, 95% CI 1.11 to 1.25, per 50 g/d). We estimated that 3.3% (95% CI 1.5% to 5.0%) of deaths could be prevented if all participants had a processed meat consumption of less than 20 g/day. Significant associations with processed meat intake were observed for cardiovascular diseases, cancer, and ‘other causes of death’. The consumption of poultry was not related to all-cause mortality. 

Conclusions: The results of our analysis support a moderate positive association between processed meat consumption and mortality, in particular due to cardiovascular diseases, but also to cancer.

So, the claims in the results section are about observed differences in a particular data set, and presumably are true.  The claim in the conclusion is that this ‘supports’ ‘an association’.  If you interpret the conclusion as claiming there is definitive evidence of an effect of processed meat, you’re looking at the sort of claim that is claimed to be 90% wrong. Epidemiologists don’t interpret their literature this way, and since they are the audience they write for, their interpretation of what they mean should at least be considered seriously.

So, how good is the evidence that 90% of epidemiology results interpreted this way are false?  It depends.  The argument is that most hypotheses about effects are wrong, and that the standard for associations used in epidemiology is not a terribly strong filter, so that most hypotheses that survive the filter are still wrong.  That’s reasonably as far as it goes.  It does depend on taking studies in isolation.  In this example there are both previous epidemiological studies and biochemical evidence to suggest that fat, salt, smoke, and nitrates from meat curing might all be harmful. In other papers the background evidence can vary from strongly in favor to strongly against, and this needs to be taken into account.

A more moderate and reasonable claim about epidemiological studies is that very few of them provide information that should make you change your behaviour.  This applies to the good ones as well as the bad ones.