There’s been a recent recurrence of the factoid that 90% of results in epidemiology are wrong. For example, @StatFact on Twitter posted ‘Empirical evidence is that 80-90% of the claims made by epidemiologists are false.’ with a link to a talk by Stanley Young at the National Institute of Statistical Sciences. I replied “For suitable values of ‘claim’ and ‘false'”, and if you don’t want to read further, that’s a good summary. (more…)
The Herald has a story claiming that facial proportions indicate racism (in men). Well, they have a headline claiming that. The story (and the research paper, even more explicitly) pretty much contradicts the headline, and says that facial proportions have nothing to do with racism but indicate whether men write magazine articles about express their racist views or hide them.
If you believe the story, the relationship is very strong
Looking at the photos from the first study, a new group of participants evaluated men with wider, shorter faces as more prejudiced, and they were able to accurately estimate the target’s self-reported prejudicial beliefs just by looking at an image of his face.
and to be fair to the journalist, that’s what the researchers said. If you look at their actual results, it’s not what they found.
They found an average difference of 1.92 on a 6-point perceived-racism scale for men who differ by 1 unit on the facial proportion scale. The full range of the facial proportion scale appears to only be about 0.7 units. The paper doesn’t tell us the actual distribution of the measurements, but according to another research paper I found on the internets, the standard deviation of this facial proportion scale is about 0.12. That means two randomly chosen men would differ by about 0.17 units, and the relationship would predict a difference in the 6-point perceived-racism scale of about 0.3 units. The association with self-reported racism was about as strong, though I haven’t been able to find enough information to compute the predicted differences (it shouldn’t be this hard).
In my book, that’s not an “accurate estimate”.
An article in Nature (not free access, unfortunately) by Australian molecular biologist David L. Vaux
“Experimental biologists, their reviewers and their publishers must grasp basic statistics, or sloppy science will continue to grow.”
This doesn’t come as a surprise to statisticians, but it is nice to get the support from the biology side. His recommendations are also familiar and welcome
How can the understanding and use of elementary statistics be improved? Young researchers need to be taught the practicalities of using statistics at the point at which they obtain the results of their very first experiments.
[Journals] should refuse to publish papers that contain fundamental errors, and readily publish corrections for published papers that fall short. This requires engaging reviewers who are statistically literate and editors who can verify the process. Numerical data should be made available either as part of the paper or as linked, computer-interpretable files so that readers can perform or confirm statistical analyses themselves.
Professor Vaux goes on to say
When William Strunk Jr, a professor of English, was faced with a flood of errors in spelling, grammar and English usage, he wrote a short, practical guide that became The Elements of Style(also known as Strunk and White). Perhaps experimental biologists need a similar booklet on statistics.
And here I have to quibble. Experimental biologists already have too many guides like Strunk & White, full of outdated prejudices and policies that the authors themselves would not follow. What we need is a guide that lays out how good scientists and statisticians actually do handle common types of experiment (ie, evidence-based standard recipes), together with some education on the basic principles: contrasts, blocking, randomization, sources of variation, descriptions of uncertainty. And perhaps a few entertaining horror stories of Doing It Rong and the consequences.
The Herald has a good story about gambling: the total expenditure net of winnings is $2 billion/year, [update or $16billion gross] about $3600 per capita. That’s quite a lot. For example, looking at the Retail Trade Survey, it’s about twice what we spend on alcohol, and about the same as expenditures on all recreational goods.
What’s harder to tell is how the expenditures break down across
It’s quite possible that the last category is a large fraction of total expenditure, while being a small fraction of total people.
[update: the $3600 figure includes winnings. Losses are a more plausible $450/capita]
Nate Silver is doing a Q &A session at reddit.com (they have a running feature I am A …. ask me anything)
Polls have Labour closing in on Nats
A One News-Colmar Brunton poll released last night and taken a week after Labour’s leadership spat, saw the party’s vote lift 3 percentage points to 35 per cent, with the Greens up one on 13 per cent.
and later on
The One News poll of 1000 had a margin of error of 3.1 per cent.
Did they have another poll? Well, the story goes on to say
Meanwhile, a 3News-Reid Research poll, also released yesterday, showed Labour on the rise at 34.6 per cent, up 1.6 percentage points. The Greens improved 1.3 to 12.9 per cent, while National was down 1.8 at 47 per cent.
It doesn’t make as good a lead, because Labour was only up 1.6%, not 3% in this poll. But if there are two polls, perhaps there are even more out there.
The poll of polls at pundit.co.nz shows a steady trend towards higher support for Labour, and a trend towards lower support for the Greens (with a bit more variation around the trend). Their last update (including these two polls) put Labour up by 1% and Greens up by 0.2%. But that would be less newsworthy.
The Herald (as our Stat-of-the-Week nomination points out) is claiming
Manukau is the luckiest Lotto suburb in Auckland, the Herald can reveal.
This looks as if it’s claiming that tickets bought in Manukau have been more likely to win. If this was true, it would still be useless, because future lotto draws are independent of past ones.
It’s even more useless because there is no denominator: not tickets sold, not people in the suburb, not even number of Lotto outlets in the suburb.
What the statistic, and the accompanying infographic, really identifies is the suburbs that lose the most money on Lotto. That’s why Manukau and Otara are ‘lucky’ and Mt Eden and Remuera are ‘unlucky’, the sort of willfully perverse misrepresentation of the role of chance that you more usually see in right-wing US outlets.
Stuff thinks so, but actually it’s hard to say. The statistics have recently been revised (as the paper complained about in April).
The paper, and the Labor spokewoman, focus on the numbers of deaths in 2008 and 2011: 18 and 27 respectively.
The difference between 18 and 27 isn’t all that statistically significant: a difference that big would happen by chance about 10% of the time even assuming all the deaths are separate cases. It’s pretty unlikely that the 50% difference reflects a 50% increase in domestic violence, but it might be a sign that there has been some increase. Or not.
The Minister doesn’t do any better: she quotes a different version of the numbers, women killed by their partners (6 in 2008, 14 in 2009, 9 in 2011), as if this was some sort of refutation, and points to targets that just say the government hopes things will improve in the future.
There’s no way that figures for deaths, which are a few tenths hundredths of a percent of all cases investigated by the police, are going to answer either the political fingerpointing question or the real question of how much domestic violence there is, and whether it’s getting better or worse. It’s obvious why the politicians want to pretend that their favorite numbers are the answer, but there’s no need for journalists to go along with it.
Prompted by a comment from Cosma Shalizi (who, irritatingly, is right as usual), I tried some simulated data on the great ape midlife crisis, and I’m now even less impressed with the paper.
There’s very strong evidence in the paper that the youngest apes are rated as happier by their handlers, and that the relationship with age is not linear. What’s less clear is that there is a U-shape.
I simulated data where the score decreased sharply at young ages and then flattened out, but didn’t go up again in old age, and analysed as the researchers did in the paper. This is what the data and true relationship look like:
Fitting the model used in the research paper gives a U-shape, because the model they fitted can only give U-shapes. As in the paper, the minimum is in middle life. The statistical significance for the non-linear term is better than in the paper.
In the paper, the fitted U-shape was rescaled to have mean 50 and standard deviation 10, and the raw data weren’t displayed, making the relationship look much stronger:
And, as in the paper, the banded model in the Appendix is at least consistent with a U-shape.
So, you can get the results in the paper with no midlife crisis at all. Now, you don’t necessarily get results like these: if you run the code with different sets of random numbers you get results this good maybe half the time. And of course, you could also get those results with a true U-shape.
The point is that the results in the paper are not strong evidence for a U-shape, and the graphs and tables in the paper give the impression of much stronger evidence than they actually contain. A much better graph would use a scatterplot smoother, to draw a curve through the data objectively, and something like bootstrap replicates of the curve to give a real impression of uncertainty. This doesn’t give a formal test, but at least it shows what the data are saying.
It would take some thought to come up with a good formal test, but a graph like this one should be a minimum threshold. If there really is evidence of a midlife crisis in apes this graph would show it, and if there isn’t, it wouldn’t. (more…)
Stuff and the Herald have the identical AP story, so you can read either one
Chimpanzees in a midlife crisis? It sounds like a setup for a joke. But there it is, in the title of a report published in a scientific journal: ‘Evidence for a midlife crisis in great apes.
The researchers asked handlers to estimate ‘well-being’ for 508 great apes: 172 orang-utans, the rest, chimpanzees. They fitted a statistical model to look for a decrease in mid-life followed by an increase, and got dramatic graphs
The x-axis is in years, showing the trough of despondency in the mid-thirties. The y-axis isn’t in anything — the curves were rescaled to look similar and the numbers are arbitrary.
The reason the curves look so dramatic is partly the higher-than-wide shape of the graph, but mostly the lack of any indication of uncertainty. The data are actually consistent with a wide range of flatter or steeper U-shapes and with the `mid-life’ crisis happening anywhere over quite a range of years. I can’t be more precise than that, because the researchers don’t even provide the necessary information to compute the uncertainty in the curve [they give uncertainties in regression coefficients, but not correlations between them].
However, they do have an appendix that looks at chopping up age into five-year bands and estimating the midlife crisis that way. They don’t give a graph, but they do give enough information to draw one. It’s not as impressive.
The U-shaped pattern does seem to probably be real (though the extent to which the so-called mid-life crisis is really the apes’ problem rather than than the handlers’ problem isn’t clear), but the graphs in the research paper are overselling it. Badly.
[Update: the intervals in the plot are +/- 1.4 standard errors for the coefficient. This should be in the ballpark for a 95% interval for the mean for that age group]