Posts from March 2014 (59)

Heavy drinking and the societal harm it causes is a big issue and attracts a lot of media and scholarly attention (and Statschat’s, too). So we were interested to see today’s new release from the Journal of the American Medical Association. It describes a double-blind, parallel-group, individually-randomised trial that studied moderate to heavy student drinkers from seven of our eight universities to see if a web-based alcohol screening and intervention programme reduced their unhealthy drinking behaviour.

And the short answer? Not really. But if they identified as Māori, the answer was … yes, with a caveat. More on that in a moment.

Statistician Nicholas Horton and colleagues used an online questionnaire to identify students at Otago, Auckland, Canterbury, Victoria, Lincoln, Massey, and Waikato who had unhealthy drinking habits. Half the students were assigned at random to receive personalised feedback and the other students had no input. Five months later, researchers followed up with the students on certain aspects of their drinking.

The overall result? “The intervention group tended to have less drinking and fewer problems then the control group, but the effects were relatively modest,” says Professor Horton. The take-away message: A web-based alcohol screening and intervention program had little effect on unhealthy drinking among New Zealand uni students. Restrictions on alcohol availability and promotion are still needed if we really want to tackle alcohol abuse.

But among Māori students, who comprise 10% of our national uni population, those receiving intervention were found to drink 22% less alcohol and to experience 19% fewer alcohol-related academic problems at the five-month follow-up. The paper suggests that Māori students are possibly more heavily influenced by social-norm feedback than non-Māori students. “Māori students may have a stronger group identity, enhanced by being a small minority in the university setting.” But the paper warns that the difference could also be due to chance, “underscoring the need to undertake replication and further studies evaluating web-based alcohol screening and brief intervention in full-scale effectiveness trials.”

The paper is here. Read the JAMA editorial here.

via Alberto Cairo, this gem from Malaprensa, a Spanish mediawatch site, originally from Marca.

This isn’t actually a pie chart, it’s a bar chart that has been horribly warped around a circle.  It shows top transfer fees in football (ie, soccer). One Neymar da Silva Santos Júnior has allegedly ended up with a transfer fee estimated at 111 million euros, through complicated arrangements. This would be a record; the originally announced figure was a mere 57 million euros, which would put Neymar in tenth place alongside Hernan Crespo

Malaprensa points out that the figures aren’t inflation-adjusted, and that they aren’t including comparable sets of payments for all the players. They don’t point out how bad the display is: compare the heights for 57 and 111 million euro, and then think about what the area comparison would be.

I’ve redrawn the bars in a sensible coordinate system,  showing the apparent differences based on the height, area, nominal euro amount, and euro amount adjusted for inflation (the last is from Malaprensa), with Crespo’s transfer fee scaled to 1 in each case

It’s much less impressive when it’s shown accurately.

The Disease Prevention Illusion: a tragedy in five parts, by Hilda Bastian

“An ounce of prevention is worth a pound of cure.” We’ve recognized the false expectations we inflate with the fast and loose use of the word “cure” and usually speak of “treatment” instead. We need to be just as careful with the P-word.

The Research Association New Zealand  has put out a new code of practice for political polling (PDF) and a guide to the key elements of the code (PDF)

The code includes principles for performing a survey, reporting the results, and publishing the results, eg:

Conduct: If the political questions are part of a longer omnibus poll, they should be asked early on.

Reporting: The report must disclose if the questions were part of an omnibus survey.

Publishing: The story should disclose if the questions were part of an omnibus survey.

There is also some mostly good advice for journalists

1. If possible, get a copy of the full poll  report and do not rely on a media release.
2. The story should include the name of the company which conducted the poll, and the client the poll was done for, and the dates it was done.
3.  The story should include, or make available, the sample size, sampling method, population sampled, if the sample is weighted, the maximum margin of error and the level of undecided voters.
4. If you think any questions may have impacted the answers to the principal voting behaviour question, mention this in the story.
5. Avoid reporting breakdown results from very small samples as they are unreliable.
6. Try to focus on statistically significant changes, which may not just be from the last poll, but over a number of polls.
7. Avoid the phrase “This party is below the margin of error” as results for low polling parties have a smaller margin of error than for higher polling parties.
8.  It can be useful to report on what the electoral results of a poll would be, in terms of likely parliamentary blocs, as the highest polling party will not necessarily be the Government.
9. In your online story, include a link to the full poll results provided by the polling company, or state when and where the report and methodology will be made available.
10. Only use the term “poll” for scientific polls done in accordance with market research industry approved guidelines, and use “survey” for self-selecting surveys such as text or website surveys.

Some statisticians will disagree with the phrasing of point 6 in terms of statistical significance, but would probably agree with the basic principle of not ‘chasing the noise’

I’m not entirely happy with point 10, since outside politics and market research, “survey” is the usual word for scientific polls, eg, the New Zealand Income Survey, the Household Economic Survey, the General Social Survey, the National Health and Nutrition Examination Survey, the British Household Panel Survey, etc, etc.

As StatsChat readers know, I like the term “bogus poll” for the useless website clicky surveys. Serious Media Organisations who think this phrase is too frivolous could solve the problem by not wasting space on stories about bogus polls.

Via @neil_, an interactive graph of ratings for episodes of The Simpsons

This comes from graphtv, which lets you do this for all sorts of shows (eg, Breaking Bad, which strikingly gets better ratings as the season progresses, then resets)

The reason the Simpsons graph has extra relevance to StatsChat is the distinctive horizontal line.  For the first ten seasons an episode basically couldn’t get rated below 7.5, after that it basically couldn’t rated above 7.5.   In the beginning there were ‘typical’ episodes and ‘good’ episodes; now there are ‘typical’ episodes and ‘bad’ episodes.

This could be a real change in quality, but it doesn’t match up neatly with the changes in personnel and style.  It could be a change in the people giving the ratings, or in the interpretation of the scale over time. How could we tell? One clue is that (based on checking just a handful of points) in the early years the high-rating episodes were rated by more people, and this difference has vanished or even reversed.

• Extrapolation: According to a US obesity researcher, by 2606 the US diet will be 100% sugar.  And according to Skepchick, we’ll be immortal. He presumably isn’t entirely serious; she totally isn’t.

• From Quartz: The problem with data journalism So here’s a set of ethics that data journalists might follow to avoid common pitfalls:

•  If people already think your data sharing plans are a bit creepy, you do not want to use the phrase “consent fetishists”. Really.

• Data visualisation: summary of  street grid angles in various US cities. The Houston one is a bit misleading because the highways are so dominant in reality but not in the summary.

Each week, we would like to invite readers of Stats Chat to submit nominations for our Stat of the Week competition and be in with the chance to win an iTunes voucher.

Here’s how it works:

• Anyone may add a comment on this post to nominate their Stat of the Week candidate before midday Friday March 28 2014.
• Statistics can be bad, exemplary or fascinating.
• The statistic must be in the NZ media during the period of March 22 – 28 2014 inclusive.
• Quote the statistic, when and where it was published and tell us why it should be our Stat of the Week.

Next Monday at midday we’ll announce the winner of this week’s Stat of the Week competition, and start a new one.

(more…)

If you’d like to comment on or debate any of this week’s Stat of the Week nominations, please do so below!

In a rant against `data journalism’ in general and fivethirtyeight.com in particular, Leon Wieseltier writes in the New Republic

Many of the issues that we debate are not issues of fact but issues of value. There is no numerical answer to the question of whether men should be allowed to marry men, and the question of whether the government should help the weak, and the question of whether we should intervene against genocide. And so the intimidation by quantification practiced by Silver and the other data mullahs must be resisted. Up with the facts! Down with the cult of facts! 

There are questions of values that are separate from questions of fact, even if the philosopher Hume went too far in declaring “no ‘ought’ deducible from ‘is'”.   There may even be things we should or should not do regardless of the consequences. Mostly, though, our decisions should depend on the consequences.

We should help the weak. That’s a value held by most of us and not subject to factual disproof.  How we should do it is more complicated.  How much money should be spent? How much should we make people do to prove they need help? Is it better to give people money or vouchers for specific goods and services? Is it better to make more good jobs available or to give more help to those who can’t get them?  How much does participating in small social and political community groups or supporting independent radical writers and thinkers help versus putting the same effort into paying lobbyists or donating to political parties or individual candidates? Is it important to restrict wealth and power of small elites, and what costs are worth paying to do so? How much discretion should be given to police and the judiciary to go lightly on the weak, and how much should they  be given strict rules to stop them going lightly on the strong? Is a minimum wage increase better than a low-income subsidy? Are the weak better off if we have a tax system that’s not very progressive in theory but it hard for the rich and powerful to evade?

As soon as you want to do something, rather than just have good intentions about it, the consequences of your actions matter, and you have a moral responsibility to find out what those consequences are likely to be.

An XKCD classic

The mouseover text says “Also, all financial analysis. And, more directly, D&D.” 

We’re getting to the point in the electoral cycle where opinion polls qualify as well. There will be lots of polls, and lots media and blog writing that tries to tell stories about the fluctuations from poll to poll that fit in with their biases or their need to sell advertising. So, as an aid to keeping calm and believing nothing, I thought a reminder about variability would be useful.

The standard NZ opinion poll has 750-1000 people. The ‘maximum margin of error’ is about 3.5% for 730 and about 3% for 1000. If the poll is of a different size, they will usually quote the maximum margin of error. If you have 20 polls, 19 of them should get the overall left:right division to within the maximum margin of error.

If you took 3.5% from the right-wing coalition and moved it to the left-wing coalition, or vice versa, you’d change the gap between them by 7% and get very different election results, so getting this level of precision 19 times out of 20 isn’t actually all that impressive unless you consider how much worse it could be. And in fact, polls likely do a bit worse than this: partly because voting preferences really do change, partly because people lie, and partly because random sampling is harder than it looks.

Often, news headlines are about changes in a poll, not about a single poll. The uncertainty in a change is  higher than in a single value, because one poll might have been too low and the next one too high.  To be precise, the uncertainty is 1.4 times higher for a change.  For a difference between two 750-person polls, the maximum margin of error is about 5%.

You might want a less-conservative margin than 19 out of 20. The `probable error’ is the error you’d expect half the time. For a 750-person poll the probable error is 1.3% for a single party and single poll,  2.6% for the difference between left and right in a single poll, and 1.9% for a difference between two polls for the same major party.

These are all for major parties.  At the 5% MMP threshold the margin of error is smaller: you can be pretty sure a party polling below 3.5% isn’t getting to the threshold and one polling about 6.5% is, but that’s about it.

If a party gets an electorate seat and you want to figure out if they are getting a second List seat, a national poll is not all that helpful. The data are too sparse, and the random sampling is less reliable because minor parties tend to have more concentrated support.   At 2% support the margin of error for a single poll is about 1% each way.

Single polls are not very useful, but multiple polls are much better, as the last US election showed. All the major pundits who used sensible averages of polls were more accurate than essentially everyone else.  That’s not to say experts opinion is useless, just that if you have to pick just one of statistical voodoo and gut instinct, statistics seems to work better.

In NZ there are several options. Peter Green does averages that get posted at Dim Post; his code is available. KiwiPollGuy does averages and also writes about the iPredict betting markets, and pundit.co.nz has a Poll of Polls. These won’t work quite as well as in the US, because the US has an insanely large number of polls and elections to calibrate them, but any sort of average is a big improvement over looking one poll at a time.

A final point: national polls tell you approximately nothing about single-electorate results. There’s just no point even looking at national polling results for ACT or United Future if you care about Epsom or Ohariu.