Beyond the margin of error
From Twitter, this morning (the graphs aren’t in the online story)
Does anybody else see the problem with this poll? Is this a representative sample? Good work @nzherald pic.twitter.com/63uwzbjZpZ
— Lennart Nout (@lennartnout) March 19, 2014
Now, the Herald-Digipoll is supposed to be a real survey, with samples that are more or less representative after weighting. There isn’t a margin of error reported, but the standard maximum margin of error would be a little over 6%.
There are two aspects of the data that make it not look representative. Thr first is that only 31.3%, or 37% of those claiming to have voted, said they voted for Len Brown last time. He got 47.8% of the vote. That discrepancy is a bit larger than you’d expect just from bad luck; it’s the sort of thing you’d expect to see about 1 or 2 times in 1000 by chance.
More impressively, 85% of respondents claimed to have voted. Only 36% of those eligible in Auckland actually voted. The standard polling margin of error is ‘two sigma’, twice the standard deviation. We’ve seen the physicists talk about ‘5 sigma’ or ‘7 sigma’ discrepancies as strong evidence for new phenomena, and the operations management people talk about ‘six sigma’ with the goal of essentially ruling out defects due to unmanaged variability. When the population value is 36% and the observed value is 85%, that’s a 16 sigma discrepancy.
The text of the story says ‘Auckland voters’, not ‘Aucklanders’, so I checked to make sure it wasn’t just that 12.4% of the people voted in the election but didn’t vote for mayor. That explanation doesn’t seem to work either: only 2.5% of mayoral ballots were blank or informal. It doesn’t work if you assume the sample was people who voted in the last national election. Digipoll are a respectable polling company, which is why I find it hard to believe there isn’t a simple explanation, but if so it isn’t in the Herald story. I’m a bit handicapped by the fact that the University of Texas internet system bizarrely decides to block the Digipoll website.
So, how could the poll be so badly wrong? It’s unlikely to just be due to bad sampling — you could do better with a random poll of half a dozen people. There’s got to be a fairly significant contribution from people whose recall of the 2013 election is not entirely accurate, or to put it more bluntly, some of the respondents were telling porkies. Unfortunately, that makes it hard to tell if results for any of the other questions bear even the slightest relationship to the truth.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
I think part of the problem could be the survey wording. It’s possible that when asked “Did you vote for Len Brown in last year’s local body elections?” a person who did not vote at all might quite reasonably respond “No”, especially if they didn’t listen carefully to all the options. That kind of response is being interpreted here as saying that they voted for someone else, which may not be the case. A better alternative to “No” could have been “I did vote, but for another candidate”.
11 years ago
Maybe? But (a) that’s exactly the sort of thing that election pollsters know how to handle, eg by first asking “Did you vote?” and (b) it still doesn’t explain the results
At the election, Len Brown got votes from 48% of those voting and 17% of those eligible. Neither matches the 31% in this sample.
11 years ago
Good stuff Thomas.
My initial response, based on some experiences reviewing these sorts of things, is “248 Aucklanders were asked”, but how many responded. Or, more likely, how were asked before 248 Aucklanders responded. If they got a response rate of more than 50% they would be doing well for these sorts of polls. Possibly this information is on the Digipoll website.
I am constantly puzzled about people having a result they can compare with something vaguely comparable to see if it makes sense but rarely doing that comparison.
11 years ago
It could be the definition of Auckland i.e. is it the CBD and inner suburbs or does it include South Auckland?
I could well believe that people on the end of landlines in central Auckland e.g. Remuera, Grey Lynn, Epsom, being white and older, voted at higher rates and didn’t vote for Len Brown.
11 years ago
That would explain the Len Brown proportion of votes, but I don’t think it explains the low “Didn’t vote.” I can’t see the turnout being *that* much higher in old Auckland City than in North Shore or Waitakere. There doesn’t seem to be a huge discrepancy between local board and councillor vote numbers in south Auckland and elsewhere, either.
And it would be a truly damning indictment of Digipoll if that’s what they did.
11 years ago
But Auckland City had a burning issue that would have drawn voters – mowing berms. It was a big deal. ;->
It would be interesting to know if Digipoll randomly chose a respondent within household and weighted by (voting-age) household size. Len Brown supporters would be expected to live in larger households.
It could also be people lying to avoid being seen as “naughty” (not voting) or being on the “wrong” side (voting for Len Brown).
11 years ago
Yes, there’s a lot about the methodology that would be interesting to know, including the exact questions.
I think lying must have been involved, and probably bad luck with sampling, and sampling bias as well.
11 years ago
Oh Lenny Boy, the berms, the berms need mowing…
11 years ago
I don’t know how you lump Grey Lynn in with Remuera and Epsom! Grey Lynn is not very white and certainly not that old. In the last general election, the vote at Grey Lynn polling stations was 35% to parties on the left and 60% to the right.
11 years ago
Obviously, the 35% was to the RIGHT vs 60% to the LEFT…
11 years ago
I did qualify with “at the end of landlines” – even in Grey Lynn I would guess the young and not white would be way less likely to be contactable by landline.
Ponsonby would have been a better choice but I couldn’t think of the name off the top of my head.
11 years ago
Another question for The Herald and the Digipollsters: is the sample for this poll the same one which was released two days ago:
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11221487
If the Digipollsters are using an omnibus telephone survey it is less likely that an entirely different survey was undertaken with a different sampling methodology. I would think it was likely that the Len Brown questions were just added in for the Auckland portion of the sample. In which case…is it just the Auckland portion which has a peculiar set of results, or was the poll of two days ago similarly afflicted with a very unusual sample? Or was the Len Brown survey an entirely stand alone survey with different methodology?
Somebody mentioned this in the 4 -5 pm segment on National Radio, and it seemed a good question.
11 years ago
One possible theory is they weighted at the national level, rather than setting targets within locations. If so, the national level results could be okay, but some odd things could happen when you analyse by smaller subgroups (due to large weighting factors, a small number of people can have a large influence over the smaller subgroup results).
This is not a criticism of the poll at the total sample level – but if robust regional analyses are important it’s good to think about the degree to which your weighting scheme increases variance in the regional result.
11 years ago
That makes some sense, and 248 does sound very much like Auckland’s fraction of a total of 750.
I still don’t think variance is enough to explain it, but the whole affair is more plausible as an appendage to a national survey than as a stand-alone problem.
11 years ago
There is a long-standing tradition of people telling porkies about whether they voted. Social desirability bias when talking to a live interviewer.
In addition to that, a survey question asking about a hypothetical match-up between a named incumbent and an unnamed challenger is subject to a different bias. This is because people who are not solidly behind the incumbent are free to imagine their ideal challenger, and then pit the flawed incumbent against the not flawed challenger. This overestimates the challenger’s likely vote share.
11 years ago
Yes, in principle, and the point about the future challenger is a good one — it must be, because I’ve made it before ;-)
It’s surprising the social desirability bias would be so strong. Everyone knows local government election turnout is low, and even so, it seems fewer people admit to not voting than admit in other surveys to smoking pot or cheating on their partners.
I’m sure that’s part of it, but I’m still surprised the results ended up *so* bad. As I said on Twitter, this isn’t puzzled-as-rhetorical-device; I don’t get it.
11 years ago