One poll is not enough

As Patrick Gower said recently about the new Newshub/Reid Research polls

“The interpretation of data by the media is crucial. You can have this methodology that we’re using and have it be bang on and perfect, but I could be too loose with the way I analyse and present that data, and all that hard work can be undone by that. So in the end, it comes down to me and the other people who present it.”

This evening, Newshub has the headline Poll: Labour crumbles, falling towards defeat. That’s based on a difference between two polls of 4.2% for Labour on its own, or 3.1% for a Labour/Greens alliance.

The poll has a ‘maximum margin of error’ of 3.1%, but that’s for support in this poll. For change between two polls, the maximum margin of error from the same assumptions is larger: 4.4%.

There’s pretty good evidence the decrease for Labour is likely to be real: at 25-30% support the random variation is smaller.  Even so, an uncertainty interval based on the usual optimistic assumptions about sampling goes from a decrease of 0.3% to a decrease of 8.1%.

The smaller change for the Greens/Labour alliance, this could easily just be the sort of thing that happens with polling. Or, it could be a real crumble. Or anything in between

Certainly, even a 3.1% decrease in support is potentially a big deal, and could be news. The problem is that a single standard NZ opinion poll isn’t up to the task of detecting it reliably. Whether it’s news or not is up to the judgement (or guesswork) of the media, and the demands of the audience.  Even that would be ok, if everyone admitted the extent to which the data just serve to dilute the reckons, rather than glossing over all the uncertainty.

If anyone wants less-exciting summaries, my current recommendation for an open, transparent, well-designed NZ poll aggregator is this by Peter Ellis.