How did the polls do?

There were two (or possibly three) features of the preliminary election results on Saturday being discussed as surprising: the large Labour margin, the win in Auckland Central by Chlöe Swarbrick of the Green Party, and possibly the win in Waiariki by Rawiri Waititi of the Māori Party.  How do these really compare to pre-election polling? I’m going to use Peter Ellis’s poll aggregator, to save having to think about it myself. It’s an obviously sensible approach and has done well in the past.

To start with, I do want to point out that the polls got the broad message correct,  in a way that you would probably not be able to do just by listening to the news and doomscrolling on social media.  The polls said that Labour would do much better than in 2017, and that ACT would do much better than in 2017 and that National would do much worse, and that the miscellaneous new parties would go the way of most miscellaneous new parties.  It’s the details that were off.

In the aggregated polls, Labour were expected to get 59 seats, plus or minus about 3, and National about 42 with similar uncertainties.  In reality, Labour are at 64 and National at 35, way out in the tail of the predictions.  Even taking the uncertainty in the model into account, Labour did surprisingly well.   The predictions for  ACT and NZ First, on the other hand, were spot-on, and for the Greens were well within the predicted range.

In Auckland Central I know of two electorate polls, which had  Chlöe Swarbrick at 24% and 26% — the latter taken 24-30 September, very close to the start of voting.  In reality, she got 34.1% and, if history can be trusted, is likely to move up slightly on special votes.  This wasn’t a straightforward swing away from National; the Labour candidate also did better in polls than in the election. It will be interesting to see if much of Swarbrick’s performance can be explained by improved turnout when we have the final numbers.

Single-electorate polling is always hard, and it’s likely to be even harder for an electorate with a young population that includes many students, and for a three-way race. Hard-to-predict electorates, though, are the only ones where polling is interesting.  Given that two polls were both off by 10+% for the winning candidate, single-electorate polling may just not be worth the effort.

By contrast, I don’t think the Waiariki result should really be surprising — the polling was indicating a good chance of one or two electorates for the Māori Party, and I’m told that people in Māori media had discussed it as plausible.

So why were the polls off nationally? It doesn’t help that the number of polling companies has fallen, but that would be incorporated in the model uncertainty, so it doesn’t explain everything.  Some of it is probably just 2020. Because opinion polls get a low response rate — and worse, a response rate that’s lower for some groups of people than others — pollers have to have ways to correct for the bias.  On top of that, they need ways to estimate who will actually vote in the election.  Given the huge change in people’s working patterns this year, and especially during August and September in Auckland, it would not be even slightly surprising if the relationship between political views and responding to an opinion poll had changed.

In the future, as things settle down, polling companies will be able to adapt to the new normal and get more accurate results (or at least better estimates of error). For now, though, polls may be working less well than they have in the past.