Non-sampled poll reporting

As the Novopay debacle continues, Stuff and the Herald are both reporting a survey done of its members by the Post-Primary Teachers Association. At Stuff, the story begins:

Nearly 36 per cent of secondary school staff are not reporting their Novopay glitches, a survey has found, casting doubt on the Government’s claims of an improvement in the payroll system.

The Post Primary Teachers’ Association found that 38.2 per cent of staff were underpaid, overpaid or not paid at all during the February 20 pay cycle.

That compares with only 1.9 per cent of staff who logged problems with the system, as reported by Novopay Minister Steven Joyce using PricewaterhouseCoopers figures.

In the Herald:

Up to 1600 teachers did not report complaints through official channels over mistakes with their pay administered through the Novopay pay roll system, according to a union survey.

The Post Primary Teachers Association surveyed 4500 teachers for the pay period ending February 20 and found 36 per cent had not formally reported errors with their pay because they were either “too embarrassed” or feared putting school administrators under more pressure.

In this case the PPTA report is easily available (59 page PDF), so we can find out what was actually done.  The union surveyed all its (roughly 18000) members, using an online poll. They received 4659 responses from members, of whom 1712 were affected.

Obviously, teachers who had experienced problems would be more likely to respond, especially if the reason they hadn’t complained to the local administrators was because they didn’t want to put them under more pressure. The PPTA report handles this issue very well. On page 13 they give calculated Novopay error rates under the assumption that 100% of those with problems responded, and under the assumption that the responses are representative. This gives upper and lower bounds, and the lower bound is substantially higher than Novopay is claiming.

In the media stories, things are a bit confused.  The 36% or 38% are proportions assuming the responses were representative. The numbers in the vicinity of 1600 look like the number assuming that everyone adversely affected responds, perhaps minus an estimate of how many of them Novopay reported. I haven’t been able to reconcile them with the PPTA report.  In any case, neither paper accurately described how the data were collected, even though this was made clear by the PPTA.