Stats Chat

1News had a new electoral opinion poll out this week, headlined Poll: National, ACT maintain wafer-thin advantage. The report gave National and ACT 61 seats out of 120. However, if you put the reported percentages and assumptions into the Electoral Commission’s seat allocation calculator, as various people did, you would get a different result, with National and ACT holding 60 seats, and a potential Labour/Green/Te Pāti Māori alliance also holding 60.  That’s strange, but you wouldn’t really expect 1News to get this wrong (unless you were the sort of social media commenter who immediately assumed it was a conspiracy rather than an error).

The first thing to check in a situation like this is the full report from the polling company. The percentages match what 1News reported, and if you scroll down to the bottom, there’s an corresponding seat allocation that matches the one reported.  At this point a simple error looks even less likely.  So what did happen?

Looking at either the 1News report or the Verian report, we see that percentages were rounded to the nearest percentage point, or to the nearest tenth of a percentage point below 4.85%. So, can we construct a set of numbers that would round to the the same reported percentages and match the reported seat allocation? Yes, easily. We need National and Labour to have been rounded down and the Greens to have been rounded up.  So there’s no evidence of a mistake, and rounding is easily the most plausible explanation.

On the other hand, one might hope 1News or Verian would notice that the headline figures are very close to even, and consider how sensitive the results might be to rounding.

On the other other hand, though, what this shows is that a single poll is never going to be enough to support a claim of “maintain wafer-thin advantage”.  The left-right split can easily be off by five seats, and that’s a big difference; if the advantage is wafer-thin, it’s way below the ability of the polling system to measure.  You can do somewhat better by combining polls and estimating biases of different pollers, as the Herald’s poll of polls is doing, using a fairly sophisticated model.  They have been fairly consistent in predicting that (assuming no major events that change things) Labour/Green are unlikely to get a majority without Te Pāti Māori, but that National/ACT have reasonable odds of doing so.

Even then, you can’t really do ‘wafer-thin’ comparisons.

• Substack counts as the media, right?  David Farrier, a NZ journalist and filmmaker, wrote about getting a spine MRI.
• Aspartame is now in Group IIb on the International Agency for Cancer Research scale of hazards.  We had reports from at least the Herald, Stuff, 1News, RNZ.  It’s important to remember that IIb, “possible carcinogen” is effectively the lowest on a three-point scale. IARC has  Group I (definitely carcinogenic at some dose),  Group IIb (probably carcinogenic at some dose), and Group IIb (possibly carcinogenic at some dose). They also have Group III (insufficient evidence). They once had Group IV (not carcinogenic) but it only ever got used once and was retired.  The “at some dose” proviso is also important; for example, sunlight is a Group I carcinogen.  The reports were all pretty good — much better than when bacon got into Group I several years ago.  Perhaps the best was at Stuff, where they actually quoted the recommended dose limit: David Spiegelhalter, an emeritus statistics professor at Cambridge University, said the guidance means that “average people are safe to drink up to 14 cans of diet drink a day … and even this ‘acceptable daily limit’ has a large built-in safety factor.” That’s safe for cancer, not for all possible problems, but it’s still a lot.  I’d also link to the Twitter thread by Martyn Plummer, a statistician and former IARC researcher, but linking to Twitter threads now doesn’t work because of the War on Twitter.
• Katie Kenny at Stuff had an excellent story about measurement accuracy and her infant son’s weight.
• Mediawatch writes about one weird trick for getting your press releases covered in the news (be sure always to call it research)

Team Ratings for Round 21

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 148 matches played, 91 of which were correctly predicted, a success rate of 61.5%.
Here are the predictions for last week’s games.

Predictions for Round 21

Here are the predictions for Round 21. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 20

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 141 matches played, 84 of which were correctly predicted, a success rate of 59.6%.
Here are the predictions for last week’s games.

Predictions for Round 20

Here are the predictions for Round 20. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 19

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 136 matches played, 82 of which were correctly predicted, a success rate of 60.3%.
Here are the predictions for last week’s games.

Predictions for Round 19

Here are the predictions for Round 19. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 18

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 128 matches played, 74 of which were correctly predicted, a success rate of 57.8%.
Here are the predictions for last week’s games.

Predictions for Round 18

Here are the predictions for Round 18. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 17

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 121 matches played, 71 of which were correctly predicted, a success rate of 58.7%.
Here are the predictions for last week’s games.

Predictions for Round 17

Here are the predictions for Round 17. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for the Super Rugby Final

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 90 matches played, 70 of which were correctly predicted, a success rate of 77.8%.
Here are the predictions for last week’s games.

Predictions for the Super Rugby Final

Here are the predictions for the Super Rugby Final. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for the Currie Cup Final

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 58 matches played, 33 of which were correctly predicted, a success rate of 56.9%.
Here are the predictions for last week’s games.

Predictions for the Currie Cup Final

Here are the predictions for the Currie Cup Final. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

When I wrote about ChatGPT-detecting last time, I said that overall accuracy wasn’t enough; you’d need to know about the accuracy in relevant groups of people:

In addition to knowing the overall false positive rate, we’d want to know the false positive rate for important groups of students.  Does using a translator app on text you wrote in another language make you more likely to get flagged? Using a grammar checker? Speaking Kiwi? Are people who use semicolons safe?

Some evidence is accumulating.   The automated detectors can tell opinion pieces published in Science from ChatGPT imitations  (of limited practical use, since Science has an actual list of things it has published).

More importantly, there’s a new preprint that claims the detectors do extremely poorly on material written by non-native English speakers. Specifically, on essays from the TOEFL English exam, the false positive rate averaged over several detectors was over 50%.  The preprint also claims that ChatGPT could be used to edit the TOEFL essays (“Enhance the word choices to sound more like that of a native speaker”) or its own creations (“Elevate the provided text by employing advanced technical language”) to reduce detection.

False positives for non-native speakers are an urgent problem with using the detectors in education. Non-native speakers may already fall under more suspicion, so false positives for them are even more of a problem. However, it’s quite possible that future versions of the detectors can reduce this specific bias (and it will  be important to verify this).

The ability to get around the detectors by editing is a longer-term problem.  If you have a publicly-available detector and the ability to modify your text, you can make changes until the detector no longer reports a problem.   There’s fundamentally no real way around this, and if the process can be automated it will be even easier.  Having a detector that isn’t available to students would remove the ability to edit — but “this program we won’t let you see says you cheated” shouldn’t be an acceptable solution either.