Three quarters or 3%?

This Herald story improves a lot after the first few paragraphs. But it would have to.

Almost three-quarters of the depressed New Zealanders who have gone to Sir John Kirwan’s website depression.org.nz are no longer depressed after finishing the six lessons the site offers.

The story goes on to say that an evaluation of the website found that only 3% of the first 13000 people who registered ended up finishing the six lessons.

Public health officials say the result makes the website, and the $5 million-a-year advertising campaign around it, one of the New Zealand’s most successful public health campaigns.

Based on the 3% figure applied to the current 40000 registrants that doesn’t sound at all plausible: the relevant figure would have to be the 26% who completed at least two lessons, 48% of whom were no longer depressed. That comes to about 5000 people out of 40000. They estimate that 10-15% would have recovered without any intervention, or about 5000 out of 40000.  So, any evidence of public-health benefit needs some information about how representative the 26% were: if they included everyone who was going to recover anyway, there’s no benefit; if they were completely representative, there’s a big benefit. Both extremes are pretty unlikely, but we can’t tell any more from the information in the story.

The evaluation report may have more detail that really does show this website has been effective, but the report doesn’t seem to be public (yet).

[Update: I hadn’t noticed that this had also been nominated for Stat of the Week, with much the same arguments.]