How much evidence would you expect to see?

[UPDATE: I got the calculations wrong, and was too kind to the MoT and the paper. The time to get good evidence is more like 20 years.]

The Sunday Star-Times has a story saying that the reduction in speed-limit tolerance, which is now on all the time in the summer, hasn’t yet shown evidence of a reduction in road deaths. And they’re right, it hasn’t, despite some special pleading from the Police Minister and Assistant Commissioner David Cliff. We’ve looked at this issue before.

However, it’s also important to ask whether we’d expect to see evidence yet if the policy really worked as promised. The Star-Times goes on to say:

While MOT data shows just 13 per cent of fatal crashes were attributable to speed Land Transport Safety Authority spokesman Andy Knackstedt said there was “a wealth of evidence” that showed even very small reductions in speed led to reductions in fatalities and serious injuries, and that lowering the enforcement tolerance meant lower mean speeds.

So, we should ask whether we’d expect a clear and convincing drop in road deaths if the theory behind the policy was sound. And we wouldn’t.

Let’s see what we should expect if the policy prevented 10% or 20% of the deaths from speeding, which works out to 1.3% or 2.6% of all road deaths.  The number of deaths last year was 286, over 366 days. Under the simplest model for road crash data, a Poisson process that basically assumes different roads and time periods are independent, we can work out how long you’d have to wait to have a 50% chance or an 80% chance of the reduction getting below the margin of error

For a 20% reduction in deaths from speeding it takes about 190 days to have an even-money chance of seeing evidence, and about 390 days to have an 80% chance. For a 10% reduction in deaths from speeding it takes about 380 days for an even-money chance of convincing evidence and 760 days for an 80% chance. Under more complex and realistic models for road deaths it would take longer.

There’s no way that we could expect to see convincing evidence yet, and given the much larger unexplained fall in road deaths in recent years, it will probably never be feasible to evaluate the policy just based on road death counts.  We’re going to have to rely on other evidence: overseas data, information on reductions in speeding, data from non-fatal or even non-injury crashes, and other messier sources.