Evaluating policy changes

When government policies change in different ways in different parts of the country, you can try to estimate the impact of the changes by comparing trends in affected and unaffected areas.  This week I’ve seen two examples, one in the NZ media and one from a Twitter request.

Chris Knox, in the Herald, has both an interactive graph and a story about the changes in petrol prices, using data from Gaspy. There was a clear jump of 11.5c/litre when the 11.5c/litre tax was increased.  What’s not so clear is whether petrol companies have subsequently tried to spread the price rise to other parts of the country.  Averaged over the whole rest of the country the gap has narrowed from the 11.5c start, but that’s driven by the South Island.  In the Waikato and Bay of Plenty regions, which were previously the most similar to Auckland, there’s no sign of price spreading.

So, right now, it’s unclear whether Auckland is bearing the whole cost (as it should) or just most of it.  The Herald story makes this fairly clear, quoting the economist Sam Warburton, who has been looking at the data — but also quoting someone who doesn’t need data to be sure that the costs are being borne by the regions. A similar analysis was done by Peter Ellis, using Sam Warburton’s data from pricewatch.co.nz, with basically similar conclusions.

The other example from this week is a research paper looking at how fatal workplace injuries changed in US states that introduced medical cannabis laws. They found a decrease starting at the time of legislation and slowly increasing to about a 1/3rd reduction after five years, compared to states without legal medical cannabis.  The analysis looks sensible and the data on fatal workplace injuries are of high quality, but there’s still a bit of potential concern about publication bias — the researchers might have published a negative result, but I wouldn’t have heard about it and told you.  I’ve written before about an analysis that suggested banning single-use plastic bags caused deaths from food poisoning —  and while the analysis looked reasonable it was clear based on additional data that the conclusion wasn’t plausible.

On the cannabis issue what’s most unclear is the contribution of various possible reasons to the reduction (if it is real):

• medical cannabis works to treat, say, pain or nausea, so people are healthier and better rested and don’t get injured
• medical cannabis substitutes for legal or illegal opioid use, and while it might increase injury risk compared to nothing, it’s better than opioids
• medical cannabis ends up in recreational use and substitutes for alcohol, and while it might increase injury risk compared to nothing, it’s better than alcohol
• medical cannabis ends up in recreational use and substitutes for alcohol, and people are less likely to use at work.
• it actually makes you safer

To the extent that the second reason is important in the US, where there’s an opioid use epidemic, it won’t generalise to New Zealand. The other reasons would probably generalise.