Finding out if policies work
The UK Cabinet Office (with the help of Ben Goldacre and David Torgerson) has come out in favour of finding out whether new policies actually work:
‘Test, Learn, Adapt‘ is a paper which the Behavioural Insights Team is publishing in collaboration with Ben Goldacre, author of Bad Science, and David Torgerson, Director of the University of York Trials Unit. The paper argues that Randomised Controlled Trials (RCTs), which are now widely used in medicine, international development, and internet-based businesses, should be used much more extensively in public policy.
As we have pointed out before, lots of people come up with potentially good ideas for dietary interventions, crime prevention, reductions in drug use, and improved education, to name just a few targets. The experience from medical research is that plausible, theoretically-sound, carefully thought-out treatment ideas mostly don’t work. In other fields we don’t know because we haven’t looked.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
It would be great to have similar evaluative frameworks here in New Zealand — quite often evaluations of new (government) initiatives seem like they have been shoe-horned in at a late stage, rather than being fully developed in the planning process.
So this often means that the evaluations are much less rigorous than we’d like them to be… or can’t answer the questions that they really should be trying to answer.
These are allegations that could also be made against a lot of academic research, of course :-)
12 years ago
Looks like a laudable goal. The real difficulty may be in getting politicians to base further policy on the resulting evidence. The badger cull in the UK is a good example of excellent science being employed to measure the effects of proposed policies only to be ignored when a policy was finally decided upon.
12 years ago
As a general principle this is good – but this is worth looking at – http://www.bmj.com/content/327/7429/1459.long (paywalled, but you only need to read the freely available abstract).
A more serious example is from Canada, where they waited for randomised controlled studies on if HIV could be transmitted by transfusion with the result that lots of people unnecessarily contracted HIV.
The point being that the idea that policy should only be made with RCTs to back it is taking things too far.
12 years ago
Yes, that would be taking things too far, but I don’t think anyone is advocating making policy only with RCTs.
The article says RCTs should be used “routinely”, and I think that’s about the right strength of recommendation — randomization should be the default, but not a universal requirement
12 years ago