Knowing what you’re predicting: drug war edition
The woman was evicted by Housing New Zealand months ago after “methamphetamine contamination” was detected at her home. The story says it’s “unclear” whether the contamination happened during her tenancy or is the fault of a previous tenant.
There’s no allegation of a meth lab being run; the claim is that methamphetamine contamination is the result of someone smoking meth in the house.
The vendors claim the technique has no false positives, but even if we assume they are right about this they mean no false positives in the assay sense; that there definitely is methamphetamine in the sample. The assay doesn’t guarantee that the tenant ‘allowed’ meth to be smoked in her house. And in this case it doesn’t even seem to guarantee that the contamination happened during her tenancy.
It’s not just this case and this assay, though those are bad enough. If predictive models are going to be used more widely in New Zealand social policy, it’s important that the evaluation of accuracy for those models is broader than just ‘assay error’, and considers the consequences in actual use.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
I also wonder if there is partly a prosecutors fallacy here. That is, in all cases where we know there was meth usage we find traces of meth in the house therefore if we find traces of meth in the house you must have used meth.
Reminds of when they started tested for cocaine in banknotes in Florida for people they arrested until someone tested banknotes for the general population and found a lot of their banknotes had cocaine traces.
9 years ago
I’ve looked in the past and haven’t been able to find good public data on the specificity of the test.
It’s conceivable that the specificity is so high that the prosecutor’s fallacy isn’t a problem — that there’s basic no chance of even one house in NZ returning a positive result if meth hadn’t been smoked in it.
However, if that’s true it must be at some cost in sensitivity — so there must be some risk that a house would falsely test negative (or inconclusive) before you moved it and then positive at a later time without any additional contamination.
9 years ago
With the banknotes and cocaine, the theory is that some of the sorting machines in banks got contaminated, rather than some general effluvium of cocaine in modern life.
9 years ago
A situation similar to above a British bus driver was sacked after a saliva test came back positive for cocaine. He won a payout after employment court accepted that the large amounts of cash he handled on the day of the test could have caused the test result as he claimed he never used drugs of any kind. He had been a driver for 22 years.
It could also be possible that the testing company encourages high number test positives because that increases business. Who knows what their testing and analysis procedures are like.
9 years ago