Posts filed under Forensic Science (12)

Rasmus Bååth, who is doing a PhD in Cognitive Science, in Sweden, has written a web app that predicts blood alcohol concentrations using reasonably sophisticated equations from the forensic science literature.

The web page gives a picture of the whole BAC curve over time, but requires a lot of detailed inputs. Some of these are things you could know accurately: your height and weight, exactly when you had each drink and what it was. Some of them you have a reasonable idea about: is your stomach empty or full, and therefore is alcohol absorption fast or slow. You also need to specify an alcohol elimination rate, which he says averages 0.018%/hour but could be half or twice that, and you have no real clue.

If you play around with the interactive controls, you can see why the advice given along with the new legal limits is so approximate (as Campbell Live is demonstrating tonight).  Rasmus has all sorts of disclaimers about how you shouldn’t rely on the app, so he’d probably be happier if you don’t do any more than that with it.

1. The Atlantic published some visualisations of taxi rides in New York
2. Chris Whong asked for the data under Freedom-of-Information laws, and got it. Of course, the taxi and driver ids were anonymized
3. Vijay Pandurangan noticed that the driver id and taxi id were really, really weakly anonymised.
4. You can find out a lot once you know the taxi id.

The NY Taxi & Limousine Commission had run the ids through a cryptographic hash function, MD5. Hash functions are designed so that if you don’t know anything about the input you can’t reconstruct it from the output, but if you know the input exactly, you can verify easily that it gives the same output.  The problem comes when you know a lot about the input, but not everything.  In this case, there are only about two million possible id numbers, and you can just try them all. Once you have the ids, you can look up.

Even if the taxi authorities had done the anonymisation correctly — replacing each id with a random number — it would inevitably have been possible to extract some of the ids with a bit of work.  That’s not the same as being able to extract all of them with a few hours’ computer time.

The Herald said, on Friday “Fewer workers stoned on the job”

Information from the New Zealand Drug Detection Agency showed 81,410 on-site drug-screening tests were carried out last year, 16 per cent up from the previous year.

But only 5.5 per cent of tests showed the presence of drugs, down from 6.4 per cent in 2013

As usual, there’s no mention of the fact that NZDDA is just one of the private companies offering drug testing services. It took me a long time to realise this, until I was tipped off by a news story advertising one of their competitors.

Presumably NZDDA don’t think their customers choose them at random, and with no real reason for wanting testing. If customers were behaving even a little rationally you’d expect an expansion of drug testing to pull in lower-risk employees. If we look at the actual number of positive tests, using the quoted figures, it was about 4480 last year and about 4490 in the previous year. Given no change in the number of positive tests and a 1 percentage point change in the proportion of positive tests, from a single company, there’s not a lot of numerical evidence for an increase in number of workers with detectable cannabis in their systems.

More importantly, there’s no evidence whatsoever for the ‘stoned on the job’ headline: absolutely no information is given about this. One of the big problems with cannabis testing is that there is no useful biochemical assay for being stoned. Detectable levels persist long after impairment is over, and even when you’re actually stoned there is not a good relationship between drug concentration and impairment.  This is a real problem for Washington and Colorado, which have legalised cannabis and need to set driving laws. In contrast to alcohol, if you actually care about safe driving and cannabis, it’s really hard to get a useful and objective test.

The story ends with two examples of disasters. In one, cannabis was definitely ruled out as a contributing factor; in the other, the conclusion was only that “it could not be excluded”. The NZDDA  press release is at Scoop, and despite how the story reads, there is surprisingly little text in common.

Prof Peter Donnelly’s TED talk. You might want to skip over the first few minutes of vaguely joke-like objects

Consider the two (coin-tossing) patterns HTH and HTT. Which of the following is true:

1. The average number of tosses until HTH is larger than the average number of tosses until HTT
2. The average number of tosses until HTH is the same as  the average number of tosses until HTT
3. The average number of tosses until HTH is smaller than the average number of tosses until HTT?

Before you answer, you should know that most people, even mathematicians, get this wrong.

Also, as Prof Donnelly doesn’t point out, if you have a programming language handy, you can find out the answer very easily.

Last night’s edition of Our Changing World had two segments by University of Auckland statisticians called James.

Allison Ballance and James Russell talked about seabirds and rat eradication

Ruth Beran and James Curran talked about forensic statistics

Thomas Lumley, tipped off by Luis Apiolaza on Twitter, pointed me to this article in the NZ Herald.

The article is yet another example of the Herald’s inability to correctly report DNA statistics. It makes the following statement:
This article reports a quote from the Crown Prosecutor, paraphrased as follows:

A man charged with raping a woman during a kidnap has pleaded not guilty but Crown says DNA evidence shows the man was “10,000,000,000,000,000 times likely” to be responsible for the crime.

To be fair to the article’s author, this may have been the statement that the Crown prosecutor made, but nNo forensic scientist in New Zealand would say this. ESR scientists are trained to give statements of the following type:

“The evidence is 10¹⁶ (=10,000,000,000,000,000) times more likely if the defendant and the victim were contributors to the stain, rather than the victim and someone unrelated to the defendant.”

It is extremely important to note that This is a statement about the likelihood of the evidence given the hypotheses rather than the other way around. A forensic scientist is bought to court to comment on the strength of the evidence and specifically not on whether the defendant is guilty.

I have commented on this before., and sent correspondence to the NZ Herald numerous times. Perhaps a mention on StatsChat will inspire change.

Update: The NZ Herald reporter, James Ihaka, has contacted me and said “The statement came from a Crown prosecutor about the evidence that the forensic scientist will present later in the trial. Taking in to consideration what you have said however, it would probably be more accurate to rephrase this.” Good on you James!

Update 2: James Ihaka has contacted me again, with the following information:

This is the direct quote from Crown prosecutor Rebecca Mann: ( I checked with her)
“It is ten thousand million million times more likely for the DNA these samples originated from (the complainant) and Mr Martin rather than from (the complainant) and another unrelated individual selected at random from the general New Zealand population.”

I apologize unreservedly for attributing this to James Ihaka, and again congratulate him for following it up.

The statement Ms. Mann should have given is

“The evidence (the DNA match) is ten thousand million million times more likely if these samples originated from (the complainant) and Mr Martin rather than if they originated from (the complainant) and another unrelated individual selected at random from the general New Zealand population.”

Last night, 3News had a scare story about positive drug tests at work.  The web headline is “Report: More NZers working on drugs”, but that’s not what they had information on:

New figures reveal more New Zealanders were caught with drugs in their system at work last year.

…new figures from the New Zealand Drug Detection Agency reveal 4300 people tested positive for drugs at work last year.

but

The New Zealand Drug Detection Agency says employers are doing a better job of self-regulating. The agency performed almost 70,000 tests last year, 30 percent more than in 2011.

If 30% more were tested, you’d expect more to be positive. The story doesn’t say how many tested positive the previous year, but with the help of the Google, I found last year’s press release, which says

8% of men tested “non-negative” compared with 6% of women tested in 2011.

Now, 8% of 70000 is 5600, and even 6% of 70000 is 4200. Given that the majority of the tests are in men, it looks like the proportion testing positive went down this year.

The worst part of the story statistically is when they report changes in proportions of which drug was found as if this was meaningful.  For example,

When it comes to industries, oil and gas had an 18 percent drop in positive tests for methamphetamine, but showed a marked increase in the use of opiates.

That’s an increase in the use of opiates as a proportion of those testing positive.  Since proportions have to add up to 100%, a decrease in the proportion positive tests that are for methamphetamine has to come with an increase in some other set of drugs — just as a matter of arithmetic.

Stuff‘s story from January just as bad, with the lead

Employers are becoming more aware of the dangers of drugs and alcohol in the workplace as well as the benefits of testing for them.

and quoting an employer as saying

“And, we have no fear of an employee turning up to work and operating in an unsafe way, putting themselves and others at risk.”

as if occasional drug tests were the answer to all occupational health and safety problems.

The other interesting thing about the Stuff story is that it’s about a different organisation: Drug Testing Services, not NZ DDA — there’s more than one of them out there! You might easily have thought from the 3News story that the figures they quoted referred to all workplace drug tests in NZ, rather than just those sold by one company.

Given the claims being made, the evidence for either financial or safety benefits is amazingly weak.   No-one in these stories even claims that introducing testing has actually reduced  on-the-job accidents in their company, for example, let alone presents any data.

If you look on PubMed, the database of published medical research, there are lots of papers on new testing methods and reproducibility of test results, and a few that show people who have accidents are more likely than others to test positive.  There’s very little even of before-after comparisons: a Cochrane review on this topic found three before-after comparisons. Two of the three found a small decrease in accident rates immediately after introducing testing; the third did not.  A different two of the three found that the long-term decreasing trend in injuries got faster after introducing testing; again, the third did not.   The review concluded that there was insufficient evidence to recommend for or against testing.

There’s better evidence for mandatory alcohol testing of truck drivers, but since those tests measure current blood alcohol concentrations, not past use, it doesn’t tell us much about other types of drug testing.

There’s been a bunch of publicity recently over claims that Bigfoot really exists and that a group of forensic scientists have the DNA to prove it.

After being rejected from the top journals either because of prejudice and hide-bound conservatism or because of not having any worthwhile evidence, the researchers have managed to publish some results in a peer-reviewed journal. That they set up for the purpose. (unkind scientists on Twitter are making jokes about the next issue, some of which are quite funny)

Ars Technica has the closest to actual information about the paper that I’ve seen, and their analysis sounds right to me. The paper says that the Bigfoot mitochondrial DNA matches humans, so the creature is a hybrid between humans and some unknown primate.  However, the mitochondrial DNA matches are mostly to sequences from Europe and the Middle East, not to Native American sequences, which looks like contamination rather than hybridisation.  Similarly, the results for nuclear DNA should show fairly long sequences matching humans, and other fairly long sequences that look similar to but not identical to other known primates, but they don’t seem to.

The genome data has only been released in PDF format, not in any of the formats that scientists normally use for storing genome sequences. When someone gets around to converting it, and the full surplus power of the world’s sequence matching software is turned loose, the results will be obvious — so the fact this hasn’t happened is not encouraging.

Is this scientific fraud?  Given the real attempts the researchers have made to publish their results, I think we can repeat an answer quoted by physicist Bob Park after the first cold fusion press conference: “Not yet.” And let’s hope it stays that way.

At the Zadar Open chess competition last month, a player who had previously been at the low end of the chess master range did extraordinarily well, playing the level of the world’s very best. Or at the level of a good computer program. There was absolutely no physical evidence to suggest that he had been cheating, but his level of improvement, and the agreement between his moves and those produced by top computer programs are striking.  On the other hand, if you are going to allow accusations in the absence of any corroborating physical evidence, it’s also essentially impossible for an innocent person to mount a defense.

KW Regan, who is a computer scientist and chess master has analysed historical chess competition data, looking at agreement between actual moves and those the computer would recommend, and he claims the Zadar Open results should happen less often than once in a million matches. In his letter to the Association of Chess Professionals, he raises the questions

1.What procedures should be instituted for carrying out statistical tests for cheating with computers at chess and for disseminating their results? Under whose jurisdiction should they be maintained?
2. How should the results of such tests be valued? Under what conditions can they be regarded as primary evidence? What standards should there be for informing different stages of both investigative and judicial processes?

There’s a New York Times story, and Prof Regan also has a blog post. (via)

More than a year ago, ESR scientists produced a report on drugs and alcohol found in blood samples taken after fatal crashes.  Now, the Drug Foundation is launching a publicity campaign using the data.  Their website says “Nearly half of drivers killed on New Zealand roads are impaired by alcohol, other drugs, or both.” But that’s not what the ESR report found. [Edited to add: the Drug Foundation is launching a campaign, but the TV campaign isn’t from them, it’s from NZTA]

The ESR report defined someone as impaired by alcohol if they had blood alcohol greater than 0.03%, and said they tested positive for other drugs if the other drugs were detectable.   If you look at the report in more detail, although 351/1046 drivers had detectable alcohol in their blood, only 191/1046 had more than 0.08%.  At 0.03% blood alcohol concentration there may well be some impairment of driving, and near 0.08% there’s quite a lot, but we can’t attribute all those crashes to alcohol impairment rather than inexperience, fatigue, bad luck, or stupidity.  At least the blood alcohol concentrations are directly relevant to impairment.  An assay for other drugs can be positive long after the actual effect wears off. For example, a single use of cannabis will show up in a blood test for 2-3 days, and regular use for up to a week.  In  fact, the summary of the ESR report specifically warns “Furthermore, it is important to acknowledge that the presence of drugs and alcohol in the study samples does not necessarily infer significant impairment.”   Regular pot smokers who are scrupulously careful not to drive while high would still show up as affected by drugs in the ESR report.  In fact, the Drug Foundation makes this distinction when they talk about random roadside drug testing, pointing out the advantages of a test of actual impairment over a test of any presence of a drug.

The Drug Foundation also did a survey of community attitudes to driving while on drugs (also more than a year ago), and it is interesting how many people think that stimulants and cannabis don’t impair their driving.  However, if you look at the survey, it turns out that it was an online poll, and “Respondents were recruited to the online survey via an advertising and awareness campaign that aimed to stimulate interest and participation in the study.” Not surprisingly, younger people were over-represented “The mean age of respondents was 38.1 years”, as were people from Auckland and Wellington. Maori, Pasifika, and Asians were all under-represented.  36% of respondents had used cannabis in the past year, more than twice the proportion in the Kiwi population as a whole.  No attempt was made to standardise to the whole NZ population, which is the fundamental step in serious attempts at accurate online polling.  [If we could use the data as a teaching example, I’d be happy to do this for them and report whether it makes any difference to the conclusions]

And while it’s just irritating that news websites don’t link to primary sources, it is much less excusable that the Drug Foundation page referencing the two studies doesn’t provide links so you can easily read them. The study reports are much more carefully written and open about the limitations of the research than any of the press releases or front-line website material.[The NZTA referencing is substantially less helpful]

For all I know, the conclusions may be broadly correct. I wouldn’t be at all surprised if many drug users do believe silly things about their level of impairment. Before  the decades of advertising and enforcement, a lot of people believed silly things about the safety of drunk driving.  And the new TV ads are clever, even if they aren’t as good as the ‘ghost chips’ ad.  But the numbers used to advertise the campaign don’t mean what the people providing the money say they mean.  That’s not ok when it’s politicians or multinational companies, and it’s still not ok when the campaigners have good intentions. [Edited to add: I think this last sentence still stands, but should be directed at least equally at the NZTA].

[Update: Media links: TVNZ,  3 News, Stuff, NZ Herald, Radio NZ]