Measuring accuracy
Stuff has a story “Scanner that can detect brain bleeds to be introduced in New Zealand.” The scanner uses infrared light to see relatively small accumulations of blood in the brain, with the aim of detecting bleeding quickly. The traditional approach of looking at symptoms can often miss a bleed until after it’s done a lot of damage.
Accuracy is important for a device like this. You don’t want to send lots of people off for CT scans, which are expensive and expose the patient to radiation, but you also don’t want to falsely reassure someone who really has a brain injury and who might then ignore later symptoms.
The story at Stuff claims 94% accuracy, but doesn’t say exactly what they mean by ‘accuracy’. Another story, at Gizmodo, says “A green light on the scanner gives the patient the all clear, and a red light shows a 90 per cent chance of haemorrhage.” The Gizmodo figures fit better with what’s on the manufacturer’s website, where they claim “Sensitivity = 88% / Specificity = 90.7%”. That is, of people with (the right sort of) bleed, 88% will be detected, and of people without those bleeds, 90.7% will be cleared.
The Gizmodo story still confuses the forward and backwards probabilities. Out of 100 people with brain bleeds, 88 will get a red light on the machine. That’s not the same as their claim: that out of 100 people who get a red light on the machine, 90 have a bleed.
Suppose about 10% of the people it’s used on really do have brain bleeds. Out of an average 100 uses there would be 10 actual bleeds, 9 of whom would get a red light. There would be 90 without actual bleeds, about 9 of whom would get a red light. So the red light would only indicate about a 50% chance of a haemorrhage. That’s still pretty good, especially as it can be done rapidly and safely, but it’s not 90%.
The other aspect of the story that’s not clear until you read the whole thing is what the news actually is. Based on the headline, you might think the point of the story is that someone’s started using this device in NZ, maybe in rugby or in ambulances, or is trialling it, or has at least ordered it. But no.
No-one in New Zealand has yet got their hands on an infrascanner, but the hope is for it to be rolled out among major sporting bodies, public and private ambulance services, trauma centres and remote healthcare facilities.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
Dear Prof. Lumey, thank you for the observation and the analysis. Unfortunately you can’t always control what reporters write, and you are correct in your observation. You are correct that the PPV (positive predictive value), which takes into consideration disease prevalence is around 50%. However the key parameter is NPV (the negative predictive value) that is about 99%, as bleeds are relatively rare, so the 91% specificity translates to a very high NPV.
8 years ago
That’s a good point. The negative predictive value is actually important in this setting — especially for use in ambulances and so on.
For sports, the positive predictive value is still relevant, because the standard of care would be to wait for symptoms (except in the most severe cases). When the scanner is beneficial it will be because definitive scans are done earlier and more often than they would be without the scanner.
8 years ago
Absolutely true. Another aspect of the data that the journalists missed is that what the neurological examination and the Infrascanner are detecting are different aspects of brain trauma, and hence are synergistic. In the pivotal study of the Infrascanner, PPV of the neurological examination was 46% as compared to CT, while Infrascanner had 63% PPV. However, if the Infrascanner was used in addition to the Neurological examination, the combined test yielded PPV of 82%, with NPV staying high. The Infrascanner greatly enhances triage capabilities of medics.
8 years ago
When you talk about ‘the journalists’ having missed this information, I think it’s also important to remember that the information source for the story was the Australia & NZ distributor of the device. He apparently didn’t provide this information, and it’s quite a lot of work to find it even for someone who knows how the medical literature works.
8 years ago