Posts filed under Risk (222)

From BBC Radio: listen online

On Start the Week Emily Maitlis talks to the Executive Chairman of Google, Eric Schmidt about the digital future. A future where everyone is connected, but ideas of privacy, security and community are transformed. Former Wikileaks employee James Ball asks how free we are online. The curator Honor Harger looks to art to understand this new world of technology. And worried about this brave new world? David Spiegelhalter, offers a guide to personal risk and the numbers behind it.

(via @cjbayesian)

Stuff has a headline “Overseas advice: ‘Don’t test party pills on animals’” and lead:

New Zealand will fly in the face of international animal welfare conventions if it allows legal highs to be tested on animals.

However, if you actually read the story, you find that this is not just misleading, but clearly untrue.

They mention responses from three countries. In Canada

… unable to find “any information suggesting” there was a ban on using animals in recreational drug or alcohol research testing in Canada.

In France and the EU

But France recently banned the use of animal testing in the cosmetic industry and policymakers are under pressure to extend that to the legal high, alcohol and tobacco industry.

The European Union has also introduced a ban on the sale of any cosmetic or personal care product which was tested on animals.

and only in the UK was the response consistent with the story

Judy MacArthur Clark, head of the animals in science regulation unit at UK’s Home Office, replied that any proposal to test for the safety of recreational drugs would be “rejected”.

You might also ask how these countries test legal highs for safety, if they don’t use animal testing.  The answer, of course, is that they don’t test legal highs for safety. Not one of these countries has legislation or procedures for approving and regulating new recreational psychoactive substances, but they all do require animal testing for new medications, two facts that Stuff might have considered mentioning.  The reference to LD50 lethal toxicity testing is especially misleading, since the same paper claimed credit in January for the fact that LD50 tests had been ruled out by the Minister.

The ‘legal highs’ regulations are new. No other country has anything similar.  New Zealand can’t base its testing regimes on what other countries are doing, because other countries aren’t doing it.  It would be useful to have some informed public discussion on how testing should be done, since it does involve the sort of tradeoffs where, in a democracy, the public should have input. This sort of story doesn’t help. Quite the reverse, in fact.

[Update: Here are three research papers involving animal (mouse and rat) testing of synthetic cannabis compounds done primarily for regulatory or forensic purposes, in Australia, the US, and Russia]

Two hundred and sixty six years ago today, James Lind began what is regarded as the first proper controlled clinical trial

On the 20th May, 1747, I took twelve patients in the scurvy on board the Salisbury at sea. Their cases were as similar as I could have them. They all in general had putrid gums, the spots and lassitude, with weakness of their knees. They lay together in one place, being a proper apartment for the sick in the fore-hold; and had one diet in common to all, viz., water gruel sweetened with sugar in the morning; fresh mutton broth often times for dinner; at other times puddings, boiled biscuit with sugar etc.; and for supper barley, raisins, rice and currants, sago and wine, or the like. Two of these were ordered each a quart of cyder a day. Two others took twenty five gutts of elixir vitriol three times a day upon an empty stomach, using a gargle strongly acidulated with it for their mouths. Two others took two spoonfuls of vinegar three times a day upon an empty stomach, having their gruels and their other food well acidulated with it, as also the gargle for the mouth. Two of the worst patients, with the tendons in the ham rigid (a symptom none the rest had) were put under a course of sea water. Of this they drank half a pint every day and sometimes more or less as it operated by way of gentle physic. Two others had each two oranges and one lemon given them every day. These they eat with greediness at different times upon an empty stomach. They continued but six days under this course, having consumed the quantity that could be spared. The two remaining patients took the bigness of a nutmeg three times a day of an electuray recommended by an hospital surgeon made of garlic, mustard seed, rad. raphan., balsam of Peru and gum myrrh, using for common drink narley water well acidulated with tamarinds, by a decoction of wich, with the addition of cremor tartar, they were gently purged three or four times during the course.

The consequence was that the most sudden and visible good effects were perceived from the use of the oranges and lemons; one of those who had taken them being at the end of six days fit four duty. The spots were not indeed at that time quite off his body, nor his gums sound; but without any other medicine than a gargarism or elixir of vitriol he became quite healthy before we came into Plymouth, which was on the 16th June. The other was the best recovered of any in his condition, and being now deemed pretty well was appointed nurse to the rest of the sick …

Lind knew very little about scurvy apart from the typical progress of the disease, and he had no real idea of how the treatments might work.  That’s a handicap in coming up with ideas for treatment, but not in doing fair tests of whether treatments work.

The trial didn’t have an untreated group: all the patients got one of the treatments recommended by experts.  There’s no need for a controlled trial to have an untreated group — if there is an existing treatment, you want to compare to that treatment; if there is none, you may want to compare immediate vs delayed treatment.

Despite the dramatic success of fruit juice in the trial, it wasn’t adopted as a treatment. That, sadly, can still be the case today.  New drugs or surgical techniques are taken up enthusiastically, but boring interventions like nurse home visits or surgery checklists get less attention. Still, things are much better than they were even twenty years ago. Nearly all of medicine accepts the idea of randomized controlled comparison, and it is spreading to other areas such as development economics.

There are two excellent, free books about clinical trials and health choices: Testing Treatments, from the James Lind Initiative, and Smart Health Choices, from Les Irwig (at Sydney Uni) and coworkers.

Much of clinical trials development is unapologetically technical, but there are important areas where public participation can help:

• The James Lind Alliance asks patients and clinicians to say what questions matter most.  Clinical trials still tend to answer questions that are scientifically interesting or commercially important, not what actually matters most to patients
• The Cochrane Collaboration is attempting to collect and summarise all randomized trials.  The Cochrane Consumers Network is for non-specialist participation — in particular as Consumer Referees to help ensure that summaries of research address the questions that are important to consumers and are presented in language that consumers can understand.
• Ethics Committees that review clinical trials and other human research in New Zealand are required to have non-specialist community members. This is a substantial commitment, but one that is important if ethics committees are to be more than just red tape.
• And if you haven’t yet signed the AllTrials petition, calling for the results of all clinical trials to be published so we can know what treatments work, now would be an excellent time.

The Herald has a completely over-the-top presentation of what might be an important issue. The headline is “Hospital choice key to kids’ survival”, and the story starts off

Where ambulances take badly injured children first seems to affect their chances, paediatric surgeons say.

Starship children’s hospital surgeons have found that sending badly injured children to the wrong hospital may be contributing to a child death rate from injuries that is twice the rate of Australia’s.

The data:

Six (7 per cent) of the 88 children who went first to Middlemore died, but so did one (8 per cent) of the 12 who went directly to Starship.

That is, to the extent the data tell us anything, the evidence is against the headline.  Of course, the uncertainties are huge: a 95% confidence interval for the relative odds of dying after being sent to Middlemore goes from a 40-fold decrease to a 12-fold increase.  There’s basically no information in the survival data.

So, how much of the two-fold higher rate of death in NZ compared to Australia could reasonably be explained by suboptimal hospital choice? One of the surgeons involved in the study says

… overseas research showed that a good trauma protocol system could cut the death rate for injured adults by 20 to 30 per cent, but there was no good data for children.

That is, hardly any of the difference between NZ and Australia — especially as this specific hospital-choice issue only applies to one sector of one city in New Zealand, with less than 10% of the national population.

On the other hand, we see

The head of Starship’s emergency department, Dr Mike Shepherd, said the major factors contributing to New Zealand’s high fatal injury rate for children lay outside the hospital system in policies such as driver blood-alcohol limits, graduated driver licensing, and laws requiring children’s booster seats and swimming pool fences.

That sounds plausible, but if it’s the whole story you would expect high levels of non-fatal as well as fatal injuries. The overall rate of hospitalisations for injuries in children 0-14 years is almost identical in NZ (1395 per 100 000 per year, p29) and Australia (‘about’ 1500 per 100 000 per year, page v).

• From Felix Salmon: US population is increasing, and people are moving to the cities, so why is (sufficiently fine-scale) population density going down? Because rich people take up more space and fight for stricter zoning.  You’ve heard of NIMBYs, but perhaps not of BANANAs
• From the New York Times.  One of the big credit-rating companies is no longer using debts referred for collection as an indicator, as long as they end up paid.  This isn’t a new spark of moral feeling, it’s just for better prediction.
• And from Felix Salmon again: Firstly, Americans are bad at statistics. When it comes to breast cancer, they massively overestimate the probability that early diagnosis and treatment will lead to a cure, while they also massively underestimate the probability that an undetected cancer will turn out to be harmless.

Stuff has a story  (borrowed from the West Island) headlined “Over 40? Five tests you need right now”.

You might have expected some reference to the other recent news about screening: that the US Preventive Services Taskforce has now joined the Centers for Disease Control in recommending universal screening for HIV (as TVNZ reported).  It’s not clear if New Zealand will follow the trend — HIV infection in people not in high risk groups is less common here than in the US, so the benefit compared to more selective screening is smaller here.   This illustrates the complexities of population screening.  Not only does the test have to be accurate, especially in terms of its false positive rate, but there needs to be something useful you can do about a positive result, and screening everyone has to be better than just screening selected people.

So, let’s  compare the suggestions from Stuff’s story to what national and international expert guidelines say you need.

Two of the tests, for high blood pressure and high cholesterol, are spot on. These are part of the national 2012/13 Health Targets for DHBs, with the goal being 75% of the eligible population having the tests within a five-year period.  The Health Target also includes blood glucose measurement to diagnose diabetes, which the story doesn’t mention.  The US Preventive Services Taskforce also recommends blood pressure and cholesterol tests, though it recommends universal diabetes screening only after age 50 or in people with high blood pressure or people with risk factors for diabetes.

One of the tests recommended in the story is a depression/anxiety questionnaire, for diagnosing suicide risk.  Just a couple of weeks ago, the US Preventive Services Taskforce issued guidelines on universal screening for suicide prevention by GPs, saying that there wasn’t good enough evidence to recommend either for or against. As the coverage from Reuters explains, these questionnaires do probably identify people at higher risk of suicide, but it wasn’t clear how much benefit came from identifying them. So, that’s not an unreasonable test to recommend, but it would have been better to indicate that it was controversial.

One more of the tests isn’t a screening test at all — the story recommends that you make sure you know what a standard drink of alcohol is.

The top recommendation in the story, though, is coronary calcium screening.  The US Preventive Services Taskforce recommends against coronary calcium screening for people at low risk of heart disease and says there isn’t enough evidence to recommending for or against in people at higher risk for other reasons. The  American Heart Association also recommends against routine coronary calcium screening (they say it might be useful as a tiebreaker in people known to be at intermediate risk of heart disease based on other factors).  No-one doubts that calcium in the walls of your coronary arteries is predictive of heart disease, but people with high levels of coronary calcium tend to also be overweight or smokers, or have high blood pressure or high cholesterol or diabetes — and if they don’t have these other risk factors  it’s not clear that anything can be done to help them.

As an afterthought on the coronary calcium screening point, the story has an additional quote from a doctor recommending coronary angiography before starting a serious exercise program.  I’d never heard of coronary angiography as a general screening recommendation — it’s a bit more invasive and higher-risk than most population screening. It turns out that the American College of Cardiology and the American Heart Association are similarly unenthusiastic, with their guidelines on use specifically recommending against angiography for screening of people without symptoms of coronary artery disease.

Dan Kahan, a researcher in the Cultural Cognition project at Yale Law School, has an interesting post on “the science communication problem”

The motivation behind this research has been to understand the science communication problem. The “science communication problem” (as I use this phrase) refers to the failure of valid, compelling, widely available science to quiet public controversy over risk and other policy relevant facts to which it directly speaks. The climate change debate is a conspicuous example, but there are many others

A few years ago, economists Carmen Reinhart and Kenneth Rogoff wrote a paper on national debt, where they found that there wasn’t much relationship to economic growth as long as debt was less than 90% of GDP, but that above this level economic growth was lower.  The paper was widely cited as support for economic strategies of `austerity’.

Some economists at the University of Massachusetts attempted to repeat their analysis, and didn’t get the same result.  Reinhart and Rogoff sent them the data and spreadsheets they had used, and it turns out that the analysis they had done didn’t quite match the description in the paper.  Part of the discrepancy was an error in an Excel formula that accidentally excluded a bunch of countries, but Reinhart and Rogoff also deliberately excluded some countries and times that had high growth and high debt (including Australia and NZ immediately post-WWII), and gave each country the same weight in the analysis regardless of the number of years of data included. (paper — currently slow to load, summary by Mike Konczal)

Some points:

• The ease of making this sort of error in Excel is exactly why a lot of statisticians don’t like Excel (despite its other virtues), so that has received a lot of publicity.
• Reinhart and Rogoff point out that they only claimed to find an association, not a causal relationship, but they certainly knew how the paper was being used, and if they didn’t think provided evidence of a causal relationship they should have said something a lot earlier. (I think Dan Davies on Twitter put it best)
• Chris Clary, who is a PhD student at MIT, points out that the first author (Thomas Herndon) on the paper demonstrating the failure to replicate is also a grad student, and notes that replicating things is job often left to grad students.
• The Reinhart and Rogoff paper wasn’t the primary motivation for, say,  the UK Conservative Party to want to cut taxes and government spending. The Conservatives have always wanted to cut taxes and government spending. Cutting taxes and spending is a significant part of their basic platform. The paper, at most, provided a bit of extra intellectual cover.
• The fact that the researchers handed over their spreadsheet pretty much proves they weren’t deliberately deceptive — but it’s a lot easy to convince yourself to spend a lot of time checking all the details of a calculation when you don’t like the answer than when you do.

Roger Peng, at  Johns Hopkins, has also written about this incident. It would, in various ways, have been tactless for him to point out some relevant history, so I will.

The Johns Hopkins air pollution research group conducted the largest and most comprehensive study of health effects of particulate air pollution, looking at deaths and hospital admissions in the 90 largest US cities.  This was a significant part of the evidence used in setting new, stricter, air pollution standards — an important and politically sensitive topic, though a few order of magnitude less so than austerity economics.  One of Roger’s early jobs at Johns Hopkins was to set up a system that made it easy for anyone to download their data and reproduce or vary their analyses. The size of the data and the complexity of some of the analyses meant just emailing a spreadsheet to people was not even close to acceptable.

Their research group became obsessive (in a good way) about reproducibility long before other researchers in epidemiology.  One likely reason is a traumatic experience in 2002, when they realised that the default settings for the software they were using had led to incorrect results for a lot of their published air pollution time series analyses.  They reported the problem to the EPA and their sponsors, fixed the problem, and reran all the analyses in a couple of weeks; the qualitative conclusions fortunately did not change.  You could make all sorts of comparisons with the economists’ error, but that is left as an exercise for the reader.

• Martin Johnston in the Herald on a nationwide blood pressure survey. Blood pressures are up.  This is not good.
• Nikki Macdonald in Stuff, on recreational genotyping (or as the story more properly calls it, “direct to consumer” genotyping).

Today we have two examples of the important issue of communicating scientific knowledge about infectious disease epidemics.

The first is the WHO, which is doing an excellent job of describing the limited information about the new H7N9 influenza outbreak in China. Their media release is here, and they’ve had someone answering questions on Twitter as well as more traditional venues.  There’s currently evidence of a small amount of human-to-human transmission, but not enough to sustain a pandemic. On the other hand, the virus does appear to have mutated to live more successfully in people, and this could continue.  They don’t advise actually doing anything specific at the moment.

The second is the UK measles epidemic, where The Independent, has as its top front-page headline “MMR scare doctor: this outbreak proves I was right”.  Of course,  it does nothing of the sort, as the story admits later . He’s claiming that the MMR vaccine should have been replaced by three single vaccines, and even if you believe that anti-vaccination campaigners would then suddenly have stopped their misrepresentations, having three single vaccinations is actually more dangerous than one combined one.

The Independent is maintaining that its story is accurate if you read the whole thing. Even if that were so, it’s still hard to imagine why the opinion of a discredited researcher and struck-off former doctor is the single most important piece of information they have about epidemic and the world today. And, as  Martin Robins writes in New Statesman

 It would be a great example of the false balance inherent in ‘he-said, she-said’ reporting, except that it isn’t even balanced – Laurance provides a generous abundance of space for Wakefield to get his claims and conspiracy theories across, and appends a brief response from a real scientist at the end.