Posts filed under Risk (222)

The Royal Statistical Society has just commissioned an opinion poll to look at beliefs about policy-relevant issues and how they relate to reality.  The first few results:

1. Teenage pregnancy: on average, we think teenage pregnancy is 25 times higher than official estimates: we think that 15% of girls under16 get pregnant each year, when official figures suggest it is around 0.6%.
2. Crime: 58% do not believe that crime is falling, when the Crime Survey for England and Wales shows that incidents of crime were 19% lower in 2012 than in 2006/07 and 53% lower than in 1995. 51% think violent crime is rising, when it has fallen from almost 2.5 million incidents in 2006/07 to under 2 million in 2012.
3. Job-seekers allowance: 29% of people think we spend more on JSA than pensions, when in fact we spend 15 times more on pensions (£4.9bn vs £74.2bn).
4. Benefit fraud: people estimate that 34 times more benefit money is claimed fraudulently than official estimates: the public think that £24 out of every £100 spent on benefits is claimed fraudulently, compared with official estimates of £0.70 per £100.

You might look up what the actual figures are in New Zealand. To get you started, Paula Bennett claimed about $200 million in benefit fraud in 2010/11 (and about 10% of that was prosecuted) from about $9 billion, or about 22c per $100.

Via Edward Tufte on Twitter, from an Indiana newspaper

(more…)

Some folks at the MIT Media Lab have put together a simple web app that takes your Gmail headers and builds a social network.

Once you log in, Immersion will use only the From, To, Cc and Timestamp fields of the emails in the account you are signing in with. It will not access the subject or the body content of any of your emails.

Here’s mine, from my University of Washington email (with the names blurred, not that communicating with me is all that incriminating)

Obviously my email headers reveal who I email, and, unsurprisingly, the little outlying clusters are small groups or individuals involved in specific projects.  More interesting is how the main clump breaks down:  the blue and pink circles are statisticians, the red are epidemiology and genomics people that I have worked with in person in Seattle, and the green are epidemiology and genomics people that I work with only via email.

The New York Times has a detailed story about one of the triggers for the Alltrials campaign, the missing studies of Tamiflu

He was curious about one of the main studies on which Dr. Jefferson had relied in his previous analysis. Called the Kaiser study, it pooled the results of 10 clinical trials. But Dr. Hayashi noticed that the results of only two of those trials had been fully published in medical journals. Given that details of eight trials were unknown, how could the researchers be certain of their conclusion that Tamiflu reduced risk of complications from flu?

Only about half of all randomized clinical trials are published, despite regulations requiring publication, and the requirements of the Declaration of Helsinki

Authors have a duty to make publicly available the results of their research on human subjects and are accountable for the completeness and accuracy of their reports.

Since the obvious conclusion is that the unpublished studies are less favorable than the published ones, patients and the medical community cannot be sure about the benefits of even the most promising treatments.  The uncertainty always matters at least to a small group of patients, but in the case of Tamiflu it matters to the whole world. The 2009-2010 influenza pandemic was relatively minor, but still killed more than 250000 people worldwide (by most estimates, more than the Iraq war). The 1918 pandemic was at least twenty times worse. Before it happens again, we need to know which treatments work and which do not work.

Stuff has a (mostly reasonable) story on alcohol and cancer, quoting Prof Doug Sellman

“The ethanol in alcohol is a group one carcinogen, like asbestos,” 

Many of the readers of this story won’t know what a “group one carcinogen” is.  Given the context, a reader might well assume that “group one carcinogens” are those that carry the largest risks of cancer, or cause the most serious cancers. In fact, all it means is that an additional hazard of cancer, whether high or low, has been definitely established, because that’s all the IARC review process tries to do. The Preamble to the IARC Monographs that define these carcinogens says

A cancer ‘hazard’ is an agent that is capable of causing cancer under some circumstances, while a cancer ‘risk’ is an estimate of the carcinogenic effects expected from exposure to a cancer hazard. The Monographs are an exercise in evaluating cancer hazards, despite the historical presence of the word ‘risks’ in the title. The distinction between hazard and risk is important, and the Monographs identify cancer hazards even when risks are very low at current exposure levels, because new uses or unforeseen exposures could engender risks that are significantly higher.

Group 1 carcinogens tend to either be very common exposures or to cause very specific types of cancer, because those are the two scenarios that make it easy to establish definitely that there is a risk.  They include asbestos, arsenic, sunlight, birth control pills, plutonium, diesel exhaust, and wood dust.

Some group 1 carcinogens, such as tobacco and hepatitis B, are responsible for large numbers of cancer deaths worldwide. Others, such as plutonium and diethylstilbestrol, are responsible only for small numbers of deaths. Some group 1 carcinogens cause aggressive, untreatable tumours; for others, such as human papillomavirus, disease is largely preventable by screening; still others, such as sunlight, sometimes cause serious disease but mostly cause relatively minor tumours.

The phrase “group one carcinogen” is only relevant in an argument over whether the risk is zero or non-zero. Its use in other contexts suggests that someone doesn’t know what it means, or perhaps hopes that you don’t.

Sense about Science (a British charity whose name, unusually, is actually accurate) have just launched a publication “Making Sense of Uncertainty”, following their previous guides for the public and journalists that cover screening, medical tests, chemical stories, statistics, and radiation.

Researchers in climate science, disease modelling, epidemiology, weather forecasting and natural hazard prediction say that we should be relieved when scientists describe the uncertainties in their work. It doesn’t necessarily mean that we cannot make decisions – we might well have ‘operational knowledge’ – but it does mean that there is greater confidence about what is known and unknown.
Launching a guide to Making Sense of Uncertainty at the World Conference of Science Journalists today, researchers working in some of the most significant, cutting edge fields say that if policy makers and the public are discouraged by the existence of uncertainty, we miss out on important discussions about the development of new drugs, taking action to mitigate the impact of natural hazards, how to respond to the changing climate and to pandemic threats.
Interrogated with the question ‘But are you certain?’, they say, they have ended up sounding defensive or as though their results are not meaningful. Instead we need to embrace uncertainty, especially when trying to understand more about complex systems, and ask about operational knowledge: ‘What do we need to know to make a decision? And do we know it?’ 

From the World Conference of Science Journalists, via @roobina (Ruth Francis), ten tweets on reporting clinical trials

1. Was this #trial registered before it began? If not then check for rigged design, or hidden negative results on similar trials.
   
2. Is primary outcome reported in paper the same as primary outcome spec in protocol? If no report maybe deeply flawed.
3. Look for other trials by co or group, or on treatment, on registries to see if it represents cherry picked finding
4. ALWAYS mention who funded the trial. Do any of ethics committee people have some interest with the funding company
5. Will country where work is done benefit? Will drug be available at lower cost? Is disorder or disease a problem there
6. How many patients were on the trial, and how many were in each arm?
7. What was being compared (drug vs placebo? Drug vs standard care? Drug with no control arm?
8. Be precise about people/patient who benefited – advanced disease, a particular form of a disease?
9. Report natural frequencies: “13 people per 10000 experienced x”, rather than “1.3% of people experienced x”
10. NO relative risks. Paint findings clearly: improved survival by 3%: BAD. Ppl lived 2 months longer on average: GOOD

Who says you can’t say anything useful in 140 characters?

Kieran Healy uses data from the time of the American Revolution to show how membership of organisations can be turned into social network information

Rest assured that we only collected metadata on these people, and no actual conversations were recorded or meetings transcribed. All I know is whether someone was a member of an organization or not. Surely this is but a small encroachment on the freedom of the Crown’s subjects. I have been asked, on the basis of this poor information, to present some names for our field agents in the Colonies to work with. It seems an unlikely task.

If you want to follow along yourself, there is a secret repository containing the data and the appropriate commands for your portable analytical engine.

You may well already have seen this, but I’ve been travelling.

In the Herald, in late May, there was a commentary on the importance of freeing-up the GCSB to do more surveillance. Aaron Lim wrote

The recent bombings at the Boston Marathon are a vivid example of the fragmented nature of modern warfare, and changes to the GCSB legislation are a necessary safeguard against a similar incident in New Zealand.

 …

Ceding a measure of privacy to our intelligence agencies is a small price to pay for safe-guarding the country against a low-probability but high-impact domestic incident.

Unfortunately for him, it took only a couple of weeks for this to be proved wrong: in the US, vastly more information was being routinely collected, and it did nothing to prevent the Boston bombing.  Why not?  The NSA and FBI have huge resources and talented and dedicated staff, and have managed to hook into a vast array of internet sites. Why couldn’t they stop the Tsarnaevs, or the Undabomber, or other threats?

The statistical problem is that terrorism is very rare.  The IRD can catch tax evaders, because their accounts look like the accounts of many known tax evaders, and because even a moderate rate of detection will help deter evasion.  The banks can catch credit-card fraud, because the patterns of card use look like the patterns of card use in many known fraud cases, and because even a moderate rate of detection will help deter fraud.  Doctors can predict heart disease, because the patterns of risk factors and biochemical meausurements match those of many known heart attacks, and because even a moderate level of accuracy allows for useful gains in public health.

The NSA just doesn’t have that large a sample of terrorists to work with.  As the FBI pointed out after the Boston bombing, lots of people don’t like the United States, and there’s nothing illegal about that.  Very few of them end up attempting to kill lots of people, and it is so rare that there aren’t good patterns to match against.   It’s quite likely that the NSA can do some useful things with the information, but it clearly can’t stop `low-probability, high-impact domestic incidents’, because it doesn’t.  The GCSB is even more limited, because it’s unlikely to be able to convince major US internet firms to hand over data or the private keys needed to break https security.

Aaron Lim’s piece ended with the typical surveillance cliche

And if you have nothing to hide from the GCSB, then you have nothing to fear

Computer security expert Bruce Schneier has written about this one extensively, so I’ll just add that if you believe that, you can easily deduce Kristofferson’s Corollary

Freedom’s just another word for nothing left to lose.

We keep going on here about the importance of publishing clinical trials.  Today (in Britain), the BBC program Panorama is showing a documentary about a doctor who has been running clinical trials of the same basic treatment regimen for twenty years, without publishing any results. And it’s not that these are trials that take a long time to run — the participants have advanced cancer. If the treatment was effective, it would have been easy to gather and publish convincing evidence by now, many times over.

These haven’t been especially good clinical trials by usual standards — not randomized, not controlled — and they have been anomalous in other ways as well. For example, patients participating in the trial are charged large sums of money for the treatment being tested (not just for other care), which is very unusual.  Unusual, but not illegal.  Without published evidence that the treatment works, it couldn’t be sold outside trials, but it’s still entirely legal to charge money for the treatment in research. It’s a bit like whaling.

According to the BBC, Dr Burzynski says it’s not his decision to keep the results secret

He said the medical authorities in the US would not let him release this information: “Clinical trials, phase two clinical trials, were completed just a few months ago. I cannot release this information to you at this moment.”

If true, that would be very unusual. I don’t know of any occasion when the FDA has restricted scientific publication of trial results, and it’s entirely routine to publish results for treatments that have not been approved or even where other research is still ongoing. The BBC also checked with the FDA:

But the FDA told us this was not true and he was allowed to share the results of his trials.

This is all a long way away from New Zealand, and we can’t even watch the documentary, so why am I mentioning it? Last year, the parents of an NZ kid were trying to raise money to send him to the Burzynski clinic, with the help of the Herald.   You can’t fault the parents for trying to buy hope at any cost, but you sure can fault the people selling it.

Wikipedia has pretty good coverage  if you want more detail.