Posts filed under Risk (222)

The discussion over the Facebook experiment seems to involve a lot of people being honestly surprised that other people feel differently.

One interesting correlation based on my Twitter feed is that scientists involved in human subjects research were disturbed by the research and those not involved in human subjects research were not. This suggests our indoctrination in research ethics has some impact, but doesn’t answer the question of who is right.

Some links that cover most of the issues

• Michelle Meyer: How an IRB Could Have Legitimately Approved the Facebook Experiment—and Why that May Be a Good Thing
• Coding Coduct: Frame Clashes, or: Why the Facebook Emotion Experiment Stirs Such Emotion
• John Foreman: Facebook’s solution to big data’s “content problem:” dumber users
• Tal Yarkoni: In defense of Facebook
• Zeynep Tufekci: Facebook and Engineering the Public
• Adam Kramer (one of the people who did it)
• (update) Tyler Cowen: Should we care that Facebook is manipulating us

When you read “The university did not reveal how the study was carried out” in a news story about a research article, you’d expect the story to be covering some sort of scandal. Not this time.

The Herald story  is about broccoli and asthma

They say eating up to two cups of lightly steamed broccoli a day can help clear the airways, prevent deterioration in the condition and even reduce or reverse lung damage.

Other vegetables with the same effect include kale, cabbage, brussels sprouts, cauliflower and bok choy.

…

Using broccoli to treat asthma may also help for people who don’t respond to traditional treatment.

‘How the study was carried out’ isn’t just a matter of detail: if they just gave people broccoli, they wouldn’t know what other vegetables had the same effect, so maybe it wasn’t broccoli but some sort of extract? Was it even experimental or just observational? And did they actually test people who don’t respond to traditional treatment? And what exactly does that mean — failing to respond is pretty rare, though failing to get good control of asthma attacks isn’t.

The Daily Mail story was actually more informative (and that’s not a sentence I like to find myself writing). They reported a claim that wasn’t in the press release

The finding due to sulforaphane naturally occurring in broccoli and other cruciferous vegetables, which may help protect against respiratory inflammation that can cause asthma.

Even then, it isn’t clear whether the research really found that sulforaphane was responsible, or whether that’s just their theory about why broccoli is effective. 

My guess is that the point of the press release is the last sentence

Ms Mazarakis will be presenting the research findings at the 2014 Undergraduate Research Conference about Food Safety in Shanghai, China.

That’s a reasonable basis for a press release, and potentially for a story if you’re in Melbourne. The rest isn’t. It’s not science until they tell you what they did.

Via The Atlantic, there’s a new paper in PNAS (open access) that I’m sure is going to be a widely cited example by people teaching research ethics, and not in a good way:

 In an experiment with people who use Facebook, we test whether emotional contagion occurs outside of in-person interaction between individuals by reducing the amount of emotional content in the News Feed. When positive expressions were reduced, people produced fewer positive posts and more negative posts; when negative expressions were reduced, the opposite pattern occurred. These results indicate that emotions expressed by others on Facebook influence our own emotions, constituting experimental evidence for massive-scale contagion via social networks.

More than 650,000 people had their Facebook feeds meddled with in this way, and as that paragraph from the abstract makes clear, it made a difference.

The problem is consent.  There is a clear ethical principle that experiments on humans require consent, except in a few specific situations, and that the consent has to be specific and informed. It’s not that uncommon in psychological experiments for some details of the experiment to be kept hidden to avoid bias, but participants still should be given a clear idea of possible risks and benefits and a general idea of what’s going on. Even in medical research, where clinical trials are comparing two real treatments for which the best choice isn’t known, there are very few exceptions to consent (I’ve written about some of them elsewhere).

The need for consent is especially clear in cases where the research is expected to cause harm. In this example, the Facebook researchers expected in advance that their intervention would have real effects on people’s emotions; that it would do actual harm, even if the harm was (hopefully) minor and transient.

Facebook had its research reviewed by an Institutional Review Board (the US equivalent of our Ethics Committees), and the terms of service say they can use your data for research purposes, so they are probably within the law.  The psychologist who edited the study for PNAS said

“I was concerned,” Fiske told The Atlantic, “until I queried the authors and they said their local institutional review board had approved it—and apparently on the grounds that Facebook apparently manipulates people’s News Feeds all the time.”

Fiske added that she didn’t want the “the originality of the research” to be lost, but called the experiment “an open ethical question.”

To me, the only open ethical question is whether people believed their agreement to the Facebook Terms of Service allowed this sort of thing. This could be settled empirically, by a suitably-designed survey. I’m betting the answer is “No.” Or, quite likely, “Hell, no!”.

[Update: Story in the Herald]

If you find that changing your diet in some way makes you feel happier and healthier, that’s a good thing.  It doesn’t matter whether the same change would be useful for most people, or only useful for you. It doesn’t matter whether the change is a placebo effect. It doesn’t even matter if it’s an illusion, a combination of regression to the mean and confirmation bias. You might check with a doctor or dietician as to whether the change is dangerous, but otherwise, go for it.

If you want to campaign for the entire community to make a change in their diet, you need to have evidence that it’s better on average for the entire community. A few people’s subjective experience isn’t good enough.  Good quality observational data might be all you can manage if the benefits are subtle or take years to appear, but if you’re claiming dramatic short-term benefits you should be able to demonstrate them in a randomised controlled trial.

The reason for mentioning this is that PETA has been making friends again. They’re trying to link milk consumption to autism. They don’t even pretend to have any evidence that milk causes autism, and the evidence that milk-free diet has a beneficial effect in people with autism is very weak.  That is, there are a few studies that suggest a benefit, but the benefit is smaller in studies with more reliable designs, and absent in the best-designed studies.  The most recent review of the evidence concluded that dairy-free or gluten-free diets should only be tried for people who have some separate evidence of food intolerance.  After reading the review, I would agree.

There are respectable arguments against dairy farming, both ethical and environmental. Scaremongering about autism isn’t one of them.

Automated location information can be very useful, but if the ‘location’ is an area and the automated result is a single point, it’s easy to get misled.

• I wrote about an interactive map of vaccine-preventable disease outbreaks, and the pertussis epidemic it maps as being in the  Ngaanyatjarra lands in Australia’s Western Desert.
• Mapping mentions of ‘Nigeria’ to the centre of the country was one of the problems of the 538 piece on kidnappings last week
• An analysis relating right-wing politics to porn viewing, among its other issues, had a lot of users mapped to Kansas, because that’s where the centre of the USA is.
• Wayne Dobson, who lives in Las Vegas, doesn’t have your cell phone. His house is where a lot of ‘uncertain location’ phone positions get mapped to.

From Cathy O’Neil: Ignore data, focus on power (and, well, most of the stuff on her blog)

From danah boyd and Kate Crawford: Critical Questions for Big Data

A new map to let you see the impact of rises in sea levels on your area: this is Auckland with 13m sea rise

This doesn’t show the impact of storm surges, which are the big problem for a lot of eastern coastal Auckland (though not so much for Manukau Harbour).

(via everyone on twitter)

The Daily Mail (and the Telegraph) recently ran puff pieces advertising a new ‘drinkable sunscreen’

Those who make the product, which is available to buy, claim that once the elixir is ingested, molecules of the product vibrate on the skin to cancel out 97 per cent of UVA and UVB rays.

Actually, the Telegraph headline was “Could this be the first drinkable sunscreen?”, and, following Betteridge’s Law, the answer is “Fsck, no!” A blog at the Guardian has a good demolition of the stories, and the illuminating byline

Dean Burnett is acutely aware of how many ways he could be sued if he wrote his initial reactions to the articles about this sunscreen. He is on Twitter, @garwboy

British libel law has been modified after the notorious attack on Simon Singh, but it’s not surprising that no-one wants to be a test case for the new law.

After seeing the reaction, the Mail has at least modified their article to add a concluding warning

Hermoine Lawson, spokesman at the British Skin Foundation, said: ‘We would advise extreme caution of any product claiming UV protection using methods not supported by clinical research. 

‘When it comes to an issue as serious as preventing skin cancer, customer testimonials cannot take the place of scientific evidence, for which this particular product cannot provide.

The Telegraph has a story about the implausibility of the claim and the lack of supporting evidence, though it carefully doesn’t mention how you might have come to hear about the claim, and there certainly isn’t an apology or a retraction.

That’s all on the other side of the world.  Here, where we have one of the highest rates of melanoma, you’d expect a bit more of a clue about sunscreen. But, no, One News has the story. They don’t have the testimonial from someone who used it on a toddler, but they do have the inventor claiming it is effective for more than 99% of people.

Although One News just describes them as a US skincare company, they have a New Zealand distributor with a New Zealand company registration and .co.nz website, and a whole bunch of ‘harmonized water’ products with superficially implausible claims, and they do include the ‘sunscreen’ and the claims:

•Neutralizes UV radiation 
•Allows for increased sun exposure (30x more than normal) 
•Enhances tanning effect from the sun 

Perhaps someone might like to ask the Advertising Standards Authority to verify that the claims are indeed valid and substantiated?

[Update: 538 now has an intro to the story explaining the mistakes and apologising. Good for them.]

So, at  fivethirtyeight.com there’s this story on mapping kidnappings in Nigeria using data from GDELT, the sort of thing data journalism is supposed to be good at. GDELT automatically extracts information from news stories to build a huge global database.

On Twitter, Erin Simpson, whose about.me page says she is “a leading specialist in the intersection of intelligence, data analysis, irregular warfare, and illicit systems – with an emphasis on novel research designs,” — and who has worked on the GDELT parser — is Not Happy.

Thanks to Storify, here are three summaries of what she says, but a lot of it can be boiled down to one point:

In conclusion: VALIDATE YOUR FREAKING DATA. It’s not true just because it’s on a goddamn map.

(via @LewSOS)

From Stuff

The road toll has moved into triple figures for 2014 following the deadliest April in four years.

Police are alarmed by the rising number of deaths, that are a setback after the progress in 2013 when 254 people died in crashes in the whole year – the lowest annual total since 1950.

So far this year 102 people have died on the roads, 15 more than at the same point in 2013, Assistant Commissioner Road Policing Dave Cliff said today.

The problem with this sort of story is how it omits the role of random variation — bad luck.  The Police are well aware that driving mistakes usually do not lead to crashes, and that the ones which do are substantially a matter of luck, because that’s key to their distracted driver campaign. As I wrote recently, their figures on the risks from distracted driving are taken from a large US study which grouped together a small number of actual crashes with a lot of incidents of risky driving that had no real consequence.

The importance of bad luck in turning bad driving into disaster means that the road toll will vary a lot. The margin of error around a count of 102 is about +/- 20, so it’s not clear we’re seeing more than misfortune in the change.  This is especially true because last year was the best on record, ever. We almost certainly had good luck last year, so the fact that it’s wearing off a bit doesn’t mean there has been a real change in driver behaviour.

It was a terrible, horrible, no good, very bad month on the roads, but some months are like that. Even in New Zealand.