One weird trick increases donating tenfold?

From the Herald:

US researchers have confirmed a strange link between touching rough surfaces and feeling for others, which could help charities raise more money.

Based on my usual complaints about this sort of claim, you might expect that the research didn’t look at donating money  or that it saw only a tiny difference. No.

There were five experiments, but only one that involved actual money. People were approached on the street and given a description of  a health-related charity, and asked to donate. One charity was real, working in a familiar disease; the other was fake, working in a real but obscure disease (all the money actually ended up with the real charity).  Half the participants were given the information and donation envelope on a clipboard with rough sandpaper on the back; the other half weren’t.

When asked to donate to the National Breast Cancer Foundation there was no difference between the rough and smooth clipboards (as you’d expect). When asked to donate to the National Sjögren’s Foundation, 10/34 with sandpaper-backed clipboards said yes compared to only 1/32 with smooth clipboards.

I’m going to go very slightly out on a limb here to say there is no way this ten-fold increase is a real and generalisable phenomenon.  So, what went wrong?  Part of the problem is what Andrew Gelman calls the ‘garden of forking paths’, after the Jorge Luis Borges story — there are many, many possible analyses and they don’t all show this dramatic difference.

For example, there wasn’t a difference in donation probability with the familiar charity. This was consistent with the researchers’ theory, but I’m pretty sure if there had been a difference the researchers wouldn’t have considered it as evidence refuting the theory. Also, the researchers note that they didn’t see a difference in donation amount with the sandpaper, just in donation probability.

Also, if you assume the ten-fold increase was overestimated even a bit, you then get into the problem of sample size. Suppose that the effect was only a two-fold increase rather than ten-fold. That still seems implausibly large to me, but the comparison would then be something like 2/34 vs 1/32 and would be completely unimpressive.  You’d need a sample size something like ten times larger.  And that’s if a bit of sandpaper on the back of a clipboard doubled the number of people who donated.

Still, these findings could have “significant implications for less well-known charities”, as the researchers suggest. If I got approached by a charity using sandpaper on the back of their clipboards, I would tend to think they were (a) poor at evaluating evidence, and (b) not all that honest. I could see that having an impact.