For small values of ‘infinitely’
Q: Did you see “women are infinitely more likely than men to suffer from virtually every form of headache there is”?
A: What would that even mean?
Q: Well, it says “statistically’, so you should know.
A: No.
Q: Could it be like ‘literally’ and it just means ‘a lot’. Like ‘five times as much’ or something?
A: I suppose, but it still isn’t true.
Q: What does the story say?
A: “a review of 24 worldwide studies published in the journal Headache in 2011 found that while more than half of women polled (52 per cent) reported having a problem with headaches at the time of the research, only 37 per cent of men did.”
Q: And is that right?
A: My English teachers would say they’re missing commas after “studies” and “2011”, but I might not have the high ground in that kind of debate.
Q: Restricting to statistical pedantry, is it right?
A: Well, the first step would be to find the paper, which they don’t make all that easy.
Q: You mean they don’t link?
A: That’s part of it, as Mark Hanna eloquently complained on Twitter. But they also don’t give the author names or the month.
Q: Did you find it?
A: Yes.
Q: And?
A: Here’s the relevant graph
Q: Wait, what? More than 80% of men and women in North America have headaches right now? But only about 50% of people in total?
A: Looking up the reference that little 1 points to, it seems ‘current’ headache really means ‘in the past three months’, or even ‘in the past year’, depending on the study.
Q: And the total being much less than either men or women separately?
A: ¯\_(ツ)_/¯ If I had to guess, maybe the studies that separated out men and women used a longer time period
Q: And is the time period why there are more headaches in North America than Europe?
A: Could be. Or the quality of the cheese. Or the fact that they’re in an election campaign, like, half the time.
Q: I’m sensing you don’t like this graph.
A: There are others
Q: Right. 7.4% in women is less than 6.4% in men. ಠ_ಠ
A: Actually, the lines are ok, it’s just that the numbers are in the wrong places. If they’d written the numbers on the y-axis like we’ve been doing for centuries, they’d be ok.
Q: Ok. But we nearly digress. You’re saying ‘infinitely more” means something in the range 1.5 times to three or four times more?
A: And that some of the comparisons are a bit dodgy.
Q: It seems to have taken more work than necessary to establish that.
A: It sure has.
Q: Would you like to quote some of Mark Hanna’s tweets on linking?
A: “Listen up, reporters & editors. Every day I see you publish articles using “studies say” that make it hard or impossible to find the studies. You should be improving public understanding of science, but instead you are training your audience to believe in “studies say”.
Yes, do write about research. But let your readers read it too. Being able to criticise research is, erm, critical to scientific literacy. You are reinforcing the perception that science is opaque, impenetrable, and not for the eyes of laypeople. But that’s not true at all.
Worse than that, by training your readers to believe in “studies say” you are priming them to be fooled by pseudoscience.”
Q: Yes, that’ll preach.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »