BBC statistics reporting: two cheers
There’s just been a review of the handling of statistics by the BBC, from a review panel chaired by a former UK National Statistician, {Dame,Dr} Jil Matheson. The report was generally favourable, but made a set of recommendations that any organisation communicating or reporting statistics should think about:
The BBC should do much more to contextualise statistics so that audiences understand their significance, particularly when it comes to ‘big numbers’.
The BBC should be better and braver in interpreting statistics for the audience to help them make sense of them. The review found that audiences were “frustrated” with a tendency to simply present different sets of statistics from either side of an argument.
The BBC should do more to go beyond the headlines and investigate figures underlying sources. The Panel warned of the dangers of reporting statistics “straight from a press release.”
More should be done to ensure that all BBC presenters are able to confidently challenge misleading/inaccurate statistical claims made by interviewees.
The BBC should take a more consistent approach to presenting risk, such as in reporting of health-related statistics, although the report considers that the BBC generally performs better than other broadcasters.
The BBC must be clear about significance in reporting statistics – there is a tendency to focus on change, but it is important to explain when a change is not in fact significant, such as in unemployment, GDP or inflation statistics.
More can be done to increase the BBC’s statistical capacity, including how to better identify, use and develop in-house expertise and to ensure non-specialist journalists have access to statistical expertise.
The BBC needs to develop and standardise its guidance for staff on reporting statistics, to ensure that it is applying standards consistently.
(ht @SciComGuy)
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
“The BBC must be clear about significance in reporting statistics – there is a tendency to focus on change, but it is important to explain when a change is not in fact significant, such as in unemployment, GDP or inflation statistics. ”
I wonder if they mean practical significance here.
Relatedly, one thing I’ve noticed about economic and financial news headlines is how often things “jump” and “plummet”. There is a very short term focus. Like the NZ dollar might jump from 70c to 71c in a day, and that is a lot of movement for a day. But if you look over the last 4 years the general trend has been down. If you’re the kind of person for whom daily fluctuations in markets are important, you probably shouldn’t be getting your news from a regular newspaper.
8 years ago
Yes, this. Felix Salmon has suggested the most useful thing newspapers could do for the average investor is to stop reporting daily changes in markets and indexes.
I think there’s also a ‘level of evidence’ issue when reporting low-precision estimates when the underlying process is what’s of interest — for example, when typical monthly differences in road deaths will be on the order of 5-7, you shouldn’t write a story (or a headline) that explicitly or implicitly describes changes of the size as increases or decreases in the underlying risk.
8 years ago