Stats Chat at national journalism teachers’ conference

Polls and surveys making various claims land on journalists’ desks every day – but not all of them are newsworthy (and the bogus, PR-driven ones that make it into print are subsequently shredded in this very forum).

The statschat.org.nz team is always keen to help people understand the difference between a reliable poll and something that should be filed in the bin. So we’re delighted that two members of the statschat.org.nz team, Andrew Balemi, a polls and surveys expert from the Department of Statistics at The University of Auckland, and adviser Julie Middleton have been given an hour at this Wednesday’s Journalists Training Organisation/Journalism Education Association conference to talk about polls and surveys.

They’re not going to each anyone to crunch numbers. What’s far more important is knowing the right questions to ask about a poll or survey to determine whether it should be taken seriously.

This is the hand-out we are providing – we have only an hour, so the list of questions isn’t complete, but it gives you an idea of how we encourage journalists to think.

Questions a reporter should ask of a poll or survey

Why is the poll/survey being done?
What do the pollsters want to find out?

Who is doing the survey?
Who paid for the survey?
Who carried out the work?
Is the person/company that carried out the survey a member of the Market Research Society of New Zealand? (ie, is it subject to a code of ethics?)

What we’re looking for: Evidence of lobbying, misinformation, public-relations or marketing spin … or just a company hoping to get editorial when it should buy an ad.

How representative is the sample?
How were those to be polled/surveyed chosen?
From what area (nation, state, or region) or group (teachers, National voters etc) were these people chosen?
How were the interviews conducted? (Internet survey, face-to-face, by phone)…

What we’re looking for: Evidence that the poll/survey is based on a genuinely representative random sample and has been conducted according to sound statistical principles. Be wary if you can’t get the original research questionnaire, raw data and a full explanation of methods.

If possible, ask about the broader picture
Does this study reflect the findings of other polls/surveys on this topic?

What we’re looking for: Evidence of similar findings elsewhere. 

Is this poll/survey worth reporting?
If you get positive responses to the above, yes. If not, this question becomes a philosophical one: Do you ignore accuracy for a sexy subject? Or run a story based on a bogus survey with a long list of caveats?

Don’t be afraid to ask professional statisticians for advice and help. They will generally be flattered – and pleased that you are taking such care.