Posts filed under Polls (132)

A question on my recent post about political opinion polls asks

– at what point does the trend become relevant?

– and how do you calculate the margin of error between two polls?

Those are good questions, and the reply was getting long enough that I decided to promote it to a post of its own. The issue is that proportions will fluctuate up and down slightly from poll to poll even if nothing is changing, and we want to distinguish this from real changes in voter attitudes — otherwise there will be a different finding every month and it will look as if public opinion is bouncing around all over the place.  I don’t think you want to base a headline on a difference that’s much below the margin of error, though reporting the differences is fine if you don’t think people can find the press release on their own.

The (maximum) margin of error, which reputable polls usually quote, gives an estimate of uncertainty that’s designed to be fairly conservative. If the poll is well-designed and well-conducted, the difference between the poll estimate and the truth will be less than the maximum margin of error 95% of the time for true proportions near one-half, and more often than 95% for smaller proportions.  The difference will be less than half the margin of error about two-thirds of the time, so being less conservative doesn’t let you shrink the margin very much.   In this case the difference was well under half the margin of error.  In fact, if there were no changes in public opinion you would still see month-to-month differences this big about half the time.

For trends based on just two polls, the margin of error is larger than for a single poll, because it could happen by chance that one poll was a bit too low and the other was a bit too high: the difference between the two polls can easily be larger than the difference between either poll and the truth.

The best way to overcome the random fluctuations to pick up small trends is to do some sort of averaging of polls, either over time, or over competing polling organisations.  In the US, the website fivethirtyeight.com combines all the published polls to get estimates and probabilities of winning the election, and they do very well in short-term predictions.  Here’s a plot for Australian (2007) elections, by Simon Jackman, of  Stanford, where you can see individual poll results (with large fluctuations) around the average curve (which has much smaller uncertainties).  KiwiPollGuy  has apparently done something similar for NZ elections (though I’d be happier if their identity or their methodology was public).

So, how are these numbers computed?  If the poll was a uniform random sample of N people, and the true proportion was P, the margin of error would be 2 * square root(P*(1-P)/N).  The problem then is that we don’t know P — that’s why we’re doing the poll. The maximum margin of error takes P=0.5, which gives the largest margin of error, and one that’s pretty reasonable for a range of P from, say, 15% to 85%. The formula then simplifies to 1/square root of N.   If N is 1000, that’s 3.16%, for N=948 as in the previous post, it is 3.24%.

Why is it  2 * square root(P*(1-P)/N)?  Well, that takes more maths than I’m willing to type in this format so I’m just going to mutter “Bernoulli” at you and refer you to Wikipedia.

For trends based on two polls, as opposed to single polls, it turns out that the squared uncertainties add, so the square of the margin of error for the difference is twice the square of the margin of error for a single poll.  Converting back to actual percentages, that means the margin of error for a difference based on two polls is 1.4 times large than for a single poll.

In reality, the margins of error computed this way are an underestimate, because of non-response and other imperfections in the sampling, but they don’t do too badly.

According to the Herald

The latest Roy Morgan Poll shows support for the National Party has fallen two per cent since early June.

 The poll is based on 948 people, so the maximum margin of error (which is a good approximation for numbers near 50%) is about 3.2%, and the margin of error for a change between two polls is about 1.4 times larger: 4.6%.

One of the difficulties with surveys on sensitive questions is that people may respond just out of emotional affiliation or based on slogans, rather than actually reflecting carefully on their beliefs.   That’s the positive interpretation of opinion poll results collected in February by a Greek public opinion firm, who appear to be respectable apart from their horrible taste in graphs (via, and)

Among the questions was a section where respondents were asked whether they Strongly Disagree, Disagree, Agree, or Strongly Agree with statements (presumably in Greek and translated in the report)

• Greece should claim by any means from Germany war reparations/indemnities
• Greece should claim by any means from Germany the payment of ‘war loans’ granted to the German Occupation Forces during the period 1941-1944
• Germany, with its current policy, attempts to dominate Europe through its financial power
• They have right all those who argue that Germany’s current policy aims at the establishment of a Fourth Reich.

It’s surprising that someone would ask these questions, but it’s even more surprising that the proportion agreeing was 91%, 87%, 81%, and 77% respectively, and mostly in the ‘Strongly Agree’ category.

The Herald had a  clicky poll on the proposed R-plate restricted-driving legislation, and has reported the results in an unobjectionable paragraph at the end of a good story getting input from experts.  I’m not sure that the poll actually adds to the story, but at least it doesn’t subtract from it.

You can report a self-selected sample responsibly: the Otago Daily Times shows how. Their lead:

Ninety percent of people who have made submissions on the Queenstown Lakes District Council representation review want to keep an Arrowtown ward and councillor.

In the Stat of the Week competition, Nick Iversen points to a poll with a margin of error given as 3.9% from a sample of 1004, where the simple formula based on sample size would give 3.1%, and writes

The US poll uses 3.9%. I wonder if they are accounting for sampling bias as well as sampling error. Sounds to me like the US pollsters are doing a more sophisticated measure of error than NZ ones.

Following the link to the pollers methodology, we find that weighting is responsible.  The poll answers are reweighted to known population totals (eg, from the census) for age, sex,  and race/ethnicity, and also reweighted based on phone numbers per household.  They say

“Once weights are final, the effective, or weighted, sample size – rather than the actual
sample size – is used to compute the survey’s margin of sampling error.”

So, the 3.9% maximum margin of error doesn’t account for sampling bias in any direct way (which would be hard to impossible), but it corrects for the reweighting that is done to reduce sampling bias.  The more variable the resulting weights, the lower the effective sample size and the higher the maximum margin of error.

There’s some discussion on Simon Jackman’s blog (Australian political scientist now at Stanford) saying that serious US polling organisations usually do adjust the margin of error because of the reweighting, but that Australian ones don’t seem to. It looks like NZ ones don’t either.

In the past, newspapers would usually at least refer to their bogus polls of convenience samples as “unscientific”.  Now the warning word simply seems to be “online”.  The Herald’s headline “Mother’s Day off the cards for most: poll” is followed by the lead “Almost half of New Zealanders won’t be celebrating Mother’s Day today, according to a New Zealand Herald online reader poll”.

If the poll were meaningful, it would also matter that the question is badly designed: there are four answers, of which you have to pick exactly one to vote.  The options are a card, a meal, breakfast in bed, or nothing.  If you’re taking Mum skydiving, or sending flowers, or buying her chocolate, or a new chainsaw, how are you supposed to respond?

Stuff is reporting a mind-boggling survey result

Nearly 15 per cent of people worldwide believe the world will end during their lifetime and 10 per cent think the Mayan calendar could signify it will happen in 2012, according to a new poll.

The obvious expectation is that this is a Bogus Poll and that nothing of the sort is true.  However, as the story says, this is a poll conducted for Reuters  by Ipsos, and the findings as given by Ipsos are just as Stuff reports them.

Presumably Ipsos, who know how to poll, are accurately reporting what people said, and the 15% and 10% figures are actually reasonably representative of what people will answer if you ask them that question.  That doesn’t mean the conclusions are true — they obviously aren’t true, since if 15% of people really believed that, they would be behaving differently.

This is a big problem with surveys — even representative polls of people can give weird results, because polls measure how people answer questions, not what they actually believe.

One of my favorite examples is a poll (via) conducted shortly after the massive  Gulf of Mexico oil spill, which found that  “28 percent of Republicans said the recent oil spill in the Gulf of Mexico made them more likely to support drilling off the coast“. This just makes no sense: you could rationally believe that the oil spill is just part of the costs of economic growth and that it doesn’t change your opinion, but supporting drilling more, because of a drilling accident that turned out to be worse and harder to fix than expected, is insane.

From a talk I’m giving tomorrow to the Canterbury Mathematical Association:

“Bogus polls look a bit like real surveys on the outside, but don’t have any of the inner machinery that you need to make them work.”