Thomas Lumley interviewed on Firstline
Auckland University statistician and the most regular contributor to Stats Chat, Professor Thomas Lumley, spoke to Firstline this morning.
Watch the interview or read the interview below:
“Now bear with us, because we’re about to talk statistics, but not the dry mathematics you might expect: it’s all about the art of picking an election winner.
In the US election, many commentators said the race was too close to call but there were some who picked the Obama victory almost perfectly like Nate Silver of the New York Times and Princeton’s Sam Wang. How did they get it right when others got it so wrong. Well for more on this I’m joined by Auckland University statistician Professor Thomas Lumley.
So people like Nate Silver and Sam Wang, were you surprised that they predicted it so accurately?”
Thomas: “No, it’s the sort of thing that you really can predict things accurately. There’s a lot of polling information in the United States, people do a lot of National polls, State polls and so on, and there’s a lot of history about how accurate they are so it is possible to put that information together.”
“So who is Nate Silver?”
Thomas: “He’s currently from the New York Times. He used to a run a separate blog called FiveThirtyEight.com and then got bought by the New York Times essentially. Before that, he was a baseball statistician, he did baseball prediction.
He was extremely close, he was actually lucky as well as smart. He was closer than he could have been just by being right: but he got every single state correct in the electoral college vote and Florida which hasn’t been decided yet, he predicted at almost exactly 50%.”
“So he’s making these correct calls and yet we’re reporting all the time: ‘too close to call’. How can there be such a gulf between what he does and what everyone else is saying?”
Thomas: “Partially, it’s what people aren’t used to what you can do by putting the poll information together. In the old days, people looked at one poll at a time, and there because of the biases in different polls and because each poll is relatively small, maybe 1000 or 5000 people you can’t tell very accurately.
But if you put them all together you can tell quite accurately unless something really novel happens – there’s a big change in who goes out to vote or something. [This is what’s called] meta-polling: putting polls together. People have been picking polls with what they would like to believe and one of the things about Nate Silver is that he’s very good at distinguishing what he wants to be true from what there’s actually data about.”
“George Will of the Washington Post said Romney would win with 320 electoral votes, another one from the New York Post said 325. They’re way off.”
Thomas: “They’re way off and those people are off more than a reliance on a single poll could be. Part of the issue is that one of the things that political journalism is valuable about it is that people talk to inside sources and learn what people inside the parties are saying and cross check it. But it doesn’t actually help in this case because there isn’t any inside information, the parties don’t know any more than the pollers do.”
“Would something similar work for New Zealand elections?”
Thomas: “It would work and there’s a couple of websites which are trying to do it. It wouldn’t work quite as well probably because there are fewer polls in New Zealand. Because New Zealand’s smaller and the polls still have to be 1000 or so people, you can’t afford as many polls in a New Zealand as a US one. There isn’t as much information. It would still work better than a single poll though.
Rachel Cunliffe is the co-director of CensusAtSchool and currently consults for the Department of Statistics. Her interests include statistical literacy, social media and blogging. See all posts by Rachel Cunliffe »
What are the websites that are trying to do it in NZ that you mention in your last answer?
12 years ago
Rob Salmond at Pundit is one (look for Poll of Polls posts). He recently wrote this article on the implications of the work in the US for NZers doing similar things. It includes links to two of the others doing it.
http://www.pundit.co.nz/content/us-election-2012-rise-of-the-nerds
12 years ago
Kiwi Poll Guy is the other one I was thinking of.
12 years ago
Thanks for those. I was hoping that someone somewhere would be talking about individual electorate contests. Although it doesn’t have as much importance in an MMP environment, bloggers and commenters on political blogs like to talk (when a general election is near) about whether some party will win or lose particular electorates, given some level of support for the party in opinion polls. Unless polls were taken in individual electorates here, one is limited to some kind of regression analysis, intuition, or what’s happening on ipredict.
I don’t envisage data mining being used by political parties to the same extent that it was apparently used in the US.
12 years ago
I’m living in the US right now, where Nate Silver is seen as some kind of voodoo priest for getting things so right.
I think the reason why people find all this mystical is because most pundits (and indeed most members of the public) apply the wrong kind of inductive reasoning to polls. Each poll in the US seemed to show a “razor-thin” margin between the two candidates.
Now, in fact, the correct inductive conclusion would be to say that the very next poll we take would have a very close margin.
But people changed the object of the inductive conclusion, and instead of saying something about the next poll, said something about the entire race itself, i.e., if all polls show a razor-thin margin, then the race must be too close to call.
12 years ago
That was a Great TV3 interview by Professor Lumley that concisely advanced the science of statistics to the public
12 years ago
Nice interview. Predicting election outcomes in NZ reminded me about iPredict and that whole “prediction sharemarket” way of doing things. Now being steeped in traditional statistics, it looks “the ultimate free lunch” to me. Is it time to throw out my Kish and my Cochran?
These guys use the low cost option of: knowing absolutely nothing about their self selected sample, no sample frame, no fieldwork, no callbacks, no confidence limits (need I go on?). So can anybody point me to a theory of how/why it works? Proponents claim this technique outperforms others (but they would say that wouldn’t they).
Puzzled of Mt Albert.
12 years ago
There are at least two, and perhaps more, potentially-sane forms of the betting-market hypothesis
A weak form says that if the odds disagree too much with the best publicly available predictions, then people will bet and bring the market back to rationality.
A stronger form says that the existence of a market will improve the best publicly-available predictions because it increases the incentive to resolve disagreements. The Aumann Agreement Theorem shows that rational people who believe in each other’s rationality must be able to come to agreement about probabilities even if they don’t have access to the same raw evidence, but since it takes work they might not do it without a financial incentive.
In practice, the election betting markets not only fail to satisfy the weak form of the hypothesis, they don’t even satisfy the basic sanity constraints of consistency. Eric Crampton describes how he bet on Romney at iPredict and on Obama at Betfair and InTrade to make a guaranteed profit.
12 years ago
but none of this can help without data to work with.
So keep sampling.
12 years ago
It works because people who are willing and able to put the time into the analysis can make money by doing so, and in doing so move the rest of the market so the information is revealed. When last iPredict ran the numbers on accuracy, they were remarkably good. About 40% of trades where people bet $0.40 on that something would happen paid out at $1, about 50% where people bet $0.50 and so on.
Thomas: the problem I was able to exploit I think was due to deposit limits at InTrade. Somebody had put a lot of money into the market at InTrade and was willing to lose it shoring up Romney’s prices; because stupid anti-gambling / anti-money-laundering legislation makes it impossible to get a lot of money into that market in a hurry, the market couldn’t self-correct. I corrected epsilon of it.
Robin Hanson and Ryan Oprea showed in the experimental lab that manipulators of this sort just increase liquidity and increase the incentive for information traders to come in and do well. Unfortunately, when they’re barred from entering the market, they can’t really do that.
12 years ago