Do we trust the police?

Cameron Slater (and the Police, and the Police Association) are Outraged about a Horizon poll on public perceptions of the police.  They do have some fair points, although a few deep breaths wouldn’t hurt.

Horizon Research conducted a poll shortly after an article in the Dominion Post.  This, in itself, is one of the claims against the company, but I don’t think this one really holds water — if you’re a public opinion firm wanting coverage, trying to capitalise on well-publicised issues seems fair enough, and it’s not as if we in the blogosphere are on the moral high ground here.

The results of the poll are clearly not relevant to the question of police conduct in the particular incident described in the Dominion Post article, since if the poll has even the slightest pretension to being representative, none of the respondents will know anything about that incident beyond what they might have read in the papers.  Also, since this is a one-off poll, the Dominion Post’s headline “Trust in police hits new low survey shows” cannot possibly be justified.  There is no comparison with a series of similar surveys in the past; respondents were just asked whether their trust in the police had changed over time.

Many of the reported findings of the poll seem reasonable, and in line with other evidence, for example: 73% have the same or more trust in the police than five years ago, and the police are thought to do well on their primary areas of responsibility

The police, in their press release, apparently as further argument against the poll results,  point out that crime rates are falling.  That’s not really relevant.  Even if the crime rates were falling specifically because of police actions (which is unproved, since there are falls in other countries too), it would only prove that people should think the police are effective in stopping crime, not that they do think the police are fair in investigating complaints.

The controversial claims are about victims of police misconduct:  firstly, that sizable majorities think the investigation procedure needs to be more independent, and secondly, that people who identify themselves as victims are not happy with how their cases were handled.  Again, this doesn’t seem all that strange: I basically trust the police, but I’m still in favour of having investigations done independently just from the ‘lead us not into temptation’ principle.  It’s not news that NZ, by international standards, gives people fairly low levels of compensation for all sorts of things. And it’s hardly surprising that people who think they were mistreated by police aren’t happy with how they were treated by police.

The problem is with how the poll was conducted, and there are at least two pieces of evidence that it wasn’t done well. The first is in the poll results themselves.  The number of New Zealanders who file complaints against the police is very small.  We looked at this back when the issue of complaints against teachers came up.  In a one-year period there were 2052 complaints, half of them minor ‘Category 5’ complaints.  That’s one Category 1-4 complaint per 4500 Kiwis per year (and if some people make multiple complaints, the effective number is even smaller).   In a representative sample of 756 people there shouldn’t have been enough people with experience of police complaints to get useful estimates of anything, let alone to the quoted three decimal places.   Even if we ignored bias and just worried about sampling variation, it would be a serious fault that no margin of error or sample size is quoted for these subgroups.

The second problem is that a Facebook page associated with the incident gave a bounty (chance of winning money and iPad) to people who clicked through to the online poll.   There are now comments on that page saying that not very many people did click through and that they might not have been counted anyway because they wouldn’t have been registered far enough in advance.  That reminds me of this XKCD cartoon. If you have a poll where it’s even conceivable that it could be biased this way, it’s not much consolation to know that the only known attempt to do it was a failure.