June 25, 2014

Not even wrong

By

The Readers’ Digest “Most Trusted” lists are out again. Sigh.

Before we get to the actual complaint in Stat-of-the-Week recommendation, we should acknowledge that there’s no way the “most trusted” list could make sense.

Firstly, ‘trusted’ requires more detail. What is it that we’re trusting these people with? Of course, it wouldn’t help making the question more specific, since people will still answer on some vague ‘niceness’ scale anyway: we saw this problem with a Herald poll at the beginning of the year, which asked opinions about five notable people and found the only one notable for his commitment to animal safety had the lowest rating for “who would you trust to feed your cat?”. Secondly, there’s no useful way to get an accurate rating of dozens of people (or other items) in an opinion poll. People’s brains overload. Thirdly, even if you could get a rating from each respondent, the overall ranking will be sensitive to how you combine the individual ratings.

So how does Readers’ Digest do it? They say (shouting in the original)

READER’S DIGEST COMMISSIONED CATALYST CONSULTANCY & RESEARCH TO POLL A REPRESENTATIVE SAMPLE OF NEW ZEALANDERS ABOUT TRUSTED PEOPLE AND PROFESSIONS. A TOTAL OF 603 ADULTS RANKED 100 WELL-KNOWN PEOPLE AND 50 JOB TYPES ON A SCALE OF ONE TO TEN IN MARCH 2014.

That is, the list is determined in advance, and the polling just addresses the ordering on the list. There is some vague sense in which Willie Apiata is the most trusted person,  or at least the most highly-regarded person, or at least the most highly-regarded famous person, in New Zealand but there really isn’t any useful sense in which Hone Harawira is the least trusted person in New Zealand. There are many people in NZ who you’d expect to be less trusted than Mr Harawira; they didn’t get put on the list, and the survey respondents weren’t asked about them.

It’s not surprising that stories keep coming out about this list, and I suppose it’s not surprising that people try to interpret being on the bottom of the list. Perhaps more surprising, no-one has yet complained that there are actually 101 well-known people, not 100, on the list.

avatar

Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »

Comments

  • avatar
    Nick Iversen

    On the other hand the list of professions does try to be reasonably comprehensive and so being at the bottom of that means something. So I think it’s reasonable to say that we trust telemarketers and politicians the least – the bottom of the list consists of people with something to sell. A theme of “serving and protecting” better than selling.

    So we can say, perhaps, that the fact that journalists are lumped in the bottom end shows that we regard them as people who have something to sell rather than to serve. Or maybe we think they are liars.

    10 years ago