Posts filed under Surveys (188)

Or perhaps not. It’s hard to tell.

The Herald’s headline is “Kiwi workers say ‘no’ to more migrants”, with the reported data apparently being on ethnic diversity in the workplace, rather than migration

• 27% want more
• 33% want less
• 40% not sure

Now, the difference between 27% and 33% is smaller than the margin of sampling error based on 200 NZ respondents (and much smaller than the usual ‘maximum margin of error’ calculation), but that’s not the main issue.

Another problem is that “More migrants” is not the same as “more ethnic diversity”, and it’s certainly not the same as “more non-English-speaking background”, which the story also mentions.  (I’m a migrant, I’m from an English-speaking background, and I don’t think preferring Aussie Rules to rugby is the sort of ethnic diversity they had in mind).

More important, though, is the question of whether this is a real survey or a bogus poll.   The story doesn’t say.  If you ask the Google, it points you to a webpage where you can participate in the survey,

In order to continue to provide the most current insights into our modern workplace we need your valuable input.

which is certainly an indicator of bogosity.

On the other hand, Leadership Management Australasia, who run the survey, also give some summary reports.  One report says

The survey design and implementation is overseen by an experienced, independent research practitioner and the systems and process used to conduct the survey ensure valid, reliable and representative samples.

which seems to argue for a real survey (though not as convincingly as if they’d actually named the independent research practitioner).   So perhaps the self-selected part of the sample isn’t all of it, and perhaps they do some sensible reweighting?

If you look at the demographic profile of the survey, though, at least two-thirds of the participants are male, even at the non-managerial level.  Now, in both NZ and Australia, male employment is higher than female, but it’s not twice as high.  The gender profiles are definitely not representative.  So even if the survey is making some efforts to be a representative sample, it isn’t succeeding.

[Updated to add: in case it’s not clear, in the last paragraph, I’m talking about the summary report for second quarter 2011]

In yesterday’s Herald story about alcohol consumption the superficial flaws hid a much more important inaccuracy.

The survey says:

“Of the 113,345,000 glasses of alcohol consumed in New Zealand between February last year and January this year, 28 per cent were drunk by men older than 50,”

Now, Stats New Zealand reports alcohol sales each year, and they say we bought 300 million litres of beer, 100 million litres of wine, and 70 million litres of spirits last year, totalling 33 million litres of pure alcohol.  To make the numbers add up, there would need to be about 290ml of pure alcohol (over 24 standard drinks) in each glass. At most two standard drinks per glass might be realistic as an average, so that’s off by a factor of more than 10  (in a statistic given to six significant figures).

Some of the alcohol would have been consumed by people under 18, who weren’t in the survey, but if they drink 90% of the alcohol sold in NZ we aren’t panicking nearly enough.  Some of it would have been sold to foreigners, and some discarded, but again there’s no way that adds up to 90%.

In fact, the number is implausible on its face: there’s 4.5 million people in NZ, and there must be at least 3 million over 18. That would give an average of only three drinks per month for adult Kiwis.

Even other survey data doesn’t agree. As I pointed out last week, the NZ  Alcohol and Drug Use Survey finds 26% of the adult population drinks more than twice per week. Just those people must rack up more than 113 million glasses per year.

Either someone has lost a decimal place somewhere, or survey respondents lie to Roy Morgan even more than they lie to government researchers.

The Herald claims “NZ says no to larger schoolrooms”  based on a street survey of “more than 70” people, of whom 81% were opposed to the changes.   The current clicky poll has 74% of about 8000 responses supporting the much weaker claim ‘Less one-on-one time can’t be good for kids’,  a statement that even John Key would probably not contest. We aren’t told what the actual questions were in the street survey, or how much the respondents knew about what the actual proposed changes were.

There are two ways you can get useful data by interviewing people.  In quantitative research, where you take a proper probability sample and ask questions with simple answers unambiguously related to what you are trying to find out.  The law of large numbers then ensures that your sample results are not too different from the population results.   The Colmar Brunton polls are an example of this. In qualitative research, you are trying to find out the full range of responses and understand people’s thinking: you get smaller numbers of people and ask much more open-ended questions.  Marketing focus groups are an example of this approach.

Just as the clicky website polls are a degraded version of quantitative survey research, the street poll is a degraded form of qualitative survey research.  A good qualitative survey would try to find out how people feel about the tradeoffs in education funding, about where they would rather make cuts, and how views differed between groups of people — do people without children give similar explanations to  those with children, for example.

In this case, however, public opinion is probably clear enough that any measurement will give a similar result, at least as long as you’re not interested in working out what policies would be better.

An interesting report in the Herald yesterday on methamphetamine use.   A forthcoming Health Ministry survey apparently shows a decrease in the proportion who report using P over the past year, from 2.1% in 2007/2008 to 1% now (yes, that will be larger than the margin of error; it’s a big survey).

On the other hand, an ongoing Massey University study that interviews police detainees about drug and alcohol use finds that the street price and availability of P have not really changed.

This evidence of a decrease on the demand side is encouraging for two reasons.  First, it suggests a decrease in the number of people who are dumb enough to want to take methamphetamine, which has to be progress.  Second, it suggests that the pseudoephedrine ban wasn’t responsible, as that should have acted on the supply side by driving up street prices.  Can we have our Sudafed back?

“We’re spending $70 per person to fill this out. That’s just not cost effective,” he continued, “especially since in the end this is not a scientific survey. It’s a random survey.”

That’s US Congressman Daniel Webster, quoted in the New York Times talking about the American Community Survey, the more-detailed part of the US Census that is administered to a 2% sample of Americans each year.  As you know, the fact that the ACS is based on a random sample is what makes it a scientific survey, and it is a very valuable one to US government and business as well as to researchers.

If the NZ census goes to a ten-year rather than five-year cycle, an interim sample similar to the ACS would be one way of maintaining up-to-date and accurate information at reasonable cost.  Or we could just leave information collection to Facebook and Wikipedia.

There were two Stat-of-the-Week nominations (from Patricia de Guzman and Samantha Post) for the Conservative Party’s use of the Durex Sex Survey to oppose providing subsidised contraception.  We’ve seen this survey number before on StatsChat and it was in the NZ media when the figures were released, back in 2007.

What’s really striking about many of  the stories is the focus specifically on promiscuity of women, when the number that the survey was claiming to estimate is the same for heterosexual men and heterosexual women as a simple matter of arithmetic. 

As is so often the case, the political position seems to be developed independently and statistics sought to drape around it.  Or as Andrew Lang put it  they “use statistics as a drunken man uses lamp-posts — for support rather than for illumination.”

Another heart-disease related story in the Herald (the American College of Cardiology is having its annual meeting now) talks about a link between sleep  and heart disease.  The Herald quotes the principal investigator, Rohit Arora

We now have an indication that sleep can impact heart health, and it should be a priority.

Based on these findings, it seems getting six to eight hours of sleep everyday probably confers the least risk for cardiovascular disease over the long term.

The study actually looked backwards in time, asking people how much sleep they (currently) get per night, and whether they have previously been diagnosed with various heart conditions.  So, it really doesn’t say much about the effect of sleep on heart disease.  Other news sources used a different quote from Dr Arora

“We don’t know whether sleeping longer causes heart complications or whether the heart problems cause someone to sleep longer”

Indeed we don’t.

It’s also a bit strange that the study, based on the large US NHANES survey involved only 3019 people.  NHANES examines more than 10000 people in each two-year wave, and even restricting to people over 45 should leave more than 3000 of them.

A story in Stuff about student drinking at Otago illustrates an important problem with surveys.  According to the story, the students drank on average two nights a week, and consumed 7.2 drinks per night of drinking.  That gives an average of about two drinks per day, which is about the same as average for the whole country based on total alcohol sales.

I don’t know about you, but I find this a little hard to believe. The newspapers should find it even harder to believe, since it contradicts their usual line about Otago students.

My guess is that Otago students do drink a bit more than the average Kiwi, but that asking people to report their drinking leads to underreporting compared to looking at total sales.  There could be several reasons.  One is that people might not want to admit how much they drink, another is that there may be difficulties in converting actual glass sizes into standard doses, and a third is that people’s “average week” is usually different from their actual week.  Perhaps you `usually’ drink less, but this week was Sam’s birthday and you drank more.  And last week was the keg party. And before that was the start of semester. And so on.

That’s not the point of the story, though.  The point is that new research finds heavy drinking can make you feel unwell and have difficulty concentrating even the next day;  the effects seem to hang over and affect your daily activities. Perhaps someone could come up with a catchy name for this phenomenon…

Stuff is reporting a survey on attitudes to overseas ownership of NZ farms.   New Zealanders aren’t just opposed to Chinese buyers of the Crafar farms because they’re racist — they say so themselves.

According to the story, the survey asked

“The Chinese company Shanghai Pengxin wants to buy the Crafar farms; do you support or oppose selling the farms to this Chinese company?”

and then

“Do you agree, or disagree, with this statement: ‘I don’t care what the nationality of the company is, I don’t want the farms to be sold to a foreign buyer’.

Not surprisingly, people who had said they didn’t want Shanghai Pengxin as the buyer went on to say that it wasn’t just Chinese buyers they were opposed to.

If you really want to find out whether people feel differently about a Chinese buyer, this isn’t the best way.  You want to ask different samples of people about Chinese buyers and buyers from, say, the UK, or Monaco, or some other country with more NZ farm ownership than China.   That way, people’s natural reluctance to admit to anti-Chinese bias won’t distort the results.

A survey like that would actually be interesting, since it’s not at all clear whether anti-Chinese bias is a big factor or a small one.

Roy Morgan have released their “State of the Nation” report, and the Herald has two stories. In the first

According to the Roy Morgan State of the Nation March 2012 report, there has been an upsurge in the number of New Zealanders who consider Maori culture to be an “essential component” of New Zealand society.

The other story says that most of NZ wealth is now  held by over-55s.

the 55-plus group has increased its share of net wealth from 43 per cent in 2002 to 52 per cent last year.

that is, by about a quarter

Their share of the population also rose in the same period, but only from 19.5 per cent to 24.7 per cent.

or, about a quarter.   Based on these numbers, in 2002, the average net wealth for people over 55 was about 2.2 times higher than the average for everyone over 14, and it’s now about 2.1 times higher.  Since the story goes on to say

The value of their own homes accounted for 56 per cent of their wealth a decade ago, and 70 per cent last year.

presumably the ratio for non-home wealth is even less favorable to the over-55s.