What happens when statistical accuracy is sacrificed for speed – a cautionary tale

Nielsen’s CMI parade rained on as ‘unprecedented’ fieldwork issues affect print readership data

From Stop Press:

The stock imagery on the release might show people laughing with magazines, but there probably aren’t too many smiles in the print industry after several unexpected fieldwork issues affected the quality of readership data for Nielsen’s newly pimped out Consumer and Media Insights readership survey.

Nielsen’s managing director Stuart Jamieson couldn’t be reached for comment, but the main issues impacting on the Q1 2011 survey, according to a letter sent to publishers and agencies, were the suspension of fieldwork in Christchurch and surrounding areas following the earthquake that resulted in a shortfall of interviews, best practice methods not being followed by all interviewers that led to an exclusion of some data and staff shortages in Auckland.

“Nielsen and the survey auditor, Professor Peter Danaher believe that the unprecedented effect of these fieldwork issues has had a negative impact on readership results for the quarter,” Jamieson said in the letter. “Not all issues with Q1 fieldwork can be resolved immediately. The survey auditor has requested that missing interviews in both Christchurch and other Regions must be replaced. We will be working with the Auditor with a managed programme to replace surveys over the remaining survey periods of 2011.”

After the Print Media Industry Research Review Group signed up Nielsen as its preferred research partner after a big pitch, the new survey was released with great fanfare a few months back and offered plenty of promise for the print sector. Fusing Statistics New Zealand’s Household Expenditure data to the survey meant publishers would be able to approach advertisers with, for example, specific data about how much readers or demographic groups spent in certain categories. It’s very valuable and helpful information, and while this data is thought to be robust and unaffected by the fieldwork issues, it’s disappointing for all concerned that its accuracy may be called into question.

Some might wonder why this data would even be released when doubts will inevitably surface about its accuracy. But, given the excitement over the new fusion, it needed to get that important piece of information out to the market and releasing the weighted data was seen as a slightly better option than not releasing any data at all; the lesser of two evils.

The usual flurry of press releases from the magazine publishers trumpeting their various gains hasn’t been forthcoming this year, due to a self-imposed moratorium on self-congratulation because of the fieldwork issues. But the biggest winners when comparing Q1 2010 with Q1 2011 are, once again, niche, special interest magazine titles, although some of the mainstream titles recorded what Nielsen refers to as ‘statistically significant readership gains’.

Continue reading at Stop Press…