What’s brown and sticky?
Q: What’s brown and sticky?
A: A stick!
Q: What do you call a cow on a trampoline?
A: A milk shake!
Q: Where does chocolate milk come from?
A: Brown cows!
There’s a popular news story around claiming that 7% of Americans think chocolate milk comes from brown cows.
It’s not true.
That is, it’s probably not true that 7% of Americans think chocolate milk comes from brown cows. If you try to trace the primary source, lots of stories point to Food & Wine, who point to the Innovation Center for U.S. Dairy, who point to Today.com, who point back to Food & Wine. Critically, none of the sources give the actual questions. Was the question “Where does chocolate milk come from?” Was it “Lots of people say chocolate milk comes from brown cows, do you agree or disagree?” Was it “Does chocolate milk come from: (a) brown cows, (b) mutant sheep, (c) ordinary milk mixed with cocoa and sugar?” Was there a “Not sure” option?
This was clearly a question asked to get a marketing opportunity for carefully-selected facts about milk. If the Innovation Center for US Dairy was interested in the factual question of what people believe about chocolate milk, they’d be providing more information about the survey and how they tried to distinguish actual believers from people who were just joking.
The Washington Post story does go into the more general issue of ignorance about food and agriculture: there’s apparently a lot of it about, especially among kids. To some extent, though, this is what should happen. Via the NY Times
According to Agriculture Department estimates going back to 1910, however, the farm population peaked in 1916 at 32.5 million, or 32 percent of the population of 101.6 million.
It’s now down to 2%. Kids don’t pick up, say, how cheese is made, from their day-to-day lives, and it’s not a top educational priority for schools.
The chocolate milk story, though, is bullshit: it looks like it’s being spread by people who don’t actually care whether the number is 7%. And survey bullshit can be very sticky: a decade from now, we’ll probably find people citing this story as if it was evidence of something (other than contemporary news standards).
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
I think it’s a good idea for serious surveys to also include a question or two like this as a sanity check. For example:
http://jaysonlusk.com/blog/2015/1/15/food-demand-survey-foods-january-2015
82% supported mandatory labelling of food containing GMOs but 80% also supported mandatory labelling of food containing DNA.
8 years ago
I actually think that particular survey is an example of really bad wording (as I wrote here). It looks like the respondents interpreted the two questions as being the same. This makes sense, since there is no other reasonable interpretation of the second one, and basic pragmatics of conversation suggest that you should prefer reasonable over unreasonable interpretations of questions.
There are situations where it’s helpful — eg, asking teens about knowledge of illegal drugs — to put in a bogus option for calibration, but even then you have to be careful there isn’t a reasonable interpretation of the question that’s different from what you’re trying to ask.
8 years ago