Difficulties in interpreting rare responses in surveys

If some event is rare, then your survey sample won’t have many people who truly experienced it, so even a small rate of error or false reporting will overwhelm the true events, and can lead to estimates that are off by a lot more than the theoretical margin of sampling error.

The Herald has picked up on one of the other papers (open access, not linked) in this year’s Christmas BMJ, which looks at data from the National Longitudinal Study of Youth, in the US. This is an important social and health survey, and the paper is written completely seriously. Except for the topic

Of 7870 eligible women, 5340 reported a pregnancy, of whom 45 (0.8% of pregnant women) reported a virgin pregnancy (table 1). Perceived importance of religion was associated with virginity but not with virgin pregnancy. The prevalence of abstinence pledges was 15.5%. The virgins who reported pregnancies were more likely to have pledged chastity (30.5%) than the non-virgins who reported pregnancies (15.0%, P=0.01) or the other virgins (21.2%, P=0.007).

and

A third group of women (n=244) not included in analysis, “born again virgins,” reported a history of sexual intercourse early in the study but later provided a conflicting report indicating virginity. Reports of pregnancy among born again virgins were associated with greater knowledge of contraception methods with higher failure rates (withdrawal and rhythm methods) and lower interview quality (data not shown), and reports from this group may be subject to greater misclassification error.

The survey had carefully-designed and tested questions, and used computer-assisted interviewing to make participants more willing to answer potentially embarrassing questions. It’s about as good as you can get. But it’s not perfect.