May 25, 2013

Error and margin of error

By

The British consumer magazine Which? ran a mystery-shopper investigation of UK pharmacies recently.  One of the topics they covered was homeopathic products, and in 13 of the 20 pharmacies where they asked about homeopathic remedies they got advice that violated the Royal Pharmaceutical Society guidelines for pharmacists.  An information sheet from the society says that pharmacists should

when asked, assist patients in making informed decisions about homoeopathic products by providing the necessary and relevant  information, particularly the lack of clinical evidence to support the efficacy of homoeopathic products. They must also ensure that patients do not stop taking their prescribed medication if they take a homoeopathic product. Importantly, pharmacists will be in a position to discuss healthcare options and be able to identify any more serious underlying medical conditions and, if required, refer the patient to another healthcare professional.

(the most current guidelines are not available to non-members, but similar advice is quoted by Which?).

One of the responses to these findings has been that the survey is too small to draw any conclusions, but as Ben Goldacre points out, if the sample is representative, a sample size of 20 is enough to be worried


With sample sizes this small, different ways of calculating the uncertainty give visibly different results (I get 41%-85% in R), but the practical implications are the same.  If we can trust the sampling, at least 40% of pharmacists are providing advice that’s contrary to their professional guidelines and, since the mystery-shopper scenario was a patient who’d had a persistent cough for a month, advice that could actually be dangerous.

It would be interesting to know what the situation is like in NZ, especially as homeopathic products are specifically exempted  from the new rules that will require evidence to support health claims.

avatar

Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »

Comments

  • avatar
    Richard Penny

    I agree that bottom of CI is pretty worrying.

    This also raises a point that I think is often not made clear to students. That is, what CI should you be seeking? The way I express it is to give the client two possible outputs, say 50% for and 55% for and ask would they do different things for those different results. Usually they won’t. I often find they have a below 25%, 25-50, 50-75 and above 75%, so the CI can be quite wide, which makes the sample size a LOT smaller.

    In other words translate the results of the hypothesis test into possible actions.

    For those clients where it is clear that differing outputs will make no difference in their actions I try to not be involved at all.

    12 years ago