August 21, 2013

Measuring what you care about

By

From the Twitters

There followed a lively discussion, with comparisons to election results in various unpleasant places.

As I’ve suggested before, the really damning aspect of the 99.9% bus punctuality stats is that they might actually be honest. That is, `punctuality’ has been defined in a way that it does not reflect passenger experience and is of almost no interest to passengers. A bus is ‘punctual’ if it starts its route (which I assume means the driver starts up the ticket system) no more than 5 minutes late. What happens later doesn’t matter, as long as the bus does eventually reach its destination.

The real-time prediction system (although it may not be great at predicting the future) knows where the buses have been, so it would be feasible to set up a punctuality summary that actually measured punctuality — perhaps the proportion of major timepoints where the bus was no more than 5 minutes late or 1 minute early in departing.

I mention this point again because I’m teaching Design of Clinical Trials at the moment, and one of the big issues is ‘surrogate outcomes’. That is, people running studies are often tempted to measure something convenient rather than what patients care about: diabetes trials may measure blood sugar rather than heart attacks and kidney failure; heart disease trials may measure blood pressure rather than heart attacks or strokes; cancer trials may measure tumour size rather than survival or symptoms.  This matters, because there are important examples where a treatment improves a surrogate outcome, but makes the real outcomes worse.

There is no way that the Auckland bus punctuality statistics are accurate measurements of something that matters to passengers. It’s not a matter of life and death as it is in clinical trials, but it’s still a waste of time and money.  And since the Auckland bus system is actually pretty good, it makes Auckland Transport and the Council look unnecessarily stupid.

avatar

Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »

Comments

  • avatar
    Brendon Brewer

    Sometimes people start caring too much about a particular thing only because it is more easily measured than other stuff that’s actually more important. For example, GDP.

    11 years ago