If it’s not worth doing, it’s not worth doing well?
League tables work well in sports. The way the competition is defined means that ‘games won’ really is the dominant factor in ordering teams, it matters who is at the top, and people don’t try to use the table for inappropriate purposes such as deciding which team to support. For schools and hospitals, not so much.
The main problems with league tables for schools (as proposed in NZ) or hospitals (as implemented in the UK) are, first, that a ranking requires you to choose a way of collapsing multidimensional information into a rank, and second, that there is usually massive uncertainty in the ranking, which is hard to convey. There doesn’t have to be one school in NZ that is better than all the others, but there does have to be one school at the top of the table. None of this is new: we have looked at the problems of collapsing multidimensional information before, with rankings of US law schools, and the uncertainty problem with rates of bowel cancer across UK local government areas.
This isn’t to say that school performance data shouldn’t be used. Reporting back to schools how they are doing, and how it compares to other similar schools, is valuable. My first professional software development project (for my mother) was writing a program (in BASIC, driving an Epson dot-matrix printer) to automate the reports to hospitals from the Victorian Perinatal Data Collection Unit. The idea was to give each hospital the statewide box plots of risk factors (teenagers, no ante-natal care), adverse outcomes (deaths, preterm births, malformations), and interventions (induction of labor, caesarean section), with their own data highlighted by a line. Many of the adverse outcomes were not the hospital’s fault, and many of the interventions could be either positive or negative depending on the circumstances, so collapsing to a single ‘hospital quality’ score would be silly, but it was still useful for hospitals to know how they compare. In that case the data was sent only to the hospital, but for school data there’s a good argument for making it public.
While it’s easy to see why teachers might be suspicious of the government’s intentions, the rationale given by John Key for exploring some form of official league table is sensible. It’s definitely better not to have a simple ranking, and it might arguably be better not to have a set of official comparative reports, but the data are available under the Official Information Act. The media may currently be shocked and appalled at the idea of league tables, but does anyone really believe this would stop a plague of incomplete, badly-analyzed, sensationally-reported exposés of “New Zealand’s Worst Schools!!”? It would be much better for the Department of Education to produce useful summaries, preferably not including a league-table ranking, as a prophylactic measure.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
There’s been some replies on Twitter to this post:
Russell Brown says: “He doesn’t acknowledge how bad this data is, or the real-world effects of its publication. But the point that if it has to be done, then it’s better done by the MoE than the press, yeah. The original research design was predicated on *no league tables*. That’s screwed now.”
Nat Torkington says: “Without a good reason for league tables, becomes “if it has to be done, best that the police kill all the Jews.””
13 years ago
[…] Lumley at Stats Chat blogs: League tables work well in sports. The way the competition is defined means that ‘games won’ […]
12 years ago
Some useful points in your article, Thomas.
I think teachers generally have two concerns. 1. The NS data is inconsistent and unreliable which exacerbates the data uncertainty inherent in any league table. 2. The issue of parent choice hasn’t been properly balanced by Govt with the evidence about the negative consequences of league tables on student learning and teacher/school behaviour. Besides which, NZ students out-perform UK/US and other jurisdictions with league tables, so where’ the evidence “importing” this policy will improve NZ schools?
Harvey Goldstein’s latest work on league tables in the public sector, including education, is good on this stuff too:
http://www.bristol.ac.uk/education/news/2012/70.html
12 years ago
[…] “League tables work well in sports. The way the competition is defined means that ‘games won’ really is the dominant factor in ordering teams, it matters who is at the top, and people don’t try to use the table for inappropriate purposes such as deciding which team to support. For schools and hospitals, not so much.” Read more here […]
12 years ago
1)
Are the data really available under the official infomation act? Given that the school is named then students (especially of small country schools) are likely to be identifiable.
And under the privacy act, the data can’t be released unless the people can’t be identified.
I mean, who wants a 7 year old exposed to the nation as being well below standard? Who is that good for?
The Privacy Act also prohibits data being used in a manner for which it was not intended. The MinofEd has said that it wasn’t collecting it to create league tables so now they can’t change their minds and say they are going to create league tables – the data was collected for the benefit of the teacher/school/student/parent and MinofEd reporting requirements – that was what parents were told.
It will just take one kid identified and parent’s will start asking for their kids data to be withheld.
2)
Hospital league tables make some sense (and their are hospital league tables but they are not a single table, IIRC there are about 12 indicators that hospital boards are ranked on) because they are mostly objective i.e. what percent of kids have been vaccinated.
But league tables based on a test process that every school makes up (i.e. what tests to use to check kids meet the standards) and each school sets it’s only grading system (i.e. in one school a piece of writing may be “at standard” and at another school it may be “above standard” ) is hardly going to be accurate.
3)
Following the newsletters of the schools in my area, I can see two things going on
a) that rich schools are asking parents to pay for subscriptions to “kill and drill” websites and
b) that rich schools are asking parents to pay for more testing (ICAS)
The first is the “teaching to the test” mentatlity and the second gives more opportunity for kids to score “above standard” – just like rich kids in the US can pay out to take the SATs multiple times to get a score that suits.
Is this in the best interests of a child’s education? Surely a statistician such as yourself would rather kids worked on rich problems rather than multi-choise, memory-based problems.
4) The measure aren’t that great anyway because they top out at a low level i.e. the class with ten kids “above standard” but operating one year ahead of their year group is very different to the class with 10 kids “above standard” who are operating 4 years above their years group. But they are both assumed the same.
Now the MinofEd might not mind how “above standard” kids are doing but a parent with an gifted kid might.
12 years ago
Megan,
As you know from reading the post, I’m not actually in favor of league tables. I am in favor of feedback to schools, but I don’t see anyone objecting to this.
However, for your first point: The Government thinks that the results are covered by OIA, so they would probably respond to requests. The NZEI also seems to think the results are covered, because they are asking for changes to the rules to stop results being released.
12 years ago
Actually the MinofEd says it’s public information and so there is no onus on them to supply it under an OIA. They don’t have to supply stuff that’s publically available.
It’s public information because details about the whole school (not just individuals) were meant to be presented to parents from last year IIRC.
That’s why the reporters are going to the schools.
I think it’s going to be one for the lawyers to sort out.
~~~~~~~~~
It really isn’t clear that you’re against league tables. From reading your post it sounds like you know that league tables collapse information too much and have high errors on the ranks but because the information is available, schools should be comapred against each other and informed and then those results made public. That sounds like league tables minus one trivial step.
~~~~~~~
But even a report to (only) schools comparing them to other like schools will still be next to useless as I said above…
But league tables or school reports based on a test process that every school makes up (i.e. what tests to use to check kids meet the standards) and each school sets it’s only grading system (i.e. in one school a piece of writing may be “at standard” and at another school it may be “above standard” ) is hardly going to be accurate.
~~
The hilarious thing is that noone ever audits the national standards. There is obviously room for an unscrupulous schools to tell the truth to parents and lie to the MinofEd.
And if anyone says there are no unscrupulous schools then they need only remember Cambridge High School and NCEA.
12 years ago
Ok, in case it’s not completely clear. I’m against league tables. I’m in favor of reports sent back to the schools on how they compare with other schools.
I think league tables put out by the government would still be better than ones put out by newspapers.
Clear now?
12 years ago