Metrics and multivariate data

If you have a large collection of measurements there isn’t going to be a unique way to put them together into a single ordering: what you get out depends to some extent on your criteria. That doesn’t mean the measurements, or even the rankings, are meaningless, but it does mean that you should work out what your criteria are before you see the data, and that criticism of the choice of criteria is perfectly reasonable.

An illustration is yesterday’s PBRF research evaluation results.  For those of you playing along at home, PBRF is one of the mechanisms the country uses to allocate research funding. Unlike individual research grants, which are based on competition between individual proposals, PBRF funding is allocated to large groups of researchers for long periods of time based on a aggregated results from a single standardised evaluation.

Though it’s not the point of the system, the existence of a large number of grades invariably tempts university management into coming up with ways to combine them to make their institution look good. And they always succeed:

• In first place, we have the University of Auckland: “secured the largest share of the fund, $80.4m or 30.6% of the national total… This is due in part to the University’s impressive 288 international quality (A-rated) researchers – the greatest number of leading researchers anywhere in the country.”
• And in first place, the University of Otago: “Otago was ranked first among New Zealand universities in the measure of research quality weighted by its postgraduate roll (AQS (P)) and second in the measure weighted by degree-level enrolments and higher (AQS (E)). The University is the only TEO to be ranked in the top four in all four AQS measures.”
• In first place, also, Victoria University of Wellington: ” the latest PBRF Evaluation ranks Victoria as number one in New Zealand….With 678 staff actively involved in research, and 70 percent of them operating at the highest levels (ranked as either an A or B), we now have external confirmation of our status as New Zealand’s most research intensive university.”
• And finally, in first place (special subject), Lincoln University:  “confirms Lincoln University’s position as New Zealand’s specialist land-based university. … Lincoln University also has the highest amount of external funding for research (measured as income/staff member), demonstrating close links with industry and relevance of the University’s research.”

Other institutions aren’t claiming first place, but are still saying that the PBRF demonstrate how successful they have been. This has been another illustration that anyone saying “the data speak for themselves” is not to be trusted. The question matters.