And again
The Surgisphere papers on COVID (the chloroquine one I mentioned yesterday, and one on the safety of blood pressure drugs that affect the angiotensin system) have been retracted, at the request of three of the four authors. They had organised an independent review of the data, but:
Our independent peer reviewers informed us that Surgisphere would not transfer the full dataset, client contracts, and the full ISO audit report to their servers for analysis as such transfer would violate client agreements and confidentiality requirements. As such, our reviewers were not able to conduct an independent and private peer review and therefore notified us of their withdrawal from the peer-review process
With the combination of widespread concern about the data veracity and the unequivocal evidence from the Guardian that the Australian data were misrepresented or falsified, they didn’t have much choice. The charitable (and quite plausible) assumption is that those three authors were dupes: someone offered them the ability to do (or at least publish) valuable analyses of COVID treatment options, and they didn’t check up on the provenance.
Ben Goldacre says this incident shows the importance of publishing code. I’m not convinced. If I were faking a 90,000 person data set and sophisticated data analyses, I’d actually do the work of faking the data, not just the results. Then I could publish the code perfectly safely, claim the data were confidential, but still be tripped up by someone willing to call up all the major hospitals in Australia and check whether they were involved.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »
It seems a high ask from the auditors to transfer the whole dataset to their servers. I would have expected that to be problematic even if the data were real. For example, you couldn’t do that if you wanted to audit the Stats NZ Integrated Data Infrastructure. You would need to get access to the Stats NZ servers, through their secure facilities, and do your audit there.
However, the inability to hand over client contracts – or even the names of any clients – to the auditors is sufficiently damning that no further investigation would be needed.
4 years ago
I wouldn’t expect data transfer like that, but contracts and the audit report. Also, the ability to ask for additional summaries at the hospital level, and potentially to check those summaries with some of the originating hospitals.
Clinical trials routinely do raw data audits, but it is a very expensive process.
4 years ago
For example, given the anomalies in smoking rates and chloroquine dosage, if the data were real I’d expect the reviewers to check with (at least some) source hospitals that they really have those smoking rates and use those doses.
4 years ago
You raise a really good point. Accessing the Integrated Data Infrastructure is not simple, even if you work at StatsNZ.
However, as you write, it would useful if it was straight-forward to get summaries so at the very least you can see if their size of the sub-populations they are comparing align with what is expected.
For people the last Census data is probably close to current population, and is published to a lot of detail. But for businesses and other like-entities they change a lot over short time periods. So looking at business population now and 2 years ago will give you different looks.
4 years ago