Posts from May 2020 (14)

The Lancet published a paper recently claiming that chloroquine and hydroxychloroquine, with or without azithromycin and similar antibotics didn’t help COVID-19 but did cause heart problems, in 96000 people hospitalised for the disease across six continents.Since that’s what most people who would pay attention to a Lancet paper already believed,  it was basically a confirmation. I argued that it made future randomised trials a bit unnecessary.

There is increasing concern about whether the paper accurately represented the source and quality of the data that went into the analysis. More precisely, sober and responsible people are even suggesting it might have been made up.

The Guardian has a headline:  Covid-19 study on hydroxychloroquine use questioned by 120 researchers and medical professionals. Which, by now, is a big underestimate.

Probably the biggest concern is that idea that a software system released last year, could already be getting real-time clinical data from 671 hospitals in six continents, apparently with the rest of the healthcare informatics industry not noticing.

Also, there’s no data about the hospitals except which continent they are in, which is unusual for research of this type.

On top of that, some of the data are clearly wrong.   I hadn’t read the paper carefully.  I looked at it this morning in a bit more detail. One thing that jumped out was the claim that 609 of the hospitalised COVID cases were at five hospitals in Australia.  This is using data up to April 14. On April 10, the Herald talked with Tony Blakely, who said there were 263 hospitalised cases in all of Australia. There’s no way these numbers match up.

It turns out that someone had noticed this already: the Australian data don’t match other data sources in other ways. In a new correction, the researchers said they had listed one hospital in the wrong continent. There were actually only 63 patients from Australia whose data  was in the study, and one of the hospitals was in Asia:

“We have reviewed our Surgisphere database and discovered that a new hospital that joined the registry on April 1, and self-designated as belonging to the Australasia continental designation,”

They’d been mixed up because the hospital said it was in “Australasia”.  Now, I’d think if you were setting up data access and non-disclosure contracts with hospitals, you’d want to know what country’s laws the hospitals operate under, but maybe machine learning can get around these issues.

A further interesting question is where this hospital could possibly be. Remember, it is supposed to have had 609-63=546 hospitalised, PCR-positive, COVID cases by April 14, and to be in Asia,  and to have self-described as being in Australasia.  We can rule out the Pacific islands that would be regarded as in Australasia — they don’t have anywhere near that many cases. Papua New Guinea has eight confirmed cases, counting up to right now.   Timor-Leste has 24.

Could it be a hospital in Indonesia? Indonesia started testing very late, so 546 PCR-confirmed, hospitalised patients by April 14 is a fair chunk of the whole country. The special COVID hospital that’s now at the Asian Games’ athletes’ village near Jakarta has been in the news, and it’s got the right sort of patient numbers, and so there might be a couple of other Indonesian candidates. But it’s hard to see them self-describing as being in “Australasia”, or as having signed up to  this new database system in the middle of a pandemic.

Home Ground Advantage

Determining home ground advantage for this new draw has been very difficult. In some cases it has been straightforward: Storm at home, Cowboys at home, Broncos at home, Warriors always away despite their being the nominal home team. But Knights versus Dragons at Bankwest Stadium? Raiders versus Sea Eagles at Campbelltown? I made the ground neutral for those. I gave home ground advantage to a team if the opponent was from interstate (not counting Raiders playing in Sydney). Also the Eels at Bankwest Stadium have home ground advantage.

To deal with the Warriors always being the away team, I have swapped the order of the teams for some matches because the first team is always the team with home ground advantage in my code.

And yes, I know that a component of home ground advantage is thought to be the home crowd, but there are no crowds, …

Team Ratings for Round 3

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 16 matches played, 10 of which were correctly predicted, a success rate of 62.5%.
Here are the predictions for last week’s games.

Predictions for Round 3

Here are the predictions for Round 3. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

• Matt Nippert has two wonderful stories about the government response to the pandemic, in the NZ$ Herald.  He’s used the huge trove of documents released by the government about a week ago to show the government scrambling frantically, but still (by luck and skill and co-operation) managing to keep up.   We were so close to having the wheels come off the public-health response system.

• “You always expect some resistance but if I had known people’s mental anchor in that negotiation was 10 cases, I might not have been so courageous in my recommendations.      So I’m pleased I didn’t know that.” Ayesha Verrall, who audited the contact-tracing system, talking to Derek Cheng in the NZ$ Herald.

•  “the stark turning point, when the number of daily COVID-19 cases in the U.S. finally crested and began descending sharply, never happened. Instead, America spent much of April on a disquieting plateau”  Ed Yong’s brilliant ‘patchwork pandemic’ story in The Atlantic
• There’s still controversy over whether the infection fatality rate of COVID-19 is around 1 in 100 or around 1 in 1000. Hilda Bastian comments on the latest meta-analysis by John Ioannidis.
• Meanwhile, in New York City, excess deaths this year reached 20,569, which works out to one person in 400.
• Good news (via XKCD and HuffPost and Pew) is that most people in the US agree with the social-distancing programs.
• Less good news: only 52% even of Democrats will call ‘false’ the statement “Bill Gates wants to use a mass-vaccination campaign against COVID-19 to implant microchips that would be used to track people with a digital ID” (via @publicaddress).

For the first time, there’s data from a reasonably large randomised trial of a COVID-19 treatment: a drug called remdesivir. And it’s modestly good news.  People in hospital, with lower respiratory tract symptoms were randomly assigned to remdesivir or placebo, and those given remdesivir recovered faster, and more of them recovered in 30 days — in the sense that they were healthy enough to leave hospital.  There wasn’t definitive evidence on the proportion dying, though there were fewer deaths in the remdesivir group.  The treatment didn’t seem to do much for people who were already on ventilators, but it’s hard to be sure about subgroups when the whole trial is only just big enough to give convincing results.  The improvement wasn’t massive (25-30% better), but it does look real, which is encouraging for other drugs that work the same way, as well as for future patients.

It turns out that I know the statistician who did the main analysis (Hi, Lori!), and who must have had an incredibly stressful couple of months (especially after the basic trend was publicised by Dr Fauci a month ago).

Update: the following study is now looking really, really dubious, so perhaps don’t bother reading about it.

There’s also data from another large observational study of chloroquine.  It’s from a collection of hospitals around the world who kept registers of who got what treatment (even though it wasn’t randomly allocated). This paper looks at 96000 people who were treated early (less than two days after diagnosis) and who were not on ventilators. There’s no suggestion of a benefit of chloroquine or hydroxychloroquine: in fact, patients getting either drug, with or without an antibiotic like azithromycin, did worse.

The paper ends with a ritual invocation “These findings suggest that these drug regimens should not be used outside of clinical trials and urgent confirmation from randomised clinical trials is needed.”  It’s getting towards the point where confirmation from randomised trials really isn’t needed any more.

MMP is, in many ways, a beautiful voting system.  As implemented in New Zealand it’s got one feature that complicates voting slightly and complicates forecasting a lot: the threshold.

In TVNZ’s Colmar Brunton poll, the Greens got 4.7% of the vote. The threshold is 5%. Getting 4.7% of the vote in an election would mean you don’t get any seats.   The margin of error that TVNZ were stating was +/- 3%, so based just on that, the Greens were basically 50:50 on whether they make the threshold.

At that level it also potentially makes a big differences how you treat the undecided voters, who made up 16% of responses. The person who pointed this out to me thought that the 16% had been left as a separate group, which you might easily think from the TVNZ web post on the poll.  But if you do the arithmetic, the parties’ quoted percentages add up to 100 (give or take rounding error), so the percentages were of those who expressed a preference.  Only 4.1% of the respondents actually said “Green”.

Normally, you don’t have to worry about this because tiny changes in preferences will only produce small changes, if any, in the number of seats. But going from 4.99% to 5.00% takes you from no seats to several (I think six) seats. Predicting the make-up of Parliament gets hard when there are parties close to the threshold.

Given the sensitivity of the results to small changes, I think the website (if not the actual news segment) should be more explicit about how undecided voters are handled in the seat projection.  And making statements about whether a party is in or out of Parliament should have a bit more explicit uncertainty when it could be wrong by six seats with minute changes in voting.

On the TVNZ website, the report says “ACT, assuming it wins an electoral seat, would pull in three MPs with its 2.2% support”, and if National’s gift of Epsom to David Seymour is enough of an uncertainty to require an explicit caveat, so is being a few voters per thousand away from the threshold. 

One of the features of the COVID pandemic is the near-elimination of delays in making scientific data available.  Everyone is moving to releasing preprints, and they’re that more rapidly than they did in the past.  In many ways this is great, but it does mean things can get publicised faster than they can get evaluated.

Two examples: one from a preprint, one from a press release

The preprint: There’s a headline in the Herald: Landmark study: Virus didn’t come from animals in Wuhan market. That’s a pretty big claim. So where did the Herald get  it? The Daily Mail on Sunday.  The Mail got it from a preprint that was uploaded a couple of weeks ago.   Most readers of the Mail and the Herald (including me) won’t be in a position to evaluate the credibility of the work. Because it sounds like it feeds into the whole quagmire of conspiracy theories and hoaxes around COVID origin, you’d really want to see some independent expert comment on the story. Writing a story like this  with quotes from the authors but no independent comments seems a bit irresponsible.

The press release: The Herald headline is Covid 19 coronavirus ‘cure’? US biotech company claims it’s found antibody to block virus. The CEO of the company is quoted as saying

“We want to emphasise there is a cure,’ Sorrento’s CEO, Dr Henry Ji, told Fox News.

“There is a solution that works 100 per cent. If we have the neutralising antibody in your body, you don’t need the social distancing. You can open up a society without fear

This is not normal.  “Works 100 per cent” is not something biotech CEOs go around saying about a product when they’re still trying to get FDA approval — and they haven’t actually tested the product in any humans.  Presumably the idea now is that it’s worth the risk of annoying the FDA if you get enough public interest. And since human tests have a nasty habit of not working nearly as well as lab tests, being able to sell your product without them is attractive.

The basic idea is sensible: using well-chosen synthetic antibodies rather than getting your body to make its own. In particular, they start working right away, where vaccines only start the process of getting your immune system to make antibodies. Also, some antibodies can have harmful effects, and you can avoid using those.  They’ll need to be administered by injection, and repeated as your body gets rid of them — synthetic (‘monoclonal’) antibodies for other conditions seem to mostly be given every 2-4 weeks. But, again, publishing a claim of a cure when it has not yet been shown to have any benefit at all in real live people, without any independent commentary is not best practice.

There’s a new COVID case from the Marist College cluster today.  The person previously tested negative, but had been in isolation (perhaps, though the Herald doesn’t say, because of the combination of symptoms and being a contact).  Now that we have plenty of testing capacity there has been follow-up testing of some clusters as well as testing of some apparently healthy people in high-risk jobs.

From what I’ve seen on social media, this has led some more people to find out about the false negative rates of the current tests.  It’s not a secret that the swab+PCR test we use in NZ misses maybe a third of infections (because there isn’t enough virus on the swab), though it hasn’t exactly been emphasised.  So, how is this acceptable? Well, “acceptable” depends on the alternative. It’s the best test we have. Researchers (and companies) are working on better ones, and things are likely to improve over time, as they did with HIV testing.  If you’ve been in contact with a case and have COVID-like symptoms and test negative, you’re still going to need to isolate until you recover.  That, plus the fact that the testing does pick up the majority of cases, means a test/trace/isolate strategy, done right, should be nearly enough to control an outbreak that’s caught early.

The current tests give basically no false positives. That’s really helpful for a test/trace/isolate strategy — we’ve done 200,000 tests, and if, say, 5% of them were false positives, that would be another 10,000 cases. Before counting all the contacts of those 10,000 people.  The low false positive rate also means the health system can say, “yes, you need to get tested”,  and then after a positive results, “yes, you absolutely must stay home”, “yes, you need to tell us about all the places you’ve been, even if some of them are embarrassing or illegal”.

There’s another testing-accuracy story in the New York Times, unhelpfully headlined Coronavirus Testing Used by the White House Could Miss Infections. It turns out that they don’t mean that it could miss infections the way all the other tests do; they mean it could miss infections that other tests detect.  The test in question is a portable testing machine from Abbott that takes only 5 minutes to process a sample, quite a bit faster than the standard testing systems.  Researchers from a testing lab at New York University’s Langone Medical Center (who liked the idea of a faster test, given the number of tests they perform) did comparisons to the machines they are currently using and published a PDF about it– some tests on the same swabs and some on different swabs taken at the same time from the patient.  They say the Abbott machine missed 1/3 (out of 15) or 1/2 (out of 30) samples where the current machine found the virus.

Abbott, on the other hand, said their evaluations showed 0.02% false negative rate.

You might wonder how two evaluations could be so different: one false negative in 50,000 samples vs 5 in 15 samples?  As usual, when two numbers don’t fit, it’s probably because they don’t mean the same. Abbott will have been referring to an estimated false negative rate in viral samples with a known virus concentration — an evaluation of the assay itself.  The NYU researchers are talking about live clinical use in samples where they know the virus is present in the swab.  And when we talk about false negatives in the NZ sampling system, we’re talking about samples where the virus probably isn’t present in the swab.

Abbott argue that the NYU researchers were using the machine incorrectly. That could be true, but it’s only reassuring to the extent that you think the White House will be better at it than a pretty highly regarded New York hospital and research centre.

From the Twitter account of the White House Council of Economic Advisors

From the Washington Post

• Models based on understanding the actual process that’s causing the trends.  The SIR models and their extensions, which we’ve seen a lot of in NZ, are based on a simplified but useful representation of how epidemics work.  Weather forecasting works this way. So do predictions of populations for each NZ region into the future.
• Models based on simplifying and matching previous inputs.  When Google can distinguish cat pictures from dog pictures, it’s because it has seen a bazillion of each and has worked out a summary of what cat pictures and dog pictures look like. It will compare your picture to  those two summaries and see what matches best.  Risk models for heart disease are like this: does your data look like the data of people who had heart attacks. Fraud risk models for banks, insurance companies, and the IRD work this way. It still helps a lot to understand about the process you’re modelling, so you know what sort of data to put in or leave out, and what sort of summaries to try to match.
• Models based on extrapolating previous inputs.  In business and economics you often need predictions of the near future.  These can be constructed by summarising existing recent trends and the variation around them, then assuming the trends and variation will stay roughly the same in the short term.  Expertise in both statistics and in the process you’re predicting is useful, so that you know what sorts of trends there are and what information is available to model them. A key part of these time-series models is getting the uncertainty right, but even when you do a good job the predictions won’t work when the underlying trends change.

The SIR epidemiology models that you might have seen in the Herald are based on knowledge of how epidemics work.  The IHME/UW models are at least based on knowledge of what epidemics look like. The cubic model isn’t.

The cubic fit is a model of the second type, based just on simplifying and matching the available data.  It could be useful for smoothing the data — as the tweet says, “with irregular data, curve fitting can improve data visualization”.  In particular, the weekly up-and-down pattern comes from limitations in the death reporting process, so filtering it out will give more insight into current trends.

The particular model that produces the red curve is extremely simple (Lucy D’Agostino McGowan duplicated it).  If you write t for day of the year, so t starts at 1 for January 1, the model is

log(number of deaths + 0.5) = -0.0000393× t³ +0.00804× t² -0.329×t – 0.865

What you can’t do with a smoothing/matching model like this is to extrapolate outside the data you have.  If you have a model trained to distinguish cat and dog pictures and you give it a picture of a turkey, it is likely to be certain that the picture is a cat, or certain that the picture is a dog, but wrong either way.  If you have a simple matching model where the predicted number of deaths depends only on the date, and the model matches data from dates in March and April, you can’t use it to predict deaths in June. The model has never heard of June. If it gets good predictions in June, that’s entirely an accident.

When you extrapolate the model forward in time, the right-hand side becomes very large and negative, so the predicted number of deaths is zero with extreme certainty.  If you were to extrapolate backwards in time, the predicted number of deaths would explode to millions and billions during December. There’s obviously no rational basis for using the model to extrapolate backwards into December, but there isn’t much more for using it to extrapolate forward — nothing in the model fitting process cares about the direction of time.

The Chairman of the Council of Economic Advisers until the middle of last year was Dr Kevin Hassett (there’s no Chairman at the moment). He’s now a White House advisor, and the Washington Post attributes the cubic model to him.  Hassett is famous for having written a book in 1999 predicting that the Dow Jones index would reach 36,000 in the next few years.  It didn’t — though he was a bit unlucky in having his book appear just before the dot-com crash.  Various unkind people on the internet have suggested a connection between these two predictive efforts.  That’s actually completely unfair.  Dow 36,000 was based on a model for how the stock and bond markets worked, in two parts: a theory that stocks were undervalued because of their relative riskiness, and a theory that the markets would realise this underpricing in the very near future.  The predictions were wrong because the theory was wrong, but that’s at least the right way to try to make predictions. Extrapolating a polynomial isn’t.

We often hear groups of people described as ‘high risk’ in the context of COVID.  The problem is that this means three different things, and they often aren’t distinguished clearly

• People who have a relatively high exposure to coronavirus, so they are more likely to catch it: nurses, doctors, supermarket staff, police (high probability)
• People who are more likely to get seriously sick if they do become infected: elderly, immunocompromised, people with chronic lung disease (high consequence)
• People who are more likely to spread the infection if they get it. Some overlap with the first group, but also migrant workers, prisoners, and at least in the US, meat processing workers. (high transfer)

The second group are very different from the others. Suppose you were doing intensive testing to try to see if there was undetected community transmission of COVID.  You’d definitely want to test the first group, because that’s where you’re most likely to find the virus, and you might want to test the last group, because missing it there would be serious (as it was in Singapore).  You might well not go after the second group, because the safest thing for them is isolation — having a bunch of health workers barge in and stick swabs up their noses is unpleasant and possibly risky.  You’d absolutely want to test the second group if there was any indication of symptoms or exposure, but not just in the ordinary course of business.  The three groups are different.

Even Cory Doctorow confused the first and second groups a bit, in his rant about the risks of contact-notification apps

The proximity sensing they do is going to miss out on people who don’t have smartphones and/or don’t have the technological savvy to install them. That overlaps broadly with the most at-risk groups: elderly people and poor people.

Epidemiology is a team sport and the most vulnerable people are the MVPs on the team. “Our app will tell you if you came in contact with an infected person (but not if that person is from the most likely group of infected people)” is a fundamentally broken premise.

Elderly people are a ‘high consequence’ group — infection is serious for them. They aren’t a ‘high probability’ group — there’s no special reason why elderly people in the community would be more likely to get infected (or in residential facilities, if good care is taken)

Nice story by Farah Hancock at Newsroom, on NZ mortality data.

In places with less successful control of COViD-19, there has been a spike in deaths confirmed as due to coronavirus, and also a spike in other deaths.  In New Zealand, there hasn’t been — there isn’t any clear excess over the average for the time of year.  There undoubtedly have been deaths due to coronavirus, but there have also been deaths prevented by the lockdown (on the roads, for example), and there may well have been deaths caused by the lockdown (eg, people not getting heart attacks treated promptly), but the overall trends are spectacularly unlike those in New York City, which has not quite twice the population of New Zealand and over 20,000 net excess deaths.

There’s still plenty of time for NZ to catch up to outbreaks in other parts of the world. Let’s not do that.