Stats Chat

There are about forty COVID vaccine candidates now, with several heading to Phase 3 trials.   The  CDC has suggested there could be a vaccine before the US presidential election. How could that happen? There’s one obvious answer, sadly, but how else could it happen?

The FDA has said it would expect that a COVID-19 vaccine would prevent disease or decrease its severity in at least 50% of people who are vaccinated.  In the current circumstances, if your vaccine was that effective it would be false economy to run a trial that was too small — you’d want to be very confident of a positive result if your vaccine met the mark.

You might design a trial that ran until you had seen, say, 150 infections or symptomatic cases or whatever. If you ended up with a split of 50 vs 100, you’d estimate a 50% reduction with the vaccine, and the uncertainty interval (‘margin of error’) in would stretch to a 30% reduction, which correspond to the FDA’s guidance.   Only 150 cases perhaps doesn’t sound like much, but remember that at a very high infection rate of 1% you’d need 10,000 people in the control group and the same number in the treatment group. That’s a big trial.

It’s possible, though, that your vaccine is much better than just a 50% reduction in cases. If it actually prevents 90% of cases, you could have run a much smaller trial, at lower cost, finished earlier, and saved more lives.  On the other hand, if you ran the smaller trial and the effectiveness is only 50%, you are in big trouble — you won’t get approved, but you’ll find it harder to run another trial.

So, what we do is to set up a big trial that can detect a 50% benefit, and peek at the data part of the way through.  If the vaccine turns out to be 90% effective, you’ll be able to stop earlier than planned and declare victory.  You’ll still have recruited more people than if you were betting everything on 90% effectiveness, but you won’t be betting everything on 90% effectiveness.

Stopping early requires strong evidence.  It requires strong evidence because it’s easy to get fooled by randomness in early trends. It requires strong evidence because early stopping means less data on safety. And it  requires strong evidence because people are going to suspect political interference.  One popular guideline for designing early-stopping rules, to give us something concrete to look at, says that you take margin you’d need to be convinced at the end of the trial, and use that margin earlier on.  In our hypothetical example, we had 50 cases difference as the success criterion at the end of the trial, so you’d look for 50 cases difference at earlier timepoints as well.  That’s a much bigger relative difference early on:  if you only had 100 cases, a difference of 50 is 25 vs 75; with only 60 cases, it’s 5 vs 55.

My understanding is that a 90%-effective vaccine would be surprising, but not inconceivable and a 70%-effective vaccine isn’t at all unlikely, so interim analysis and the potential for early stopping do make sense.  Political interference doesn’t help, though: even more than most drugs, a vaccine only counts as effective if lots of people are willing to take it, and a trial that doesn’t engender that willingness is a failure no matter the statistical results.

• A surprising (to me) statistic from the Washington Post: 100 US law enforcement officers have died of COVID contracted at work. That’s more than all other line-of-duty deaths this year put together. And it  doesn’t include 150 more deaths that the Officer Down Memorial Page is still reviewing.
• A particularly malign case of ‘predictive policing’ in Florida.
• You’ve probably heard about the UK’s  plan to impute A-level grades after cancelling exams. A good discussion of why this is a hard problem and how to avoid solving it. 
• “Sooner or later we are going to have to have an inquiry into the role of statisticians in the Covid-19 crisis,” declared the journalist Ross Clark in the Daily Telegraph recently.” David Spiegelhalter would relish the opportunity.
• Alert-level decision in NZ are hard; getting them wrong either way can have very bad consequences. Ars Technica has an excellent report on a similar sort of decision: whether or not to evacuate Houston as Hurricane Laura approached.
• The FDA Commissioner, talking about plasma from recovered COVID patients as a treatment, managed to get both the estimated benefit and the reliability of the evidence badly wrong. Hilda Bastian writes about the implications for informed consent by future patients

There’s an interesting poll in the NZ$ Herald asking whether the four-day extension to the Auckland level 3 ‘lockdown-lite’ was appropriate or not. Here’s a graph of the regional results (excluding the worryingly-small fration of “don’t knows”). The purples are ‘yes, should have been extended’, with the light purple being ‘the four day extension was appropriate’ and the dark being ‘longer would have been better’. The oranges are ‘no, should not’, with the light orange being ‘should not have been extended’ and the dark being ‘should not have happened at all’

As you can see, there’s pretty strong consensus across the country.  You’d expect people outside Auckland, who get the benefits but with less of the cost, to want tighter restrictions, and the patterns seem to fit that.

The Herald also explores differences by age, gender, and income. It’s hard to say anything too strong about many of the differences, because we’re only quoted a margin of error for the whole survey, not for any of the subgroups.  In some cases I can work it out: if Auckland and Canterbury were represented in the sample in proportion to their actual population sizes, the difference between the purple bars would be right about at the margin of error.  In general, though, the overall margin of error is pretty useless for most of what makes the story interesting.

Oh, and also.

Auckland is strongly divided over whether extending the lockdown was an appropriate response to the resurgence of Covid-19, a new poll shows.

But the exclusive new poll shows rest of the country was far more accepting of the Super City being kept in alert level 3 for almost three weeks – with many wanting it extended even longer.

Emphasis added.

No. That’s really not what the poll says

Team Ratings for Round 10

As an indication of the care needed at present in determining home ground advantage, the comprehensive scores site flashscore.com.au still has the Rebels v Waratahs game as being played in Melbourne. (Click on the match result to see the scorers and times and match details at the bottom.) Fortunately I checked beforehand that it was to be played in Sydney so I gave the Waratahs home ground advantage and predicted them to win. Otherwise I would have predicted the Rebels, even if played on a neutral ground.

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 18 matches played, 15 of which were correctly predicted, a success rate of 83.3%.
Here are the predictions for last week’s games.

Predictions for Round 10

Here are the predictions for Round 10. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 18

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 102 matches played, 66 of which were correctly predicted, a success rate of 64.7%.
Here are the predictions for last week’s games.

Predictions for Round 18

Here are the predictions for Round 18. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for the Semi-finals

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 101 matches played, 78 of which were correctly predicted, a success rate of 77.2%.
Here are the predictions for last week’s games.

Predictions for the Semi-finals

Here are the predictions for the Semi-finals. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 17

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 128 matches played, 88 of which were correctly predicted, a success rate of 68.8%.
Here are the predictions for last week’s games.

Predictions for Round 17

Here are the predictions for Round 17. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 17

I got caught out a bit this round with games being played earlier in the week and then again on the weekend. As usual my predictions are what they would have been before the games were played, the predictions are produced by running my algorithm over the past data and information about home grounds.

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 96 matches played, 61 of which were correctly predicted, a success rate of 63.5%.
Here are the predictions for last week’s games.

Predictions for Round 17

Here are the predictions for Round 17. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Auckland is going back to level 2 (God willing and the creek don’t rise) on Monday.  Together with my local circle of health and stats nerds, I’m viewing this with some concern. It hasn’t been very long since we had a case show up with no previously known contact to the cluster. Like, Tuesday.  It’s quite possible there are still a few other people in the cluster who haven’t been found yet.

The concern is not that it will actually be dangerous on Monday to be going to work or going shopping. The number of undetected cases will be small; your chance of getting infected on Monday is tiny.  The problem is, as long as there are undetected, infectious cases, your chance of getting infected on Tuesday is very slightly higher. And slightly higher again on Wednesday, and so on in exponential increase. Eventually, it may get dangerous, and  stopping it then is much more costly in health and freedom and money.  In Victoria today they are talking  about the psychological boost of getting a day with less than 100 new cases. If we’re careful and lucky, the current testing and tracing will be enough to stop this cluster exploding; if we’re not, maybe not.

One challenge in COVID risk communication is that the risk is longer-term and social, not immediate and individual. It’s important that as many  people as possible take precautions against spreading the virus: distancing, masks, getting tested if you have symptoms, working from home if that’s feasible, avoiding places with poor ventilation. But it’s not so much important for your or your family’s immediate safety: the risk is currently very low. If someone jogs past you at close range without a mask or stands a bit too close in  a supermarket line, you don’t need to panic — you almost certainly won’t catch the coronavirus. On the other hand, the more people behave that way, the more chance that they outbreak will slowly get out of control.  The appropriate level of care is a lot higher than the appropriate level of fear.

Ideally, everyone would be as careful as if the virus was everywhere, but nowhere near as scared as if it was everywhere. That’s a very difficult balance, and a reason for the ‘be kind’  message when it doesn’t quite work out.

Q: Did you see this study saying vaping makes you five times more likely to get COVID?

A: Yes, but it’s not in the news, so it doesn’t count for StatsChat

Q: Newshub covered it.

A: Ok. Not entirely convinced

Q: They did a survey and they did lots of reweighting, the way you like. And said exactly what questions they asked.

A: Yes…

Q: So it’s not dodgy like the paper about heartburn drugs

A:  No, not like that.

Q: What’s your problem, then

A: The first problem is the proportion of people with COVID tests. No, actually the first problem is that COVID is so rare that this isn’t a reliable way to estimate proportions, and the second problem is that getting a test depended on a lot of other factors back then.  The third problem is the proportion of people with COVID tests

Q: Which is?

A:  Over 5% of people 13-17 and over 10% of people 21-24. By May 14, when the total cumulative number of tests in the whole US was only about 5% of the population — and you’d expect lower testing rates in younger people.

Q: Where are those numbers in the paper?

A: It’s a combination of the user and non-user columns in Table 1, using the proportions in the Supplementary Material. Which, again, the authors should get credit for providing.  What they call “COVID-related symptoms” are also very high: 14% of non-vapers and 26% of vapers reported having the symptoms right at the time they were surveyed.

Q: You’d think we would know if vaping increased these symptoms that much, separately from COVID. But if they oversampled people at high risk of COVID, it should at least be comparable across their survey

A: They did separate surveys for users and non-users of e-cigarettes, so that’s not actually obvious.

Q: But weighting?

A: Yes, but that doesn’t help as much with matching the surveys to each other, especially as they don’t have separate census totals for vapers and non-vapers.  In particular, in mid-May COVID was concentrated in relatively small areas of the US, and it would have been more valuable to make sure the locations matched up.

Q: But we know that smokers are at higher risk of catching the coronavirus, so this just confirms that.

A: Surprisingly, no.  Smokers don’t seem to be at higher risk of getting infection — and I guarantee that it’s not because no-one tried to show they were.  They may be at higher risk of getting seriously sick if they are infected, but even that’s not as clear as you’d expect.

Q: So should we believe this?

A: It’s not as simple as that.  This study does provide some evidence, but not as strong evidence as the researchers think. It certainly isn’t strong enough evidence to change policy on the regulation of e-cigarettes; whatever  you believed about that before seeing this study, you should believe about the same afterwards. And you probably do — it’s not a topic where people are noted for changing their minds.