Vaccine effects and effectiveness: fair comparisons

Thinking about vaccine effectiveness is tricky, but Senator Rand Paul has a medical degree so he has no excuse

The Pfizer vaccine had seen 8 Covid cases in 22,000 people vaccinated.  If the way you computed vaccine effectiveness was to divide the number of infections by the number exposed, the way Paul has done for ‘naturally acquired Covid-19’, the effectiveness would be 21992/22000= 99.96%. Sounds pretty good!

On the other hand, if that was the way you computed effectiveness then just being in the US would be 95% effective — more than 95% of people in the US have yet to get Covid.  As being in the US increases your risk of Covid, we can be sure this isn’t the right way to do the computation.

Vaccine effectiveness requires a fair comparison between two groups: one group who gets the vaccine and one group who doesn’t.   We do this with randomised trials because it’s really hard to be confident about fair comparisons any other way.   When we say the Pfizer and Moderna vaccines are 95% effective in preventing symptomatic Covid-19, we mean that the proportion of people getting symptomatic Covid-19 in the vaccine group was 95% lower than the proportion in the placebo group.

There’s currently no real basis for saying immunity based on infection is higher or lower than immunity based on vaccine (except for the trivial point that getting Covid naturally is 0% effective as a way of not getting Covid at all). It’s a hard problem.

We obviously can’t randomise people to having had ‘naturally acquired’ Covid-19 infection. What we’d need to do to estimate the effectiveness is find large groups of comparable people who were and weren’t infected back earlier this year, make sure we gave these groups the same risks of exposure to Covid-19 and the opportunities to get tested, and count the number of new cases.   So, we’d need to find some region that had very high rates of infection back in February/March, with reliable testing, and that has very high rates again now, again with reliable testing.  You couldn’t do this study in the US, because infection rates are currently high in different parts of the country from the first wave. You couldn’t do it in Wuhan, because rates there are low. Sadly, it looks like there is one candidate region, Lombardy in northern Italy, but they have other priorities right at the moment.

Because we don’t have direct comparative evidence on natural immunity, we’ve only managed to do two sorts of analysis. First, by looking at people who have had two sets of viral genome sequencing, we can be 100% sure that some people get reinfected. Second, by looking at immune responses of people infected early in the pandemic, we know that the biochemical markers of immunity are looking pretty stable out as far as we have data, which is only six months or so.

The same sort of problem happens for vaccine adverse reactions.  First, an important distinction: adverse events are bad things that happen after you got the vaccine; adverse reactions or adverse effects are bad things that happen because you got the vaccine.  You can observe adverse events; adverse reactions are a theoretical explanation.

In randomised trials, we know the people who did and didn’t get the vaccine were otherwise comparable, so we do know that any big differences in adverse events must be caused by the vaccine — they are adverse reactions.  The Covid vaccines have a high rate of mild to moderate short-term adverse reactions (including pain at the injection site, fatigue, fever, chills).  These only last a short time, and they are better than Covid, but they are not trivial.   There are also small numbers of serious adverse events in the trials, and we’ll hear more about the extent to which these are likely to be caused by the virus. Because so many people were in the trials, we know that any adverse reactions we haven’t seen in the trials must be rare (or long term). Against all of that, we know there are serious medical, social, and economic effects of not ending the pandemic, even here in relatively-secure New Zealand.

However, when we start vaccinating people there will be lots of other adverse events, because there are always adverse events.  If you gave a placebo injection to everyone in New Zealand there would be about 25,000 new cases of cancer over the following year — because 25,000 new cases of cancer is what happens in a typical year in New Zealand.  About 5800 people would die of heart disease, because 5800 people dying of heart disease is what happens in a typical year in New Zealand. About 140 people would be diagnosed with motor neurone disease and maybe 60 with Guillain-Barré syndrome, again, because that’s what happens in a normal year. If you give a vaccine injection to everyone in New Zealand, then on top of any real effects of the vaccine, the same things will happen, and some of them will look as though they are caused by the vaccine. Many of these would make good stories, and I’d hope the media will be careful what they do with them.

The best bet for distinguishing adverse reactions from adverse events that would have happened anyway is careful statistical analysis of big medical databases here (through the Centre for Adverse Reactions Monitoring) and even bigger ones in the US (the Sentinel Initiative), but even there it will be hard to tell whether a moderately higher rate of a rare event next year is coincidence or a side effect.  It’s quite possible that there will be real, rare vaccine effects, and we can be absolutely sure there will be spurious apparent vaccine effects.