Nationwide non-representative sample and COVID

Time magazine has a new story Popular Heartburn Drugs Linked to Heightened COVID-19 Risk, and it’s on Reuters, so it’s going to spread.  Here’s the preprint, and here’s the press release, which comes from the American College of Gastroenterology, the major professional association in that field and the journal publisher.

The drugs in question are proton-pump inhibitors (PPIs). These inhibit the cellular pumps that push hydrogen ions into the stomach fluid, making it acid. If your stomach acid isn’t acid enough, it’s plausible that the coronavirus could survive and get into your gut, where it will find cells with the receptors it needs to attack.  So the theory is not at all unreasonable.  Kiwis will remember that Michelle Dickinson and Siouxsie Wiles both emphasised the potential risk of coronavirus getting in through your mouth.

The press release says “We have now tested the hypothesis in a rigorous study of more than 50,000 Americans and found it to bear out, albeit in an observational study.”  As you’ve probably guessed, I’m not buying that.

In the research paper, the study is described as

we used data from a population-based, online, self-administered survey of Americans collected from May 3 to June 24, 2020. We collaborated with an online survey research firm (Cint) that recruited a nationwide, representative sample based on U.S. Census data on age, sex and region

That could work, provided it really was a representative sample, and provided you could get a good handle on the other reasons why taking a medication long-term might be correlated with getting a positive COVID test.

The sample of 54,000 people included 3,386 (6.4%) who reported having had a positive COVID test.  As of June 24, only 2.35 million people in the US had tested positive for COVID. There are about 250 million adults in the US, so even if all the positive tests had been in adults, that’s less than 1%.   The sample has over six times the national average for positive COVID tests. Regional bias wouldn’t be enough to explain this: even in New York City, only about 2.5% of the population has tested positive.

Even with this very high rate of COVID, the sample is missing a lot of cases. 95% of people with positive tests reported symptoms. That’s not surprising, as symptoms are how you get tested, but it does mean the ‘control’ group will contain at least another few thousand cases who didn’t get tested, or got tested at the wrong time.

If you just look at the PPI data, 75% of people with a COVID diagnosis were regularly taking PPIs, as were 30% of the other participants.  In a population-based health survey that makes serious efforts to be representative, NHANES, only 8.7% of participants reported taking PPIs.

So that’s the overall representativeness. The other thing to worry about is that regularly taking a medication and getting access to a COVID test may well be correlated, so you’d worry about whether the COVID cases were different in other ways — remember, we already know sampling was far from representative.

A few differences jump out. The COVID cases were more than 8 times as likely to have a household annual income over $200,000: nearly two-thirds of them did. That’s despite them being less likely to have a college degree.  More than two-thirds of the cases reported Latinx/Hispanic ethnicity, but only 3.5% were non-Hispanic Black. Only 10% of cases were from the Northeast of the US, where the epidemic has been worst until recently;  nearly 70% were from the South.  The cases were much less likely to report a diagnosis of gastroesophageal reflux disease, a primary reason for taking PPIs.

The researchers did make some efforts to adjust for the non-representative sampling.  The relative risk of COVID in people taking PPIs daily or less often went down from nearly 8 before adjustment to 2.15. For people taking PPIs twice a day, the relative risk went down from 5.7 to 3.7.  However, the researchers didn’t use a lot of the variables in this adjustment, and they didn’t try to reweight the data to known national proportions for age, race/ethnicity, gender, and region, a fairly standard technique in national surveys (eg, election polls, market research).

Given the clearly non-representative sample, I don’t think the evidence could be convincing without a lot more exploration of the biases (and quite likely not even then).  As a drug class, PPIs have form for this: there have been other conditions correlated with PPI use in initial reports, where the correlations go away with better-quality data.

I’m not saying the study shouldn’t have been done, though I think it should have been analysed better. But the journals shouldn’t have pushed it out into the media with an urgent pre-publication press release, especially when even the authors won’t publicly claim it’s good enough evidence to change treatment.  The true take-home message of this study, apparently, is that it gives doctors an opportunity to stress the importance of hand-washing.

If I were more cynical than I am, I would have pointed out much earlier in this post that one of the authors of the study  is co-Editor-in-Chief of the journal, and is quoted in the press release with that title.