Stats Chat

Various sources are telling us that coffee and tea consumption can lower the risk of dementia (the Independent is clickbaiting it to “the common drinks linked with reducing risk of dementia“, and 9News in Australia is even more extreme with The everyday act that could reduce your risk of dementia, according to Harvard study).  The subtext is definitely that caffeine is responsible for the decrease.

The research (paywalled, sadly) comes from two large studies of health professionals: the Nurses’ Health Study and the Health Professionals Follow-up Study.  You will have heard of them before; the participants have now been studied for 30-40 years and thousands of research papers written.  The rate of dementia was about 20% lower in  people who drank above-average amounts of tea or caffeinated coffee, but this reduction was not seen in people who drank decaf coffee. Since about 1 in 10 of the participants ended up with dementia, a 20% lower rate would mean preventing about two cases per 100 people. That’s not huge, but it’s not trivial either.  If you’ve been following medical news, it’s about the same reduction in dementia claimed for the shingles vaccine.

Unlike the shingles vaccine, which took advantage of a change in the rules that approximates randomisation, the coffee finding is correlations. Should we believe it?

It helps that the study is quite large (so random noise is less likely to give big spurious differences) and that participants’ coffee and tea consumption was measured from early on in the study. This study,  unlike small studies, would probably have been published whatever its findings, especially as the lead researcher is a Harvard PhD student.  It also helps that we know coffee and tea are pretty safe — many people who are suspicious of drugs and/or fun have tried quite hard to find harmful effects, with remarkably little success.

One negative fact, at least for the caffeine explanation, is the finding for tea.  The estimated risk reduction for a group of people who drank an average of 1 cup of tea per day is about the same as for a group who drank an average of 2.5 cups of coffee per day — and 2.5 cups of coffee is a lot more caffeine than one cup of tea.

I don’t think the data are all that convincing — this is really below the limit of what can reliably be done with long-term diet data — but we are not going to get better correlational data on coffee, and a randomised trial is outside the range of plausibility. If you drink tea or caffeinated coffee, it’s nice to think that you might be protecting your brain. If you don’t, there’s probably some reason you don’t. I’m not sure these data should change your mind.

A story in the NYTimes and also an unpaywalled story at 404 Media report a study of chatbots for medical advice, saying they are Bad and Not Good.

The research study is published in Nature Medicine. It’s a randomised controlled experiment, where people pretending to be patients were given a set of symptoms and some background health and lifestyle information.  These people were randomly assigned to talk to one of three large language models or just to use whatever information they would normally use at home for a health problem.

The three chatbots were chosen because they were able to recognise the medical situation in nearly every case and typically give appropriate advice  when directly given the same information that the pretend patients had.   When used in chat by non-medical people, though, the bots did much less well. One highlighted example was a scenario of a severe, sudden-onset headache, with sensitivity to light and a stiff neck.   In this scenario, the sudden onset and the stiff neck are both signs of a very serious event — the scenario was based on subarachnoid haemorrhage, a type of stroke.  One pretend patient emphasised the suddenness of the headache and got the correct advice (Ambulance! Now!), another didn’t mention the onset and got advice for a migraine or a tension headache (“lie down in a dark room”).  The bots weren’t any worse than unaided lay people, but they weren’t any better either.

You might think it’s a bit unfair to the chatbot that it wasn’t given all the information, but an important part of the training of doctors (as with statisticians and lawyers and plumbers) is learning what questions to ask when dealing with a non-specialist member of the public.  Obviously, even if you think there’s a barrier in principle to statistical algorithms making great art, there’s no barrier in principle to statistical algorithms learning to take adequate medical histories. They aren’t there yet.

Unless you have been living in a cave* you will probably be aware that the lead performer was one Benito Antonio Martínez Ocasio, a Puerto Rican rapper who performs as “Bad Bunny”.  The story is complicated a bit because of prediction markets.  The idea of prediction markets is that they can predict the future by letting experts get paid for integrating all the information about a question and betting correctly.

There are reasons to be somewhat skeptical.  The best way to make money out of a prediction market is to have inside information, but if that is too common then no-one sensible who doesn’t have inside information will bet and lose, so the incentives go away. It’s not clear how well they can work in practice.  On the other hand, two US companies, Kalshi and Polymarket, have discovered that gambling can be rebranded as a prediction market, with less regulation, lower minimum age for participants, and more favorable tax treatment.  It’s possible that sports gamblers will also help rescue prediction markets by providing uninformed money.

The other problem with prediction markets about complicated questions is deciding whether the event happened.  According to Business Insider, quite a number of people had bet on predicted whether Cardi B would do the Superbowl half-time show. You and I and probably many of those people might have expected this binary yes/no question to be easy to resolve. In fact, Kalshi and Polymarket resolved it in opposite directions.  The complication is that Cardi B (along with various other well-known performers) was there on stage, so that precise definitions are going to matter.

It’s possible that some fiendishly clever people predicted this confusion and correctly predicted that Kalshi and Polymarket would split on the question and extracted a big win. If so, go them! Otherwise, whether any hypothetical smart money won or lost would depend on the luck of which market it chose.

* “on Mars, with your eyes closed and your fingers in your ears” as the Simpsons’ Sideshow Cecil put it

Team Ratings for Delayed Games

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 84 matches played, 56 of which were correctly predicted, a success rate of 66.7%.
Here are the predictions for last week’s games.

Predictions for Delayed Games

Here are the predictions for Delayed Games. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 17

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 112 matches played, 91 of which were correctly predicted, a success rate of 81.2%.
Here are the predictions for last week’s games.

Predictions for Round 17

Here are the predictions for Round 17. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for the Super Bowl

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 284 matches played, 172 of which were correctly predicted, a success rate of 60.6%.
Here are the predictions for last week’s games.

Predictions for the Super Bowl

Here are the predictions for the Super Bowl. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Week 11

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 76 matches played, 50 of which were correctly predicted, a success rate of 65.8%.
Here are the predictions for last week’s games.

Predictions for Week 11

Here are the predictions for Week 11. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 16

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 105 matches played, 85 of which were correctly predicted, a success rate of 81%.
Here are the predictions for last week’s games.

Predictions for Round 16

Here are the predictions for Round 16. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

Team Ratings for Round 11

The basic method is described on my Department home page.
Here are the team ratings prior to this week’s games, along with the ratings at the start of the season.

Performance So Far

So far there have been 50 matches played, 37 of which were correctly predicted, a success rate of 74%.
Here are the predictions for last week’s games.

Predictions for Round 11

Here are the predictions for Round 11. The prediction is my estimated expected points difference with a positive margin being a win to the home team, and a negative margin a win to the away team.

• From the BBC “Vitamin D deficiency linked to hospital admissions”. This is from a large British study correlating vitamin D levels in the blood with hospital admissions for respiratory infections. You might say “Someone ought to do a clinical trial to see if giving people vitamin D reduces infections or if  it’s just correlation”. Someone has, here in NZ. Also, if you combine all the trials on this question you get an estimate of somewhere between a 10% reduction and a 4% increase. It’s still possible that it works in people with especially low vitamin D or something, but across a range of diseases randomised clinical trials of vitamin D have been robustly disappointing in comparison to correlational studies.
• From Derek Lowe, a post on extremely bad clinical trial conduct. This isn’t fraud by Big Pharma — if it counts as fraud, Big Pharma is a victim (along with some, but possibly not all, of the trial participants)
• “A very detailed map of Trump’s job approval” from Strength In Numbers (click to embiggen). “The basic idea is that we fit a model predicting approval based on demographics and geography, then use Census data to weight those predictions by the actual population composition of each area. Election results are used to calibrate estimates to sensible baselines, so we have a real-world check against our survey data. It lets us produce reliable estimates even for places where we only have a handful of direct survey respondents.”
  
• From the Guardian, Australian supermarket online prices per each can be very different from the in-store per-kg prices. Most dramatically, green capsicums were allegedly 50% more expensive per each.
• Bogus polls show increases in church attendance by young adults: Pew Research