I hope this is the last post I write about Covid-19. With luck, the pandemic will wind down in the United States in a few months, and, if we are generous with our doses and money, its spread will be curbed greatly in the rest of the world as well. I should, then, be able to return to my regularly scheduled programming: the content you know and love about how tech companies suck, why the rent is too damn high, the fecklessness of Democrats, the callousness of Republicans, the rottenness of the media, etc. The pandemic was, in some ways, a year-long distraction from our country’s longstanding, systemic problems (but, of course, in other ways, it only helped heighten awareness of those problems). I am deeply grateful to science and our government-backed pharmaceutical companies that it is not going to be a several-year-long distraction.
I’ve been somewhat alarmed to read essays and articles about Covid that I believe to be scientifically illiterate and statistically innumerate. Normally, I would expect this sort of nonsense from the right-wing: masks don’t work, herd immunity is right around the corner, we should let the virus wash over us, the cure is worse than the disease, etc.
But there is a similar strain of nonsense on the center-to-left-wing side of the political spectrum. So as not to turn this into a 5000 word essay, let’s focus on one specific set of claims, about vaccine efficacy.
The first excerpt below is from Lawyers, Guns, and Money (LGM), a left-of-center blog run by lawyers who, naturally, opine about all sorts of stuff, not just the law; the second is from CNN:
If somebody had told you last spring, when COVID-19 was basically shutting down normal life all across the world, that in less than a year hundreds of millions doses of a vaccine that was 100% effective in preventing serious illness from the disease would be available, so that by the summer of 2021 pretty much any adult in the USA who was willing to take it would be vaccinated, what would you have said?
I mean we’re experiencing something very close to [the] absolute best-case scenario here are we not? Sure the rollout could have been much better — too bad that government-wrecking fascist grifters were in charge, maybe we should avoid that in the future — and there are still worries about highly contagious new variants, people who will refuse the vaccine, and so forth.
But the message from public health officials, and the framing of it by the media, strikes me as bizarrely dour. It’s basically something like, “don’t expect to get anything like your normal life back for a very long time, because the vaccine isn’t a magic bullet.”
I get that you want people to continue to exercise caution, wear masks in public, etc., but at some point all this reflexive pessimism becomes counterproductive.
We needed medical science to hit a grand slam while we were down by three runs in the ninth inning, and it did. How about a little more joy in Mudville?
On Wednesday, the FDA released an analysis of vaccine clinical trial data submitted by Johnson & Johnson, saying the vaccine met the FDA’s requirements for emergency use authorization.
According to that analysis, the single-dose vaccine showed about 66% efficacy globally against moderate to severe/critical Covid-19 cases 4 weeks after inoculation. But in the United States, the vaccine showed a 72% efficacy. Against severe forms of the disease, it’s even higher, offering nearly 86% protection.
And that’s what people should be focused on, experts say.
“That’s probably the most important part of this vaccine or any of the Covid vaccines is the protection against hospitalizations and severe disease and dying,” said Dr. Paul Goepfert, professor of medicine in the Division of Infectious Disease at the University of Alabama at Birmingham and director of the Alabama Vaccine Research Clinic.
“If we could prevent people completely from dying or getting hospitalized with Covid, we don’t have a problem anymore,” Goepfert said.
Dr. Wen said what really matters is a vaccine’s ability to prevent severe disease, and beyond that, disease that’s severe enough that leads to hospitalization or even death. She pointed to Johnson & Johnson’s vaccine trial, in which there were no individuals who had a severe enough case of Covid-19 that they ended up in the hospital or on a ventilator or died.
“In this regard,” she said, “all of the vaccines are essentially a hundred percent.”
There are a few reasons to doubt the claim that all vaccines are “a hundred percent” effective at preventing severe disease, as opposed to mere illness. (I do agree with these authors that it is the severe health outcomes that we should focus on; I would be perfectly happy if, after being vaccinated, I simply suffered a mild version of the flu.)
First is that the clinical trials were never designed to evaluate the efficacy, let alone effectiveness, of the vaccine at preventing severe health outcomes, including hospitalizations and deaths. This has been known for a long time, but is, for some reason, rarely mentioned in media or other popular accounts. I’ll quote from a piece by Drs. Peter Doshi and Eric Topol, titled, “These Coronavirus Trials Don’t Answer the One Question We Need to Know”,
According to the protocols for [Moderna, Pfizer, and AstraZeneca’s] studies, which they released late last week, a vaccine could meet the companies’ benchmark for success if it lowered the risk of mild Covid-19, but was never shown to reduce moderate or severe forms of the disease, or the risk of hospitalization, admissions to the intensive care unit or death.
To say a vaccine works should mean that most people no longer run the risk of getting seriously sick. That’s not what these trials will determine.
The Moderna and AstraZeneca studies will involve about 30,000 participants each; Pfizer’s will have 44,000. Half the participants will receive two doses of vaccines separated by three or four weeks, and the other half will receive saltwater placebo shots. The final determination of efficacy will occur after 150 to 160 participants develop Covid-19. But that is only if the trials are allowed to run long enough. Pfizer will look at the accumulating data four times, Moderna twice and AstraZeneca once to determine if efficacy has been established, potentially leading to an early end to the trials.
This problem arises in many experimental studies. If designed intelligently, these studies use a “power analysis” to ascertain the number of observations that the trial needs to make a definitive conclusion. (Obviously, this should be done before the trial begins, not after it has ended.) The power analysis reduces the chance of a “type II error” to an acceptably low level. A type II error is a “false negative”: in this context, it means a determination that the vaccine does not work, when it does in fact work. We do not want to spend millions of dollars on a vaccine trial and collect evidence insufficient to prove the desired outcome, if it is indeed true.
The problem is that the power analysis depends sensitively on that desired outcome. If the desired outcome is “a statistically significant difference in detected cases between the vaccine and placebo groups”, the number of observations needed is much lower than if the desired outcome is “a statistically significant difference in severe cases/hospitalizations/deaths between these two groups”. A study that is “adequately powered” for the former is likely severely underpowered for the latter.
One (slightly imprecise) analogy is to analysis of polling data. The margin of error for the overall poll result (e.g., the margin between Biden and Trump) is much smaller than the margin of error for the result for any subgroups (e.g., the margin between Biden and Trump for Asian voters). But that has not stopped many political observers from reading deeply into these subgroup trends, even when those trends could have been caused by statistical noise alone. Quite similarly, in the case of severe cases, hospitalizations, and deaths, the numbers under consideration are minuscule, and any related conclusions should be treated as uncertain. In the Pfizer study, as an example, there were ~20000 people in each group; some ~170 in the placebo group who were infected, ~10 in the vaccine group who were infected; 9 people who had a severe case after the first placebo dose, and 1 with a severe case after the first vaccine dose. The last set of numbers is tiny. Unsurprisingly, the authors couch their conclusions in appropriately careful language, saying, “The severe case split provides preliminary evidence of vaccine-mediated protection against severe disease”.
Whether we should believe that the Pfizer vaccine greatly reduces severe cases of Covid is a separate issue: there are many things which we should believe but for which statistics provides no definitive answer. That being said, claiming that “100%” of bad outcomes are prevented by all vaccines (not just Pfizer) seems ludicrous to me. In fact, the first real-world study of vaccine effectiveness, in Israel, reported that “nine people who were fully vaccinated died from Covid-19, compared to 32 people who had not yet been vaccinated.” Obviously, this is substantial protection, and I would rather have the vaccine than not have it. But 9 vs 32 is nowhere near 100% efficacy, and people who should know better should stop claiming that it is. (It is also unclear to me whether even the Israeli study of 600,000 paired individuals is adequately powered to answer this question about severe health outcomes, although I suspect it is, at least for severe cases + hospitalizations.)
(At the top, I quoted LGM’s ridiculing of people claiming that the vaccine isn’t a “magic bullet”. One of the experts advising the Israeli government said, after viewing the results of this study, that “these vaccines are not a force field around you that negates the chance that you will have an illness or that you will have a severe illness”.)
The other issue, besides the statistical/epistemological ones mentioned above, is that even the definition of a “bad” or “severe” health outcome is somewhat ambiguous. It is obvious that ending up in the hospital or, even worse, dying, is a bad health outcome. But, at least anecdotally, I know some people whose condition did not warrant hospitalization, but had a scary episode nonetheless. My boss tried to power through his infection, not taking any sick days from work. A few days afterwards, he came out of one meeting unable to breathe and thought he was suffering a heart attack. Mercifully, this episode passed quickly, and he was not hospitalized. But, since then, he has taken several days off work to undergo heart tests and speak with his doctor. I also know a relative of a friend who had a relatively mild case but completely lost her sense of smell, and, with it, much of her enjoyment of food and other quotidian pleasures. It has not returned yet (3 months have elapsed since the initial infection). There is also the issue of “Covid long-haulers”. There seems to be substantial evidence that even mild cases of Covid can later spawn severe symptoms. The New York Times reported that “A study that analyzed electronic medical records in California found that nearly a third of the people struggling with long Covid symptoms — like shortness of breath, cough and abdominal pain — did not have any signs of illness in the first 10 days after they tested positive for the coronavirus. Surveys by patient-led groups have also found that many Covid survivors with long-term symptoms were never hospitalized for the disease.”
We simply don’t know, at least yet, whether and to what extent vaccines mitigate these problems. As Drs. Doshi and Topol mentioned, the initial vaccine trials did not focus on these issues; the subsequent real world studies are very new and insufficient time has elapsed to examine the “long Covid” problem. My suspicion — and it is just a suspicion — is that vaccines will provide good but not perfect protection against these non-hospitalization-related, but still “severe,” outcomes. (And I further suspect that the vaccines with higher efficacy will do a better job in this respect than those with lower efficacy, although it is worth bearing in mind that vaccine efficacy numbers are not directly comparable across studies, because the conditions of those studies are often materially different.)
(There is also the separate question of what happens to vaccine efficacy and effectiveness when Covid-19 keeps mutating, as it has in Brazil and other parts of the world. I don’t have time to dive into this topic, but it is certainly a matter of concern.)
Regardless, though, I would contend that it is scientific malpractice to make the claims that LGM and large swathes of the center-to-left media have been making. Vaccines are not 100% effective against bad health outcomes. Differences in the efficacy of the various vaccines might matter, and it is reasonable to want a vaccine with a better efficacy. We are undercounting bad health outcomes, and it is not just hospitalizations and deaths that matter. And, most of all, it is not “vaccine Eeyorism” or a sign of insufficient fealty to “science” to argue these claims, particularly when many of the people writing these rosy stories are both out of their depth and, oddly, unembarrassed by that fact.
It seems to be the case that much of Covid reporting and analysis, even by people who aren’t right-wing loons, is colored by political considerations or other prejudices. LGM, for instance, wants to tell a story about how many people are being unnecessarily pessimistic, and how we should simply clap harder for science and Joe Biden. It raises the thorny question of who we can trust, particularly when we don’t have oodles of time to investigate every issue ourselves (nor do we necessarily have the technical expertise). In the meantime, I hope, our liberal lawyer friends will either pick up a statistics book, or return to posts about how John Roberts is ruining the country.
A Humble Politics Blog 