Suppose there are two vaccines available. Suppose also that:
- Both vaccines have been shown to be 90% effective in double-blind clinical trials.
- Vaccine X has rather unpleasant side effects, which disappear after about 24 hours. Vaccine Y appears to have no short term side effects at all.
- Both vaccines are identical in all other relevant ways you can think of — cost, probability of long-term side effects, possibility of collateral benefits, etc.
Which vaccine do you prefer to receive, X or Y?
I’ll give my answer in a few days, or chime in sooner if someone else gives my answer first.
Hat tip to the ever-thoughtful Romans Pancs, who emailed me the relevant analysis.
Edited to add: Well, that didn’t take long. Jim Ancona nailed it in comment #2 — and expressed it so clearly that I feel no need to explain it in any words other than his.
I also want to commend the first of the two answers in Dave’s comment #1, which brings up another factor that hadn’t occurred to me. Of course (in Dave’s scenario) you won’t be the only one thinking this way, so it’s not clear that in equilibrium you’ll prefer X, but it is clear that some people will prefer X for Dave’s reason. Once enough of them have chosen X, you can be indifferent between X and Y.
I have two quick guesses:
If you believe that both vaccines are 90% effective, but perhaps susceptible to different strains/mutations of a virus for the remaining 10%, then you are better off getting the one with unpleasant side effects (X), since you can reason that others will prefer the one without side effects (Y), which helps protect you from the strains Y protects against slowing by the spread of them, and you are immune to the remaining strains from the unpleasant vaccine.
Either that, or there will be a longer wait to receive Y, since it will be in more demand, so X would give earlier immunity. In that case, you’d have to weigh the costs of the side effects against the benefits of earlier immunity.
Assuming that both were tested in the normal double-blind protocol with a placebo that itself had no side effects, I’d prefer X, the one with the unpleasant side effects. The reason: People in the X trial who suffered side effects would have been pretty sure they got the active vaccine rather than the placebo. Because they assumed they were protected, they were less careful (and thus more likely to be exposed to the virus) than those who received the placebo. Given that, if the trial showed 90% efficacy, the true efficacy would be even higher!
Well its obviously X. Otherwise there would be no point to the puzzle.
Maybe Y’s lack of side effects are a sign that it isn’t really as effective?
Seems like proper experimental design would be to use placebos that had some noticeable side effects for such experiments, to make all the subjects believe they had gotten the real thing. They wouldn’t know beforehand what the side effects would be, so something unexpected should probably be randomized: head ache, drowsiness, intestinal convulsions… In the long run, it might make it more difficult to get volunteers, though…
Another small point in X’s benefit – the side effects may act as a sign that the vaccine has been properly administered, or their absence as a warning that it hasn’t. Of course, non administration may show up as part of the 10% ineffective rates (especially of Y?) I’m the trials, but you could believe that administration technique might very between study conditions and clinical practice. Say it’s 10% extra who don’t actually the vaccine as intended when distributed en masse; with Y that will degrade the effectiveness to ~81%, while with X those without side effects can go back and get revaccinated.
I’m not sure I would have thought of the answer if I hadn’t read this thread, which Tyler Cowen linked to: https://twitter.com/eliowa/status/1331956943171313667
The Pfizer and Moderna vaccines had significant side-effects and a saline placebo, while the AZ/Oxford one seems to have had a Meningitis vaccine as a placebo. So in some sense, Pfizer and Moderna are more like X in Steve’s example (greater side effects in vaccine arm), while AZ/Oxford is like Y (similar side effects in both arms). Except even with the benefit of that, which should have raised measured efficacy, AZ/Oxford demonstrated a significantly lower efficacy.
It seems that Jim’s answer requires the test subjects to have faith in the vaccine. They did not know the results of the clinical trial while it was still ongoing. A vaccine can have side effects and still not be effective, so the magnitude of the effect on behavior is not clear to me. If this magnitude is small compared to the cost of the side effects, it can still be rational to avoid the side effects. This is especially true when the 90% rate is close to the upper bound.
Justin (#7): Points well taken.
I wanted to have a stab before reading the solution. I presume the answer must be X, but I can’t see why.
Ken B used to talk of “good puzzle theorem” or similar, which means you can’t pick out some unknown that was not in the puzzle or could reasonably be deduced. Since this an economics blog, I presume it must be to do with how other people act. We are given the choice, so we must assume that both are equally available. We can presume that most will choose Y. So the question is, how does having the vaccine that fewer people take give you an advantage?
Since we know nothing else about the vaccines, we could model them to be exactly the same, but X has an ingredient that gives you side effects. That would be consistent with the information given. In this case, I cannot see why having X would be advantageous. An answer that still gives X as preferred would be the most elegant.
Alternatively, the vaccines do produce a different immune response, so taking X would put you in a minority having that specific response. Say Y was preferred by 90% of people. Say X provided immunity to spike protein X and Y to spike protein Y. If Y spike mutated, then 90% of people would be susceptible and the disease would spread rampantly, but you would still be immune. If spike X mutated, then only 10% would be susceptible, and it would not spread. So then vaccine X would be preferred. That is my solution, but it is not elegant as it requires too many assumptions.
Phew, read the answer and it is elegant. I am kicking myself now. My initial thought that the data was consistent to both being the same but X had an ingredient that caused side effects is not correct if the puzzle is to be accepted. That vaccine would have had a lower efficacy result.
Oxford did use a meningitis as placebo for two reasons – the initial side effect was one, but also the placebo arm (pun?) was given something of value.
Saline injections can result in a sore arm, so the placebo group would not necessarily know they were placebo, but more serious and unpleasant side effects would give it away. From an influenza phase 1 trial of 60 subjects “Pain and tenderness at the injection site were the most commonly reported symptoms in both vaccine and placebo groups.”
The conclusion here does question the valididty of placebo controlled trials where the subjects are effectively unblinded. It is very hard to see how it could result in over-estimation of its performance, so little problem for the public except that a useful vaccine may not get approved. We are not going to get an ineffective vaccine from this problem.
Is “PorqueNoLosDos.gif” an option?
Your preferred answer #2 makes some assumptions that I believe are less consistent with our current regulatory reality than answer #1.
Your preferred answer (but not the puzzle itself) assumes a situation where side effects are so severe and frequent that they compromise the blinding of the phase 3 trial. However, if a vaccine has side effects severe and frequent enough to be obvious to all the participants who received it in phase 1 and 2 trials, that is sufficient to prevent the vaccine from proceeding to phase 3.
So IMO the proper way to read the puzzle is that Vaccine X is the Pfizer or Moderna one, which have “rather unpleasant side effects, which disappear after about 24 hours” in a small percentage of cases.
In a realistic scenario like that one, the second answer of #1 appears to be the correct one: for the duration of the pandemic, the choice will be between two vaccines for which we have no reason to believe there is any knowable difference in efficacy, and the best one to take is the one that’s available first, before herd immunity kicks in and while COVID risk is still high.
In the most likely real-world scenario, that will be dominated by logistical considerations. In the interpretation of your puzzle world consistent with present reality, there would be people who would refuse the vaccine with side effects, so the equilibrium would settle on people who don’t mind side effects as much and/or who are more worried about getting COVID taking Vaccine X when it becomes available to them.
Once vaccination has been made available to everyone who wants it, we’ll have herd immunity. At that point, Vaccine Y would be the rational choice unless we observe that the virus is still circulating as a cold and there is reason to believe that an escape mutation from Vaccine Y is possible that would result in full-blown COVID in Y-vaccinated individuals but would be protected against by Vaccine X. That seems immunologically unlikely, though, so the equilibrium between Vaccines X and Y would more likely be achieved via differential prices and quantities produced and distributed.
Scott Leibrand (#11):
if a vaccine has side effects severe and frequent enough to be obvious to all the participants who received it in phase 1 and 2 trials, that is sufficient to prevent the vaccine from proceeding to phase 3.
It is of course entirely possible that you’re far better informed about how these things work than I am. But I would be both shocked and horrified if the above were true. Does this mean that a vaccine that causes a 6-hour stomach ache followed by near-immunity from Covid would never make it to market?
> Does this mean that a vaccine that causes a 6-hour stomach ache followed by near-immunity from Covid would never make it to market?
I think the assumption underlying #11 is that any vaccine that causes such considerable side-effects in everyone will cause HORRENDOUS side effects in a significant minority and dangerous ones in a tiny minority.
That may or may not be the case, but is in any case irrelevant for the thought experiment under discussion here.
The thought experiment also works if only a large minority of the X group have noticeable side effects.
How large a minority we need to find a statistical effect depends, of course, on the overall sample size, and on how large the impact of altered behavior is.
Jim Ancona’s take isn’t even very theoretical. I and several people I know are in the Pfizer trial, and because they didn’t use an active placebo it’s pretty darn clear who got the vaccine and who did not. It will be interesting to see if the 95% number drops now that they’ve published it, because it’s kinda hard to not adjust one’s risk level when you know that you’re vaccinated with a 95% effective vaccine.
Another reason to prefer X is that the side effects are one way of knowing that your immune system actually responded, giving you information that you’re in the 90% rather than the unlucky 10%. Knowing this could easily be worth one mildly uncomfortable night and a couple days of sore shoulder.
Well, the “solution” seems to be confirmed by at trial participant’s experience as a J&J stage-3 volunteer:
“When I got a shot, I was nearly bedridden for 24 hours; it felt as if I had the flu, and the effect was far more pronounced than for any other vaccine I’ve had. If I got the placebo, I need a psychotherapist. Though I plan to remain generally responsible and not take too many incremental risks, given I’m only mostly sure I got a vaccine that is still unproven, I’m sure my assumption that I’ve been vaccinated will influence my behavior, and the behavior of anyone else who has had significant side effects from their injection too.”
See Marginal Revolution: https://marginalrevolution.com/marginalrevolution/2020/11/why-we-should-be-optimistic-about-various-vaccines.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+marginalrevolution%2Ffeed+%28Marginal+Revolution%29
Another factor is that the side effects may be similar to Covid symptoms. people suffering from these symptoms at the start of the trialmay not have been tested, as it was assumed it was side effects. If they were tested, then because we know there are false positives, we could get skewed information if many more vacccine group were tested than control group. This again would seem to reduce the apparent efficacy rather than falsely increase it, because we would expect to find more positives in the vaccine group. It seems possible to control for this by testing all participants after 7 days or so, regardless of symptoms. I don’t know if this was done.
#15. ” Another reason to prefer X is that the side effects are one way of knowing that your immune system actually responded,”
You are assuming that side effects means you have produced antibodies. That may not be the case. You may have reacted to the vaccine but not produced the relevant antibodies.
I agree that such reaction coud reasonably identify you in the vaccine rather than placebo arm, but I do not think it reliably informs you of the effectiveness of the vaccine in your case.
“You are assuming that side effects means you have produced antibodies. That may not be the case. You may have reacted to the vaccine but not produced the relevant antibodies.”
No, it is only necessary for side effects to make it *more likely* that your immune system has produced the relevant antibodies and/or whatever mechanisms are necessary for immunity.
From what I’ve heard, this is usually the case. Do you have any reason to believe that this is not the case, or even that this is not *reliably* the case, or it just a “We don’t know, it could conceivably not be the case”?
I think it may be reasonable conclusion now we have the efficacy results in, but at the start of the trial we did not have this. Assuming that side effects meant effective immunity would be a stretch at that point. But at the margin, it probably would increase confidence a bit.
My position was that it was not reliably the case.
Thanks for the fun puzzle (without seeing its statement, I never would have considered going w/ the side-effects option in real life).
The choice rationale I came up with is quite a stretch but points in the same direction as Jim’s and Dave’s: Whichever vaccine you choose, there’s a small chance it has unanticipated severe widespread side effects which take longer to manifest than the study length, for which intensive care is both essential and effective. If you develop such side effects, you’d be better off having chosen X than Y, since you’ll be likelier to have ICU access.
As an aside to the puzzle, but still germane, I’d like to know how they confirm 90% (or other) effectiveness.
They don’t actually dope anyone with the virus, so how do they recognize those protected by the vaccine, distinguished from
those who were never infected? Or, recognize the ones infected, and vaccine protected, from those infected who were naturally asymptomatic?
Given the existence of (relatively cheap) antibody testing, we can assume that the trial is only single-blind. Anyone who wanted to know whether they get the vaccine instead of a placebo could find out.
Therefore, by the reasoning of this puzzle, if the trial shows an efficacy of 90%, we can assume it’s a lower bound.
#22. It was only possible because there was a pandemic going on. They vaccinated 30,000, gave placebo to 30,000 (or some large number) and left them to go about their lives whilst monitoring them. Because the infection was prevalent, some of them got infected. When they unblind the trial, they discover that 90% of the infected were in the placebo group. Hence 90% effective. I think it is a bit more complex, but that is the essence.
They also measure antibodies produced, which is a good indication but cannot by itself prove efficacy.