There are (at least) two ways to test the efficacy of a vaccine. The Stupid Way is to administer the vaccine to, say, 30,000 volunteers and then wait to see how many of them get sick. The Smarter Way is to adminster the vaccine to a smaller number of (presumably much better-paid) volunteers, then expose them to the virus and see how many get sick.
A trial implementing the Smart Way is getting underway at Imperial College London. In the United States, we do things the Stupid Way, at least partly because of the unaccountable influence of a tribe of busybodies who, having nothing productive to do, spend their time trying to convince people that thousands of lives are worth less than dozens of lives. Those busybodies generally refer to themselves as Ethicists, but I think it’s always better to call things by informative names, so I will refer to them henceforth as Embodiments of Evil.
Last night, while I was attempting to calculate the amount of damage that these Embodiments of Evil have caused, I was interrupted by a knock on my door. It turned out to be a man from Porlock, who wanted to consult me on some mundane issue. At first I tried to turn him away, explaining that I was in the midst of a difficult calculation and could not be distracted. But my visitor brought me up short by reminding me that the economist’s job is not just to lament bad policies, it’s also to figure out ways to circumvent them. So we put our heads together and this is what we came up with:
First, design a vaccine trial that is, to all appearances, set up the Stupid Way. We vaccinate people, we let them go their own ways, and we track what happens. But we add one twist: Any volunteer who gets sick after being vaccinated receives an enormous payment. Call it something like “Compassionate Compensation”.
Here are the advantages:
- Surely the Embodiments of Evil can’t object to compassion! This ought to sail right past them.
- But: If the payment is high enough, then everyone we vaccinate will, on his or her own, go out and try to become infected — which is just as effective as if we tried to infect them ourselves.
I thought that was pretty good, until my visitor pointed out that it’s even better than that. Here’s why:
Suppose you’ve already got Covid. Then once our study gets underway, you are going to find yourself in high demand. Our study volunteers will pay to hang out with you! This gives you an incentive to hoard your capital (i.e. your viruses) by infecting only paying customers. After all, you don’t want to go around giving something away for free when there are people lining up to pay for it! Even more importantly, anyone you infect immediately becomes your competitor, which you really want to guard against.
So one side effect of our study is that infected people will be incentivized to quarantine rigorously (with exceptions of course for contact with the recently vaccinated volunteers). That’s a pretty great side effect.
So let me start over: There are (at least) three ways to test the efficacy of a vaccine: The Stupid Way, the Smarter Way, and the Sneaky Smarter Way. I am cheerfully on board with anything that’s not Stupid.
the way I understand it, vaccine testing at this stage – outside the lab on the general population – is not about figuring if the vaccine works. they should have a pretty good idea about that. it is about seeing if the vaccine has any small probability, negative side effects. so you need lots of people and I can imagine you actually need more non-affected people (because of the low incidence of the virus you are actually going to vaccinate more healthy than ill people when you’re going to roll it out for real).
We completely agree on the subject of “medical ethicists”.
The current fashion in medical ethics is unbelievably cruel and stupid.
Human suffering caused by restrictions on medical research appears to have virtually no weight in the ethical calculus.
Even setting economic considerations completely aside, the restrictionist approach to human and animal testing, by slowing down medical progress (or by completely stopping it in some fields) causes immense human suffering and the loss of vast human potential.
That test works great if the vaccine has 100% effectiveness. But let’s say it only has 50% effectiveness. How are you going to tell if it works? You have no idea what percentage of people altered their behavior to try to sick, or how hard they altered their behavior. Suppose the chance of getting sick is 1% if you don’t change your behavior. If you vaccinate 100 people and 30 get sick, did the vaccine work or not? If all 100 had sick people cough in their face, then it worked great. However if only 30 of them had sick people cough in their face then it didn’t work at all. How would you monitor people’s change in behavior?
I’ve spent a considerable amount of time thinking about how to test pharmaceuticals and vaccines more effectively.
There’s some things speaking for “the stupid way” of vaccine testing, and some problems with your approach.
First of all, you have to inoculate a very large number of people anyway just to figure out if the vaccine has rare side effects.
Secondly, if we take your approach of paying people large amounts of money for getting sick, we won’t get realistic results.
You can theoretically overwhelm ANY vaccine protection and cause an infection, it’s just a question of how much virus you hit the patient with.
Your approach would yield the following two very undesirable consequences:
1. Inoculated people would expose themselves to increasingly huge viral doses (many orders of magnitude higher than natural doses) until they defeat their immune protection.
The vaccine would fail the trial.
2. Someone would have to produce massive amounts of virus. We would incentivize the private sector to engage in bioweapon manufacture, basically. That is not a thing we want to do.
Albert: The idea is to set the payment so high that we can safely assume that approximately 100% of the subjects had sick people cough in their faces.
TL/DR:
If you pay people large amounts of money for a certain outcome, they will tend to find surprisingly creative ways to give you that outcome, whether you want them to or not.
There’s a reason economist don’t do experiments (accept in their dreams). They’re not very good at designing them.
Another way is to do a controlled trial on a sample of people who are highly likely to come in contact with an infected person, say people in families where there is a person with covid. I believe that is how some trials do it. Wasting time on people who are not likely to become exposed is useless and medical researchers know that and try to avoid it.
There is another way. Once they are convinced that the vaccine is reasonably safe, they could approve it for the general public, and do a followup study on the early customers. Those early customers would have to accept the risk and report results. The rest of us would get data on how the vaccine affects typical customers.
Paying people to get infected seems unnecessary and expensive.
The reason that some Ethici– excuse me, Embodiments of Evil object to the reasoning that says “Hey, we can do whatever we want to volunteers, as long as we pay them enough!”, is that it reduces pressure on society to alleviate poverty in more humane ways, for the people who are desperate enough to sign up for the cash. Or, as one blogger put it, “Hey, why should you get welfare when you still have two kidneys?”
What can I say? As is oft the case another enjoyable post from “sneaky smart” Steve :-)
>> Bennett Haselton
That is a good point, but it doesn’t really apply too well in Western countries other than the US.
And the problem is that the Embodiments of Evil also object to using unpaid volunteers.
I think vaccine trials are trying to figure out both 1) Does the vaccine prevent someone getting Covid? and 2) Are there side affects from the vaccine? To figure out the answer to 1, vaccinating a relatively small number of people and then exposing them to Covid makes sense. To figure out 2 requires vaccinating large numbers of people.
The problem with the large payment for those who get sick is first, defining “sick,” since there’s a continuum of symptoms; and second, given any threshold, it counters the selection of volunteers away from those who are likely to get the mildest symptoms, i.e. people under 30 who are otherwise healthy. I would think those are the ones who should (and would) be the volunteer guinea pigs here.
Here is a sneaky plan that got stopped — give the vaccine to shopping mall Santas!
https://www.wsj.com/articles/health-agency-scraps-coronavirus-ad-campaign-leaving-santa-claus-in-the-cold-11603630802
(article is paywalled)
>> That is a good point, but it doesn’t really apply too well in Western countries other than the US.
Do you mean that other Western countries don’t have people who are so desperately poor that paying them for an experiment would qualify as exploitation? (I’m not sure that’s true but I get what you mean.)
On the other hand, this reasoning would still allow you to pay people to participate in an experiment, as long as they meet income verification requirements to show that they are not too badly off already. This removes the problem of “Hey, why should you get welfare if you still have two kidneys?”
I suspect that is the reasoning underlying the intuition of people (like me) who would feel a lot less icky hearing that someone who makes $100,000/yr sold a kidney for $10,000, than hearing that someone who makes $15,000/yr sold a kidney for $10,000. Even if the second person needed the money more.
> Do you mean that other Western countries don’t have people who are so desperately poor that paying them for an experiment would qualify as exploitation?
Of course other Western countries have poor; but these generally have access to health care, education for their kids, and housing.
> as long as they meet income verification requirements to show that they are not too badly off already.
Why would you need that? Welfare is generally means tested.
People who receive money from organ sales or participation in challenge trials would see their welfare cut.
If they nevertheless decide to participate in those trials, that is their choice. Some doubtlessly would, in order to give back to society for what it has given to them.
Though of course we wouldn’t want a “kidney sale” as an eligibility for welfare requirement.
The major problem with the proposal is that the trials are blind. We have a control group, and nobody knows if they are in the control or treatment group. It is probably not a good idea for the control group to deliberately try to get infected. If you avoid the control group you have weakened the trial significantly. There are good reasons to have blinded trials.
There have been a great many horrific abuses of medical research. The response has been a set of restrictions to prevent similar abuses. As with any set of restrictions, the line will be drawn in the wrong place for many individual trials.
We probably would not agree with a challenge trial for Ebola, for example. It is OK for Covid-19 because for healthy, young people the risks are small. It is plausible that you could find some people desperate enough to sign up for an Ebola trial if the payments were high enough, but it would be considered by many to be unethical. I presume that Steve would consider it to be fine if the volunteers were not coerced in any other way except offered large amounts of money.
The usual standard for trials is “informed consent.” That “informed” is something of a slippery principle. It has proved quite easy to persuade people without actually lying to them and obtaining consent that is almost certainly not fully informed. We could ameliorate this by using neutral people not involved in the trial to explain, as they would have no interest in mis-informing. It does still leave the door open to abuse. In a similar calculation to “it is better to allow nine criminals to go free than to convict one innocent person”, it has been considered by many that avoiding abuse is more important than allowing some useful trials to go ahead.
The challenge trials at Imperial/Royal Free will not replace phase 3 trials, but could accelerate the development of vaccines to enter phase 3.
Phase 3 trials have 2 purposes. Is the vaccine safe and is the vaccine effective? For the first you need the large numbers anyway. For the second you need people to behave normally.
When the phase 3 trials started, the number of infections had dropped in many countries, particularly Europe. There was concern that there would be too few infections to measure the effectiveness quickly. Unfortunately, that problem has gone away.
The AstraZeneca/Oxford trial was put on hold following a patient becoming ill. This could be an adverse reaction, or it could be coincidence. Trials do not occur in a vacuum. There is a strong anti-vaxx movement that will seize on any hint of adverse reactions to reduce vaccine uptake across the board. This must be taken into consideration. It may be that a cost benefit analysis for Covid-19 would allow a slightly higher adverse reaction rate than for other vaccines, but this would reduce vaccination rates for all diseases, so the cost/ benefit calculation cannot only consider Covid-19.
“People who receive money from organ sales or participation in challenge trials would see their welfare cut.”
I understand — but then it seems like:
1) If the person would make more money than they would lose in welfare benefits, then that still affects the debate over welfare (“If these people want *more* money, why aren’t they selling their kidneys?”)
2) If the person’s welfare benefits go down by exactly the amount of the kidney sale, then they are de facto doing it for free anyway.
> 1) If the person would make more money than they would lose in welfare benefits, then that still affects the debate over welfare
I don’t think that would be the case.
Prostitution is quite legal in Europe, and yet no one expects women (or men) to prostitute themselves rather than go on welfare.
So just a thought. But if there are already studies going on in the UK surely the easiest solution for whoever is funding/conducting the study is just to do it in the UK or somewhere else where challenge trials are legal.
I could be missing something obvious and I get that it’s helpful to think about these things when the “do this somewhere else” option is eliminated. But that seems a few orders of magnitude simpler (and hence less buggy) than the sneaky way?
Steve, out of curiosity, roughly how much Compassionate Compensation would need to be offered in order for you to volunteer for a vaccine trial and then get sick people to cough in your face?
Ted: A great deal more than it would have when I was younger.
#23. “A great deal more than it would have when I was younger.”
I presume that is a comment on the increasing risks with age, but it could be a comment on the value of a statistical life, or both.
#21. The misunderstanding in the post is that the challenge trials are to improve phase 2 and the large trials are phase 3. The challenge trials are not intended to replace the large phase 3 trials, so the stupid way and the smart way are comparing apples to oranges. Phase 2 usually assesses success by measuring antibody production in the subjects. It is “better” to measure actual resistance in a challenge trial as well as antibody levels because antibody production may not be the whole story.
The phase 3 trials are to investigate efficacy *and safety*. Thus large numbers still need to be tested.
That does not of itself mean the smart way could not be a better way to do phase 3, but it is wrong to use the challenge trials as a counter example.
So how would we do phase 3 the sneaky way? We still need to inject 30,000 people, plus controls, to investigate safety. We need to wait until any adverse reaction could reasonably be expected to have occurred and compare control to treatment groups.
For efficacy, in the standard phase 3, we wait until sufficient numbers have become infected, then we can compare control and treatment groups. Given the current prevalence, this will not take as long as was once suspected. The sneaky way is an attempt to speed up this stage by encouraging contact, so the required number of infections are obtained sooner. The problem is that we have to abandon the control group, and the behavior of the subjects is not the same as would occur in people once the vaccine is rolled out. Therefore we would not necessarily be able to conclude if the vaccine was effective in normal use. The sneaky way unfortunately undermines two key aspects of the trial.
Trials are fraught with ethical difficulty. During the AIDS crisis, the only way to get treatment was to enroll on a trial, and then you might get placebo. Subjects were sharing their medicines because they knew that some had placebo, and the patients reasonably considered that some dose of a possibly effective medicine was better than no dose for a disease that will kill you. This obviously invalidates the trial. There was successful lobbying to use different methods without controls to test medicines. One of the heroes of setting up the trials was Dr Fauci. The authorities initially approached it with a blinkered mindset of how trials are done. Engagement with the patients allowed changes to be made to the trials structure that were more successful because the victims were engaged, and they had more access to treatment.
We need to continually reconsider how we do such trials. Each situation is different. However, the sneaky method suggested here is not the answer.
Incidentally, there is a reason the largest and most successful Covid trial, the RECOVERY trial is occurring in the UK. It is because we have a single healthcare system. Coordination of trials of this scale is much harder in the US. There are downsides and upsides, but this is one of the upsides. Unfortunately, another reason is that we have had one of the largest number of patients per capita, which made recruitment easier. USA has recently overtaken UK in deaths per capita, to become the 9th highest in the world. UK is 10th. USA is 14th in cases per capita.
I am unsure why the fact that your visitor was from a coastal village in Somerset was significant enough to mention.
According to Wikipedia and using round numbers, 16M Americans served in all branches of the armed services in WW2, and 400K were killed. This is a warfare-participation fatality rate of 2.5%.
https://en.wikipedia.org/wiki/World_War_II_casualties#Total_deaths_by_country
The all-ages COVID-19 infection fatality rate (IFR) appears to be around 0.7%. The IFR for younger people is most likely substantially lower than 0.7%.
https://reason.com/2020/07/23/there-is-more-than-one-covid-19-infection-fatality-rate/
If young, well-paid volunteers are recruited for challenge vaccine trials, they would be at least an order of magnitude safer than the not-always-voluntary U.S. participants in WW2. I presume most U.S. ethicists approve of the U.S. participation in WW2. Why would they not approve of challenge trials?
Greg: I think you’re new here. Welcome, and thanks for this great post. In fairness, the ethicists might reply that winning WWII was a lot more important than shaving several months off the pandemic. But that’s an argument about how much we should be willing to pay for volunteers, not about whether we should be glad to find takers at the price we’re willing to offer.
Focusing on the vaccine trials is something of a red herring, I think. The challenge trials will probably not get us there that much quicker, but if healthy individuals can be paid to do so, then in my opinion it should be done, as it is in the UK.
The bigger problem seems to be that there has been no serious effort to figure out how to minimize fatalities. At the start we did not know that much about Covid-19, so some sort of a lockdown was probably the only thing to do. It might have turned out that the disease was much worse for young people than it has turned out to be, and that was too high a risk.
Now we do know, so it would seem that all our efforts should be directed towards ensuring the vulnerable do not get infected. There does not seem to be policy initiatives in this direction. Maybe they have been thought about and rejected, but we have seen little discussion of options. In very broad terms, this would be how would we keep the old separated from the young, and let the young carry on as they wish.
The Great Barrington Declaration advocates for this, but does not offer any way to do it. The general reception is that it simply cannot be done, so we should forget this approach. So far, it is true that we have no options for how to do this. But given the massive cost, can nobody think of a way to use some of these resources to achieve it? Obviously it cannot be done for free, but maybe it could be done at less cost? To be fair, I cannot think of a way to do it off the cuff, but the declaration would carry a bit more weight if there were some suggestion of how to do it. As it stands, if we allow those at less risk to carry on we know that the vulnerable will get infected.