Uncertain Risks
This piece was written by Profs. Alyssa Goodman and Immaculata De Vivo, Harvard University and Radcliffe Institute for Advanced Study, and submitted for publication with the tagline: We know how to express risk and uncertainty with numbers, but people don’t always take practical actions based on those numbers. The COVID crisis brings this contrast into sharp focus.
“One shot.” Any fan of the classic film, The Deer Hunter, will recognize that two-word phrase, and its double-meaning. One shot to kill a deer is humane, and one shot in the game of Russian Roulette is all that’s needed to kill its player–or not. The odds of shooting a deer with a single shot while hunting depend on the skill of the hunter, the weather, the quality of the rifle, and more–and so are quite hard to estimate with great certainty. The odds of being shot in the head in a single round of Russian Roulette are, on the other hand, very easy to estimate--they are exactly 1 in 6. One bullet loaded into a six-chamber gun barrel that’s then spun to a random stopping point and then fired by the player at their own head can be expected to blow a hole in the player’s head exactly one in six times. This kind of certainty when assessing risk is extremely uncommon. Life is usually much more like hunting–with many factors influencing risk, so that estimating odds is hard, and uncertain.
Life is filled with decisions about risk in the face of uncertainty, so misunderstanding the words “risk” and “uncertainty” can be dangerous. These terms are commonly used In everyday life, where we talk about activities being “safe” or “risky”, and we talk about being “sure” or “uncertain,” but we don’t usually attach specific numerical odds to such statements. Instead, psychologists like Dan Gilbert tell us, we humans typically group likelihood into three categories: it will happen; it won’t happen; and it might happen. Percentages and numerical odds are relatively new constructs for us, so using the word “might” to capture everything between “will” and “won’t” is understandable. But, in some cases, numbers can--and should--mitigate fear.
Our minds are not just bad at estimating and understanding nuanced risk: they also despise uncertainty. Evolution has left us survivors with a “fight or flight” response to fear and risk, so it can be difficult--even for those of us with plenty of mathematical training--to not let instinct overrule calculation.
What does it look like, though, to see risk and uncertainty through a mathematical lens, unclouded by instinct? Let’s consider the answer to that question using a morbid but fully unambiguous example: the risk of death.
In considering the risk of various experiences causing death, a numerically-inclined scientist will start with one fundamental question: What fraction of people having the experience being evaluated die? Notice that this question is not “how many people died?” “Fraction,” defined simply as a numerator divided by denominator, is the most important word in the question scientists ask. They want to know how many people died (the numerator) and how many could have died (the denominator). In estimating the fraction of people dying, good scientists will also try to quantify uncertainty, by asking “how sure are we about the estimates of the numerator and the denominator in this fraction?,” and acting upon the answers.
The fraction scientists seek is a quantitative measure of risk, which can also be expressed in words or percentages, such as “1-in-a-hundred” or, equivalently, “1%.” The uncertainty in this risk can similarly be expressed quantitatively in words or numbers. For example, if someone states a “factor of 10” uncertainty on a 1% risk, that means the estimate of risk should lie somewhere between 1% divided by 10 (=0.1%) and 1% times 10 (=10%). If instead a “5 percent” uncertainty is assigned to the same 1% risk, the range in the estimated odds is much smaller, lying between 1.05% and 0.95%.
To solve for the fraction in What fraction of people having the experience being evaluated die?, one needs to know how many people undertake the experience (the denominator), and how many die doing it (the numerator). It is rare that the chance-of-death fraction’s numerator and/or denominator can be known or estimated perfectly. Instead, the calculated risk of death is associated with an uncertainty, which is ultimately determined by how well the numerator and denominator needed to calculate the risk as a fraction are known.
In everyday language, we often refer to risk using the word “odds,” and uncertainty is just a measure of how well we know the odds. We can illustrate how scientists think about odds using some examples that illustrate extremes in the risks and uncertainties of dying from various experiences.
Consider extremely low-uncertainty experiences first.
Our Deer Hunter Russian Roulette story is an extraordinarily good example of a low-uncertainty experience with high risk. To remind anyone who hasn’t seen the movie’s gut-wrenching climactic scene, in a single round of Russian Roulette a six-chamber revolver is loaded with only one bullet. The revolver’s cylinder is spun, stopped at random, and then fired by the player at their own head. So, as we explained at the outset, the risk of dying in a round of Russian Roulette is thus 1 (bullet) in 6 (chambers), or using our deaths-per-experience fraction, (1 death)/(6 possible experiences)= ⅙=17%. The uncertainty in estimating this 17% risk of death is extraordinarily low–essentially equal to zero. Unless the gun has been tampered with, or the player’s hand slips, a bullet to the head will almost inevitably cause death.
At the opposite end of the risk spectrum from Russian Roulette, consider the extremely safe act of lying on a couch. Unlike firing a gun with 1 full chamber at one’s own head, the risk of death from lying on a couch is near-zero, but like Russian Roulette the uncertainty in the risk is also near-zero.
How do we know the uncertainty is so low for Russian Roulette and couch-lying? In Russian Roulette, the mechanism, which we can state as “a bullet shot by a revolver to the head kills,” is so clear, that unless the gun malfunctions, or a person can survive a bullet to the head, death is assured 1 in 6 times. In calculating the 1-in-6 risk’s low uncertainty, we tacitly assume that the “unless” situations (bad guns, surviving shots to the head) are also extremely unlikely. How do we know that they’re unlikely? Copious amounts of prior observation, which we can call “data.” Many many revolvers have been fired, and they almost never malfunction. Many people have been shot in the head, and they almost never live. Similarly, a tremendous amount of observational data tells us that lying on a couch is, with extraordinarily high certainty, almost never fatal.
Extremely high-uncertainty experiences also come in low-risk and high-risk varieties.
Ponder, for example, the high-risk and high-uncertainty Apollo 11 voyage to the Moon in 1969. No human-occupied craft had ever before visited and landed on the Moon; space is extremely inhospitable to humans; and rockets have a not-insignificant tendency to malfunction or even explode. These factors would clearly combine for a very high risk of death for astronauts’ involved in the first lunar landing, but evaluating exactly how high that risk was before the first landing would be very difficult, given the paucity of prior data (no Moon landings!) and lack of information on mechanisms (limited understanding of human physiology and rockets’ behavior on the Moon)--so uncertainty was also high.
When it comes to low-risk/high-uncertainty combinations, the best example that comes to mind is, alas, a new disease that behaves erratically but kills a low percentage of the time.
The problem with COVID-19 is uncertainty.
Before we say any more about our present pandemic, let us be clear that we are NOT recommending a wishful-thinking return to life-as-usual before good treatments or vaccines for the virus are widely available. Our goal here, instead, is to use the framework above to argue that what scares many people--without their knowing it--about COVID-19 is the uncertainty around the risk it poses.
By now, most of a year into the pandemic, responsible media has made it clear to most people that: certain populations are more vulnerable to the ravages of this disease than others; face coverings seem to reduce airborne transmission; and that large super-spreading events can be extraordinarily dangerous.
Upon seeing predictions that about 350,000 Americans are likely to die from COVID-19 in 2020, one typically reacts viscerally, given that any avoidable or unexpected loss of life is devastating. Most people stop at that visceral reaction and do not naturally next ask “What fraction of people having the experience being evaluated die?” And, almost no one asks for quantitative measures of uncertainty on the forecasted risk.
But, what if people--especially in the media--did ask these quantitative follow-up questions about the risk of dying from COVID-19? Would the tradeoffs of safety vs. risk we would be willing to make personally or institutionally be different? Let’s think about the answers by looking first at COVID-19 risk in the United States and then at its uncertainty.
What fraction of people having the experience being evaluated die from COVID-19? If “having the experience” simply means “living in the United States,” and thus being exposed to some level of the SARS-CoV-2 virus that causes COVID-19, then calculating this fraction is straightforward. An estimate of 350,000 deaths needs to be divided by a population of 330 million, giving 350,000/330 million, or about 0.1%. To put that 0.1% risk into quantitative context, take note that about 1% of the US population dies every year, from all causes of death combined. So a 2020 death toll of 0.1% from COVID-19 would represent a 10% increase in the US death rate. This calculation does not mean that COVID-19 is not dangerous: it only means that--with current mitigation strategies in place--the pandemic can be expected to give about a 10% surge in annual deaths.
What is the uncertainty in this “10%” surge in deaths? For a new disease like COVID-19, it can be argued that calculating the uncertainty associated with projected risks is actually harder than calculating the risk itself. Earlier on in the pandemic, we did our own evaluation of the uncertainty projections in the IHME models and found that, while the models’ early predictions of deaths over time were surprisingly accurate given the paucity of data and information about mechanism, the range of outcomes offered (the uncertainty) was typically much too small.
How much better are uncertainty estimates around COVID-19 risks now than they were seven months ago? To answer, remember that both data and understanding of mechanisms are needed to reduce uncertainty. When the initial forecasts of COVID-19’s course were made, data were super-scarce, and the new virus’ mechanisms for spreading and causing harm were not at all well-understood. Today, as tragic as the damage has been, we have much more information about COVID-19’s effects, and about the risk it poses to individuals and populations. We find, unfortunately, that individuals with particular co-morbidities or predispositions, and that particular populations who live in crowded conditions with poor access to healthcare, are much more likely to fall ill. Given the new information, any COVID-19 model’s uncertainty should be smaller than it was at the start of the pandemic, in much the same way that it was easier to forecast risk for the fifth moon landing than it was for the first.
Looking at current projections for March 1, 2021--likely just before widespread vaccine distribution--the IHME COVID-19 forecast graph for the US with the “uncertainty” feature turned on, shows an absolute worst-case scenario of 775,000 total COVID-19 death and a best-case scenario of 405,000 deaths. The 370K difference between 405K and 775K represents an uncertainty of plus or minus 185K around a “representative” (average) projection of 590K US deaths. Expressed as percentages, these IHME-based forecasts give a random American a 0.2% plus or minus 0.05% risk of dying from COVID-19. So, the uncertainty in the risk is equal to 25% (= 0.05/0.2).
For comparison, keep in mind that between 1918 and 1920, about 3% of the world’s population died from Spanish flu, a rate fifteen times higher than the 0.2% expected in the US from COVID-19. Worldwide (not just in the US), the death rate from COVID-19 is currently projected at 0.03%, 100x lower than the Spanish flu.
So why are people so scared of contracting coronavirus? There are three answers: we have gotten used to living in a low-risk world; humans fear death; and people misunderstand uncertainty.
As we sit here in 2020, we are used to a much safer world than our counterparts of 1920. In 1920, 150 babies out of 1000 died before the age of five. Today, that number is under 7--more than twenty times lower. This lowering of risk is true across nearly all possible causes of death, as well documented in Stephen Pinker’s recent book “Enlightment Now.”
But being used to low levels of risk is only part of the issue. The other pieces involve our visceral reactions to death and to uncertainty. Any feeling person wants to minimize suffering, and leaders throughout history have always had to separate emotion from difficult policy decisions that affect longevity. Dan Gilbert’s “might” category applied to a risk of death includes everything from COVID-19 (~0.2%) to Spanish flu (~3%) to Russian Roulette (17%). We hope we have shown here, though, that the differences between levels of “might” can be measured, and expressed as a risk with an associated uncertainty around that risk.
Next time you read about COVID-19, ask yourself, the media, and our leaders: are we reacting with facts and data or just reacting?