In the Fall of 2020, my colleague, Prof. Immaculata De Vivo of the Harvard School of Public Health, and I wrote an essay about the public perception of risk and uncertainty, especially with regard to COVID-19. In this post, we are gathering comments from students in the Spring 2021 edition of "GenEd 1112: The Past and Present of the Future," an undergraduate course I teach at Harvard. Students were asked to read the essay, and then comment here on which part(s) of the discussion they expect would be most illuminating for non-quantitatively-inclined readers --and/or to suggest another framing of the issues discussed that would be more effective.
top of page
bottom of page
I think that contextualizing the uncertainty and the percentages with more observed historical factors is likely to be the most illuminating of this piece. Humans are generally terrible at quantifying probabilities, and have a deep instinct to overreact to uncertainty and the unknown. By contextualizing it with known data, we can make the message come across more usefully.
It could also be useful to expand into discussing how fractions are often more easily conceivable to the general population than probabilities. I really like the quality of low uncertainty event examples that were given.
I thought the real-world scenarios that Dr. Goodman utilized to explain low-uncertainty experiences were excellent choices to appeal to a non-quantitative audience. The Russian Roulette example vividly demonstrates a situation in which there is a high risk of death and low uncertainty. A reader that is unfamiliar with statistics can still easily understand the certainty embedded within a system where the bullet will kill with only "one shot."
Conversely, the "lying on a couch" example showcases an everyday activity with an extremely low risk of death, but high certainty as well. This pair of scenarios demonstrates quantitative, statistical truths, in a way that translates the abstract concept of uncertainty into concrete examples.
I think the most illuminating part of the discussion would be the part which discusses how us humans are bad at estimating nuanced risks, and I think that plays into a lot of decisions we make, especially for those of us who are non-qualitatively inclined. This is why people allow their anxiety to take over, and not make rational decisions (it is safer to fly than drive a car, but people seem to think there is more uncertainty with flying, and less within their control, so they do not).
I think another way to frame it would be to state conclusively what we know (I.e. how many ppl have died, and how that will affect the future). I think the discussion here looked a bit more about what happened in the past and could we have predicted it, and the answer of its complicated is I think to some readers more confusing. I understand is designed to highlight the uncertainty, but I think some people could get bogged down in what they think they know even though as the article attempts to demonstrate, they do not. So I would approach of this is what we know, this is what our prediction would be, and this is our uncertainty and other factors.
After rereading “Uncertain Risks” by Professor Goodman and Professor De Vivo, I expect their emphasis on understanding how many people were evaluated in a study to be most illuminating for non-quantitatively-inclined readers. Understanding the context for data is essential and can often be overlooked by the public. For example, it can be helpful to think about the fraction of how many people died to the number of people who could have died. As Professor Goodman and Professor De Vivo explain using COVID-19 as an example, the number of deaths by COVID-19 is striking—350,000 deaths in the U.S. in 2020. Nonetheless, when looked at as a 0.1% death rate (deaths/total population), the statistic is nowhere near as concerning to the public, especially with 1% of the population dying every year. Overall, I think Professor Goodman’s and Professor De Vivo’s article on uncertainty is extremely helpful in encouraging the average person to reframe how they look at statistics. When news sources are trying to grab your attention with a very high number, you should always ask “but what was the total number of people evaluated?” to gain more context.
Another way that might be useful to frame the discussion would be the shape of the variance and how that matters in terms of real-life context. For example, if there were two strategies for fighting Covid-19, a high-variance one (where different samples of people were saved or died based on certain characteristics such as age) or a low-variance one (the death rate is exactly the same for every sample), they might have the same overall mean and expected number of deaths but the outcome would be very different depending on the strategies used. A high variance might be along the lines of taking a more lax response to Covid, meaning that more elderly people die of Covid. A low variance response might be along the lines of taking a strict response to Covid and saving more elderly but losing more young people to mental health consequences of being quarantined. Overall these strategies and variance in the overall discussion have an understated impact on how outcomes play out.
I found the discussion of the interaction between risk and uncertainty within the Russian roulette example to be the most simplistic and helpful. It clearly laid out to the reader how risk is the odds something happens while uncertainty acts as the first derivative of risk. Namely, it essentially sets the bounds for which our measure of risk is accurate. This framework for understanding how the two measures relate, applied to a very simplistic and understandable example, made the two concepts very digestible.
I enjoyed the real world example and graphic about where covid fits into the uncertainty and risk spectrum was helpful to visualize a lot of how our discussions fit into our current lives, especially how we all went through the pandemic. One of the other comments above mentioned the lack of stem classes at Harvard and I agree that this was a very helpful introduction to data science and prediction that I would not have otherwise had.
I think the inclusion of the Deer Hunter and Russian Roulette analogies throughout the article prove to be helpful to non-quantitatively inclined readers. As is addressed early in the article, misunderstanding or confusing the words "risk" and "uncertainty" can be dangerous, and doing so is a key reason why many misunderstand COVID-19. I think the first paragraph of the article does a great job of defining these words through the easily-understood analogies of hunting a deer and Russian Roulette. The article addresses all of the variable factors involved in determining the odds of killing a deer in a single shot which perfectly defines uncertainty, and the blunt 1 in 6 chance of dying in Russian Roulette which defines the other extreme, nearly complete certainty. These examples clearly differentiate risk, the chance of death, from uncertainty, how sure we are about the amount of risk, which I think is really helpful for non-quantitatively inclined readers.
1. Which part(s) of the discussion do you expect would be most illuminating for non-quantitatively-inclined readers?
I really like how the article suggests we put uncertain events into four "buckets" - high-risk high-uncertainty (Apollo 11), high-risk low-uncertainty (Russian roulette), low-risk high-uncertainty (getting COVID-19), and low-risk low-uncertainty (driving). While this approach applies "stereotypes" to uncertain events, it's often much easier for humans to think about things when using categories as compared to the raw numbers.
2. Suggest another framing of the issues discussed that would be more effective.
One method I love for evaluating risk of death is the micromort along with confidence intervals. In keeping with conventions for metric units, one micromort is a 1/1,000,000 (one in a million) chance of death. For example, skydiving comes with a risk of 8 micromorts per jump (8 in a million chance of death), while traveling 230 miles by car comes with a risk of 1 micromort. This unit lets us convert risk from a hard-to-understand fraction into a nice, low, round number, which is much easier to comprehend.
Additionally, micromorts let us use error bars that make sense on our numbers. For example, skydiving has a risk of 8.4 micromorts per jump, with a 95% confidence interval between 7.7 micromorts and 9.3 micromorts - a low-risk low-uncertainty activity. But going to space, by contrast, we now know has risk of 32,000 micromorts (not taking into account modern advances in spaceflight technology), with a 95% confidence interval between 21,000 and 49,000 micromorts - far less certain than skydiving.
As someone who wasn't entirely sure of the difference between risk and uncertainty, I found the example of the gun and Russian roulette extremely enlightening in differentiating the uncertainty of death -- which is almost 0 in the case of a bullet entering the brain -- and risk, which is the probability that the bullet will be in the chamber fired; in this case, 1/6. I also appreciated the examples of other easily visualizable isntances of different uncertainty/risk tradeoffs, such as the living room couch for low risk/low uncertainty and space exporation (Apollo 11) for high risk and high uncertainty. These examples really helped flush out the concepts before they were applied to the low risk, high uncertainty scenario of COVID-19
As someone who has never taken a rigorous stem course at Harvard nor considered themselves a quantitatively inclined reader, this essay was surprisingly helpful and clear for me when it came to understanding the difference between risk and uncertainty. Even though the risk/uncertainty matrix professor Goodman supplied was a qualitative representation of the issue, the quantitative aspects of things like covid modeling help us understand why mitigation is important in cases of high uncertainty particularly due to the human factors at play. I thought the article blended these two perspectives really well through the numerical examples provided in the covid examples in particular.
I think for non-qualitatively inclined readers, the covid risk discussion is the most useful. Particularly, when it says that the average deaths from the United States are only estimated to increase by 10%. I think that most people think that a relatively low number of people die each year, and so just one more person dying per 10 estimated deaths in a normal year would not concern them greatly. As such, I think they would be able to understand the chance of dying from Covid is not necessarily that great, and they can view the risk with more of a probability assessment rather than a "might happen" assessment.
2. Another way I might choose to frame this discussion has to do with another way to think about uncertainty. Rather than think about where the “real odds” may be, one can think of it in terms of how much you should be willing to adjust your odds. For instance, one can be very certain in a prediction that a coin flip will be heads exactly 50% of the time. Because we can have high certainty, it would take a lot of information to change your opinion. If someone told you “they knew” heads only came up 30% of the time you wouldn’t believe them; and if you flipped a coin 10 times and got 8 heads it wouldn’t really change your opinion on the probability. But let’s say you have a weighted coin, but you don’t have any idea how it is weighted. Now if you flipped the coin 10 times and got 8 heads, you would believe that the coin is probably a little weighted towards the head. Thus, one can have a real understanding of what uncertainty in prediction means through a more application-based explanation.
I found this essay very engaging and provided the reader with commentary and discussion that could be more easily digested than trying to interpret lots of statistics and quantitative evidence about this subject. Personally, I would consider myself a non-quantitatively-inclined reader and find that my best understanding of topics comes from broken down discussion and interpretation that is focused on the topic rather than trying to interpret all of the numbers that support that topic. In this essay, I appreciated the lens that deconstructed the complexities of these ideas; even in the first paragraph, there was a quote from a famous movie that one might know and then some explanation that was connected to common known activities like hunting and Russian roulette. From that point, I already had a better grasp of the knowledge trying to be conveyed than if I had tried to read a daunting quantitatively focused essay. The fact that the Deer Hunter Russian Roulette story returned to the conversation to describe the differences in uncertainty and risk which was also very illuminating and grounded my understanding. The reader is then able to connect the examples at the beginning of the essay to how it all relates to COVID-19 and epidemiology, which was a great way to help the reader to best understand the basics of risk and uncertainty and then more easily apply it to their uptake of this knowledge surrounding the current pandemic.
I think using the example of the One Deer Hunter Russian roulette story throughout the explanation and then weaving different components of the topic to be understood using the same example is something that non-quantitatively inclined readers would find to be most helpful. Especially since it allows them to apply this knowledge to a real-world working scenario and view the different definitions of the risk and uncertainty can be more easily applied. The example is used in the first paragraph to compare more complex odds with many factors playing a role such as deer hunting to the easily calculate odds when playing Russian Roulette. By then using these same scenarios to explain low uncertainty and high experiences we are able to better understand the details behind them.
I believe the main takeaway of the essay for non-quantitatively-inclined readers should be the discussion of why consider uncertainty in the first place. Humans are not perfect and when we attempt to theorize, experiment, or evaluate observable phenomena there is no way our results will hold 100% of the time. However, having a notion of 'error-spread' or uncertainty is crucial to make this idea transparent and thus make better science and even everyday decisions.
The section that I expect would be most illuminating for non-quantitatively inclined readers would be the part two situations with different uncertainties. In this section it looks first at a low-uncertainty situation (Russian Roulette) then at a high uncertainty situation (Apollo 11). I think it is very important to apply these easily comprehensible examples towards two drastically levels of uncertainty because it makes it a much easier access point. Jumping straight into numbers with little context can lead to people just giving up and tuning out the rest of the information. However, using these tangible situations and very little, simple math it is much easier to get a foot in the door of this information. Once that ground layer of understanding people can feel calmer and more confident that they will be able to comprehend the rest of the information. Also, having two different examples with different levels and factors of uncertainty allows the coinciding information about COVID-19 much easier to comprehend.
I believe that the thoroughness and the many examples given when exploring what risk and uncertainty mean at a fundamental level are extremely helpful for non-quantitively-inclined readers. One of the main objectives of the essay is exactly giving a quantitative look at a problem many people view only qualitatively (will happen, may happen, won't happen). Considering there is no escape from the quantitative analysis, I believe the way the essay handles it is excellent, by explaining at a basic level what is risk and uncertainty, using several examples along the way, in a way that almost all readers paying attention and that understand the basics of fractions and percentages will be able to understand. I also believe that the connection back to the pandemic will be specially illuminating for readers who hadn't considered uncertainty in risk from a quantitative POV before, as it equips them to better think about the numbers and predictions that are being shown to them in the media, and reflect on what they really mean, and what would be a reasonable response.
Unequivocally, the passage in this essay on COVID statistics was eye-opening to read for someone "non-quantitatively-inclined" such as me. While the language of low-uncertainty and high-uncertainty experiences was highly interesting and illuminating, I can't imagine organically changing behaviour based on these discoveries. When the statistics are set out in such a way that they are in the essay though; in plain language, and particularly in comparative context; to the average annual death rate, to the death rate of the Spanish flu, etc., I feel like I'm being introduced to a coherent narrative, based in fact, which will tangibly impact my thoughts, opinions, and thus behaviours.
The content, then, I think is excellent! It sounds silly, but perhaps a good way to impart it other than in essay format would be through a TikTok. While a short, tiktok format videos starts off mostly being watched by the younger demographic that likes the app, I see the videos being shared now frequently on other platforms, such as Twitter and Facebook, which have a larger reach for broader swathes of the population!
I surely agree, @vincentli -- these are the draft images we ultimately didn't include, but clearly should!