Introduction to Uncertainty
by Alyssa A. Goodman, August 10, 2021
In life, decisions are harder in the face of uncertainty. Do I really need an umbrella? Which job offer should I accept? Where should I go on vacation? It’s hard to know without seeing the future.
The more sure we are about a prediction, the less fraught a decision based on it. And, for predictive systems that rely on mathematical algorithms, “sure” can usually be quantified. “Uncertainty” is the perhaps unfortunate standard term term describing the “certainty” of a given mathematically-based prediction.
Hurricane forecasts offer a good example of predictions that come with clear visual representations of uncertainty estimates. The sample NOAA forecast here shows the most likely track of 2017’s Hurricane Irma as a black line with “M”’s along it, and the white contour outlines a less likely, but still possible, range of possible other tracks. Notice how uncertainty (the spread of possibilities perpendicular to the black line) grows as time progresses? That’s because it’s harder to predict a storm’s path, even with amazing computer weather models, farther into the future.
A similar kind of uncertainty growth with time can be seen using the Take-a-Sweater weather forecast app we use in PredictionX, where forecasts are typically most uncertain for the day’s farthest into the future.
Weather forecasts rely on a wide variety of measurements (of wind speeds, pressure, temperature, etc.) being input into complex computer models that use combinations of chemistry, physics, and past experience to predict everything from simple numbers like temperature to complicated phenomena like hurricane tracks. For just about every weather forecast made, uncertainty measurements exist or can be made—even though they are only sometimes communicated to the public.
In the PredictionX “Framework for Predictive Systems,” the “Predictive System” used is evaluated for accuracy. A good estimate of uncertainty captures how accurate one can expect a system to be.
In our “Uncertainty about Uncertainty” article on 2020 COVID-19 forecasting, we emphasize that the flaw in the IHME predictions was not their inability to predict COVID deaths, but instead their inability to correctly evaluate the uncertainty associated with the forecasts. The snapshot of early Italy forecasts below captures this problem well—showing how actual death rates consistently fell far outside of the blue “uncertainty” bands representing where 95% of the forecasts should have fallen.
In today’s world, computer simulations —of climate, weather, finance, health, traffic, and more— are critical to daily life, and they are all around us. But, most of us don’t know or even think about how these simulations work, no less about why it is so hard to estimate the uncertainty associated with their predictions. So, to make it a little easier to appreciate the meaning of simulations and uncertainty, we’ve created a really simple game you can play right here by clicking on the image below.
In the “Slide-the-Puck” game, we’ve stripped off all the complexity usually associated with the simulations you use but don’t think about daily (like weather forecasts) and instead created a simulation that has only one source of uncertainty—the roughness of a virtual wooden table.
The game simulates a person (you!) sliding a puck on a level table at a target (exactly in the middle of the virtual “wall.” If the table were perfectly smooth, the puck would hit the target every time, and this game would be very boring. There would be no uncertainty in the puck’s point of impact. Instead, imperfections in the surface of the table cause the puck’s path to wiggle some as it moves toward the wall. You can adjust the level of imperfection with a slider, and you’ll see the the rougher you make the table, the wider the “distribution” of ending positions you see at the end of your game(s). I
f one knew everything about the materials of which the puck and table were made, which determine’s the “roughness” between their surfaces, then physics can predict the width of the distribution you’ll see in this simulation. Knowing that, a physicist in a real puck+table situation would call the distribution’s width the “uncertainty” around the expected (exactly centered) impact spot of the puck. (And, if this were a real carnival game, where uncertainty figured into the odds, the physicist would have a big advantage!)
Slide-the-Puck is a really simple game, built purposely to be easy to simulate, and to have only one source of uncertainty. At the other end of the complexity spectrum, where there many factors mix together in determining outcome, such as human behavior mixing with disease mechanism in the case of a pandemic, making and explaining forecasts and their uncertainty is difficult, and complicated.