Uncertainty Analysis

by: Greg Paoli

Risk managers, risk assessors and the broader stakeholder community inevitably grapple with the need to make decisions in the presence of substantial uncertainty. One of the defining elements of the field of risk analysis has been the focus on methods for characterizing what is known and not known about a particular threat to health or the environment. This sub-discipline within the risk sciences is often given the general name of uncertainty analysis. A primary role of uncertainty analysis is to contribute to the risk characterization phase of risk assessment. A key responsibility for the risk analyst within the risk characterization phase is to adequately describe the key sources of uncertainty, and to characterize their individual and collective impact on specific conclusions and policy options. An adequate description of uncertainty requires honest reflection on the state of knowledge (or conversely, the extent of ignorance) that remains with respect to the ability to predict the level of human health risk with currently available data and knowledge. It also requires attention to methods for quantifying individual uncertainties, for appropriately combining uncertainties to achieve an overall measure of uncertainty, and the avoidance of pitfalls associated with qualitative methods for communicating about risk and uncertainty.

Types of Uncertainty

There are numerous ways in which uncertainty may be defined and categorized. Taxonomies have been proposed that help to differentiate uncertainty according to its many sources (Morgan and Henrion (1990), NRC (1994), Cullen and Frey (1999) and Krupnick et al. (2007)). One key distinction that is often advocated is the conceptual separation of uncertainty and variability. The term uncertainty is most often used to describe limits in knowledge. Uncertainty is expressed using a range of numbers or a probability distribution to capture the reality that we do not know what the true value is. Variability is used to describe real differences that exist in the world among individuals, practices, behaviors and the inherent variability in the natural world, which may or may not have an available explanation. When variability is expressed using a range of numbers or a probability distribution, it reflects the fact that there is no single true number that fully describes a phenomenon. In practice it is often difficult to completely separate the two. This is particularly difficult when attempting to perform a critical task in population health risk analysis that combines these two concepts, namely, expressing our uncertainty in the nature and extent of variability.

In describing uncertainty as a limitation of knowledge, risk analysts have found it useful to distinguish between two main types of uncertainty, parameter uncertainty and model uncertainty. When a mathematical model is used to describe a risk-generating system, there will often be uncertainty with respect to specific values that need to be assigned to variables in the model in order to predict the level of risk. This type of uncertainty has often been called parameter uncertainty. This type of uncertainty can be contrasted with what might be considered a more fundamental form of uncertainty in which the structure and relationships in the mathematical model are themselves uncertain. Whenever there is an incomplete understanding of the causal structure of a system, (i.e., there are competing explanations for some observed phenomenon), there will necessarily be alternate mathematical models that might legitimately be used to make predictions about the level of risk. The existence of competing explanations, (and thereby competing models), is often referred to as model uncertainty. When the model uncertainty is so great as to result in questions of the very existence of causal relationships (as opposed to competing models of the strength and exact nature of the relationship) this may be referred to as fundamental causal uncertainty (NRC, 2009).

Forms of Uncertainty Analysis

The approaches taken to describe and analyze uncertainty are quite variable. One way of considering the variations is on a continuum from narrative to quantitative approaches.

In any risk characterization, there is an expectation that the level of uncertainty in the results will be provided. This can be done in a simple narrative form, with the analyst describing the key sources of uncertainty and describing the impact of these uncertainties on the conclusions, but without quantifying the uncertainty or the impact.

The narrative component can be augmented by quantifying the range of possibilities associated with an uncertain variable, and demonstrating how the conclusions would change when this range of possible values is employed. This approach is often referred to as univariate (i.e., one variable) sensitivity analysis. This approach can be expanded to include more than one variable, to demonstrate the full range of possible conclusions from the analysis, by combining the impact of more than uncertain variable at once.

One of the most complete forms of uncertainty analysis is called probabilistic uncertainty analysis. With this approach, uncertain variables are described using probability distributions to describe the range and relative likelihood of alternate possible values of each variable. In order to provide a complete picture of the net impact of all of the uncertain variables, the analyst needs to estimate how likely different risk estimates are, given the uncertain variables. This is often achieved through a technique called Monte Carlo simulation. The result is that the estimate of interest (e.g., a risk estimate, or an estimate of risk reduction) is represented as a probability distribution (rather than a single number) that captures the impact of all of the uncertainties that were quantified in the uncertainty analysis.

The level of uncertainty analysis that is required is highly dependent upon the nature and context of the decision that the risk assessment is intended to support. Choosing an approach to uncertainty analysis is a key element of risk assessment design and is key to successful communication between risk assessor, risk manager and other stakeholders.

Suggested reading:

Cullen, A.C., and H.C. Frey. 1999. Probabilistic Techniques in Exposure Assessment: A Handbook for Dealing with Variability and Uncertainty in Models and Inputs. New York, NY, Plenum Press.

Krupnick, A., R. Morgenstern, M. Batz, P. Nelson, D. Burtraw, J.S. Shih, and M. McWilliams. 2006. Not a sure thing: Making regulatory choices under uncertainty. Resources for the Future. February 2006.

Morgan, M.G. and M. Henrion (1990). Uncertainty. New York, N.Y.: Cambridge University Press.
NRC (National Research Council). 2009. Science and Decisions: Advancing Risk Assessment. Washington, DC: The National Academies Press.