Using Probabilistic Methods to Enhance the Role of Risk Analysis in
Decision-Making

With

Case Study Examples

Prepared by Risk Assessment Forum

PRA Technical Panel Working Groups

EPA/100/R-09/001

“THIS INFORMATION IS DISTRIBUTED SOLELY FOR THE PURPOSE OF
PRE-DISSEMINATION PEER REVIEW UNDER APPLICABLE INFORMATION QUALITY
GUIDELINES. IT HAS NOT BEEN FORMALLY DISSEMINATED BY EPA. IT DOES NOT
REPRESENT AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY AGENCY
DETERMINATION OR POLICY.”Disclaimer

This document is a peer review draft. It has not been formally released
by the U.S. Environmental Protection Agency and should not at this stage
be construed to represent Agency policy. It is being circulated for
comments on its technical merit and policy implications. Mention of
trade names or commercial products does not constitute endorsement or
recommendation for use.

This document was produced by a Technical Panel of the EPA Risk
Assessment Forum.  The authors drew on their experience in doing
probabilistic assessments and interpreting them to improve risk
management of environmental and health hazards.  Interviews,
presentations, and dialogues with risk managers conducted by the
Technical Panel have contributed to the insights and recommendations in
this summary and the associated White Papers.Foreword

Throughout most of the Environmental Protection Agency's program offices
and regions, various forms of probabilistic methods have been used to
answer questions about exposure and risk, to humans and other organisms,
and the environment.  EPA risk assessors, risk managers and others,
particularly within the scientific and research divisions have
recognized that sophisticated statistical and mathematic approaches
could be still more fully utilized to enhance the quality and accuracy
of Agency risk assessment and risk management.  Various stakeholders,
inside and outside the Agency, have called for a more comprehensive
characterization of risks, including uncertainties, in protecting more
sensitive or vulnerable populations and life stages.

Therefore, the Office of the Science Advisor of the EPA, together with
the Science Policy Council and members of the Risk Assessment Forum
(RAF), identified a need to examine the use of probabilistic approaches
in Agency risk assessment and risk management.  An RAF Technical Panel
developed papers (this paper and a managers' summary) which provide a
general overview of the value of probabilistic analyses and similar or
related methods, and some examples of current applications across the
Agency.

The goal of these papers is not only to describe potential and actual
uses of these tools in the risk decision process, but also to encourage
their further implementation in human, ecological and environmental risk
analysis and related decision making.  The enhanced use of probabilistic
analyses to characterize uncertainty in assessments would not only
reflect external scientific advice on how to further advance EPA risk
assessment science, but will also help to address specific challenges
faced by managers and improve confidence in Agency decisions

Kevin Teichman

Acting EPA Science Advisor

Acknowledgments

We would like to acknowledge the scientists and risk assessors who
assisted in the preparation of this paper.

Contributors and Reviewers of this Document

Chris Frey, John Paul, and Pasky Pascual (Leads)

Gary Bangs, Mike Clipper, Kathryn Gallagher, Bob Hetes, Michael Messner,
Keeve Nachman, Haluk Ozkaynak, Zachary Pekar, and Woodrow Setzer

The PRA Technical Panel

Table of Contents

  TOC \o "1-3" \h \z \u    HYPERLINK \l "_Toc221604244"  Using
Probabilistic Methods to Enhance the Role of Risk Analysis in
Decision-Making	  PAGEREF _Toc221604244 \h  4

  HYPERLINK \l "_Toc221604245"  1. Introduction: Relevance of
Uncertainty to Decision making: How Probabilistic Approaches Can Help
PAGEREF _Toc221604245 \h  5

  HYPERLINK \l "_Toc221604246"  1.0. What Is Probabilistic Risk
Analysis, and How Does It Address Variability and Uncertainty?	  PAGEREF
_Toc221604246 \h  5

  HYPERLINK \l "_Toc221604247"  1.1. Goals and Intended Audience
PAGEREF _Toc221604247 \h  5

  HYPERLINK \l "_Toc221604248"  1.2. Overview of This Document	  PAGEREF
_Toc221604248 \h  5

  HYPERLINK \l "_Toc221604249"  1.3. What Are Common Challenges Facing
EPA Risk Decision makers?	  PAGEREF _Toc221604249 \h  6

  HYPERLINK \l "_Toc221604250"  1.4. What Are Key Questions Often Asked
by Decision makers?	  PAGEREF _Toc221604250 \h  7

  HYPERLINK \l "_Toc221604251"  1.5. Why Is the Implementation of
Probabilistic Risk Analysis Important?	  PAGEREF _Toc221604251 \h  7

  HYPERLINK \l "_Toc221604252"  1.6. How Does EPA Typically Address
Scientific Uncertainty and Communicate Variability?	  PAGEREF
_Toc221604252 \h  8

  HYPERLINK \l "_Toc221604253"  1.7. What Are the Limitations of Relying
on Default-Based Deterministic Approaches?	  PAGEREF _Toc221604253 \h  9

  HYPERLINK \l "_Toc221604254"  1.8. What Is EPA’s Experience with the
Use of Probabilistic Risk Analysis?	  PAGEREF _Toc221604254 \h  10

  HYPERLINK \l "_Toc221604255"  2. Probabilistic Risk Analysis	  PAGEREF
_Toc221604255 \h  12

  HYPERLINK \l "_Toc221604256"  2.1. What Are Variability and
Uncertainty, and How Are They Relevant to Decision-making?	  PAGEREF
_Toc221604256 \h  12

  HYPERLINK \l "_Toc221604257"  2.1.1. Variability	  PAGEREF
_Toc221604257 \h  12

  HYPERLINK \l "_Toc221604258"  2.1.2. Uncertainty	  PAGEREF
_Toc221604258 \h  12

  HYPERLINK \l "_Toc221604259"  2.2. When Is Probabilistic Risk Analysis
Applicable or Useful?	  PAGEREF _Toc221604259 \h  13

  HYPERLINK \l "_Toc221604260"  2.3. How Can Probabilistic Risk Analysis
Be Incorporated into Assessments?	  PAGEREF _Toc221604260 \h  13

  HYPERLINK \l "_Toc221604261"  2.4. What Are the Scientific
Community’s Views on Probabilistic Risk Analysis, and What Is the
Institutional Support for Its Use?	  PAGEREF _Toc221604261 \h  14

  HYPERLINK \l "_Toc221604262"  2.5. How Can Probabilistic Risk
Assessment Provide More Comprehensive, Rigorous Scientific Information
in Support of Regulatory Decisions?	  PAGEREF _Toc221604262 \h  14

  HYPERLINK \l "_Toc221604263"  2.6. Are There Additional Advantages of
Using Probabilistic Risk Analysis?	  PAGEREF _Toc221604263 \h  15

  HYPERLINK \l "_Toc221604264"  2.7. What Are the Challenges to
Implementation of Probabilistic Analyses?	  PAGEREF _Toc221604264 \h  15

  HYPERLINK \l "_Toc221604265"  2.8. How Can Probabilistic Risk Analysis
Support Specific Regulatory Decision making?	  PAGEREF _Toc221604265 \h
16

  HYPERLINK \l "_Toc221604266"  2.9. Does Probabilistic Risk Analysis
Require More Resources Than Default-Based Deterministic Approaches?
PAGEREF _Toc221604266 \h  16

  HYPERLINK \l "_Toc221604267"  2.10. Doesn’t Probabilistic Risk
Analysis Require More Data Than Conventional Approaches?	  PAGEREF
_Toc221604267 \h  17

  HYPERLINK \l "_Toc221604268"  2.11. Can Probabilistic Risk Analysis Be
Used To Screen Risks or Only in Complex or Refined Assessments?
PAGEREF _Toc221604268 \h  17

  HYPERLINK \l "_Toc221604269"  2.12. Does Probabilistic Risk Analysis
Present Unique Challenges to Model Evaluation?	  PAGEREF _Toc221604269
\h  17

  HYPERLINK \l "_Toc221604270"  2.13. How Do You Communicate Results of
Probabilistic Risk Analysis?	  PAGEREF _Toc221604270 \h  18

  HYPERLINK \l "_Toc221604271"  2.14. Are the Results of Probabilistic
Risk Analysis Difficult To Communicate to Decision-makers and
Stakeholders?	  PAGEREF _Toc221604271 \h  19

  HYPERLINK \l "_Toc221604272"  3. Findings and Recommendations
PAGEREF _Toc221604272 \h  21

  HYPERLINK \l "_Toc221604273"  3.1 Findings: How Probabilistic Risk
Analysis and Related Analyses Can Improve the Decision-making Process at
EPA	  PAGEREF _Toc221604273 \h  21

  HYPERLINK \l "_Toc221604274"  3.2. Recommendations for Enhanced
Utilization of PRA in EPA	  PAGEREF _Toc221604274 \h  21

  HYPERLINK \l "_Toc221604275"  3.3 Guidance and Policy	  PAGEREF
_Toc221604275 \h  22

  HYPERLINK \l "_Toc221604276"  3.4 Challenges	  PAGEREF _Toc221604276
\h  23

  HYPERLINK \l "_Toc221604277"  Appendix A: An Overview of Some of the
Techniques Used in Probabilistic Risk Analysis	  PAGEREF _Toc221604277
\h  24

  HYPERLINK \l "_Toc221604278"  A.1. What Is the General Conceptual
Approach in Probabilistic Risk Analysis?	  PAGEREF _Toc221604278 \h  24

  HYPERLINK \l "_Toc221604279"  A.2. What Are the Multiple Types of
Probabilistic Risk Analyses, and How Are They Used?	  PAGEREF
_Toc221604279 \h  25

  HYPERLINK \l "_Toc221604280"  A.3. What Are Some Specific Aspects of
and Issues Related to Methodology for Probabilistic Risk Analysis?
PAGEREF _Toc221604280 \h  27

  HYPERLINK \l "_Toc221604281"  A.3.1. Developing a Probabilistic Risk
Analysis Model	  PAGEREF _Toc221604281 \h  27

  HYPERLINK \l "_Toc221604282"  A.3.2. Conducting the Probabilistic
Analysis	  PAGEREF _Toc221604282 \h  27

  HYPERLINK \l "_Toc221604283"  Appendix B: Glossary	  PAGEREF
_Toc221604283 \h  31

  HYPERLINK \l "_Toc221604284"  Appendix C: References	  PAGEREF
_Toc221604284 \h  36

  HYPERLINK \l "_Toc221604285"  Bibliography	  PAGEREF _Toc221604285 \h
38

  HYPERLINK \l "_Toc221604286"  Probabilistic Risk Analysis
Methodology(General	  PAGEREF _Toc221604286 \h  38

  HYPERLINK \l "_Toc221604287"  Probabilistic Risk Analysis and Decision
Making	  PAGEREF _Toc221604287 \h  38

  HYPERLINK \l "_Toc221604288"  Probabilistic Risk Analysis
Methodology(Specific Aspects	  PAGEREF _Toc221604288 \h  38

  HYPERLINK \l "_Toc221604289"  Sensitivity Analysis	  PAGEREF
_Toc221604289 \h  39

  HYPERLINK \l "_Toc221604290"  Case Study Examples of Probabilistic
Risk Analysis(EPA (see also the PRA Case Studies White Paper)	  PAGEREF
_Toc221604290 \h  39

  HYPERLINK \l "_Toc221604291"  Case Study Examples of Probabilistic
Risk Analysis(Other	  PAGEREF _Toc221604291 \h  40

  HYPERLINK \l "_Toc221604292"  Appendix D: Case Study Examples of the
Application of Probabilistic Risk Analysis in U.S. Environmental
Protection Agency Regulatory Decision-Making	  PAGEREF _Toc221604292 \h
42

  HYPERLINK \l "_Toc221604293"  1. Introduction	  PAGEREF _Toc221604293
\h  46

  HYPERLINK \l "_Toc221604294"  2. Overall Approach to Probabilistic
Risk Analysis at the U.S. Environmental Protection Agency	  PAGEREF
_Toc221604294 \h  47

  HYPERLINK \l "_Toc221604295"  2.1. U.S. Environmental Protection
Agency Guidance and Policies on Probabilistic Risk Analysis	  PAGEREF
_Toc221604295 \h  47

  HYPERLINK \l "_Toc221604296"  2.2. Categorizing Case Studies	  PAGEREF
_Toc221604296 \h  47

  HYPERLINK \l "_Toc221604297"  2.2.1. Group 1 Case Studies	  PAGEREF
_Toc221604297 \h  48

  HYPERLINK \l "_Toc221604298"  2.2.2. Group 2 Case Studies	  PAGEREF
_Toc221604298 \h  49

  HYPERLINK \l "_Toc221604299"  2.2.3. Group 3 Case Studies	  PAGEREF
_Toc221604299 \h  49

  HYPERLINK \l "_Toc221604300"  3. Case Study Summaries	  PAGEREF
_Toc221604300 \h  53

  HYPERLINK \l "_Toc221604301"  Group 1 Case Studies	  PAGEREF
_Toc221604301 \h  53

  HYPERLINK \l "_Toc221604302"  Case Study 1: Sensitivity Analysis of
Key Variables in Probabilistic Assessment of Children’s Exposure to
Arsenic in Chromated Copper Arsenate (CCA) Pressure-Treated Wood
PAGEREF _Toc221604302 \h  53

  HYPERLINK \l "_Toc221604303"  Case Study 2: Assessment of Relative
Contribution of Atmospheric Deposition to Watershed Contamination
PAGEREF _Toc221604303 \h  55

  HYPERLINK \l "_Toc221604304"  Group 2 Case Studies	  PAGEREF
_Toc221604304 \h  57

  HYPERLINK \l "_Toc221604305"  Case Study 3: Probabilistic Assessment
of Angling Duration Used in Assessment of Exposure to Hudson River
Sediments via Consumption of Contaminated Fish	  PAGEREF _Toc221604305
\h  57

  HYPERLINK \l "_Toc221604306"  Case Study 4: Probabilistic Analysis of
Dietary Exposure to Pesticides for use in Setting Tolerance Levels
PAGEREF _Toc221604306 \h  58

  HYPERLINK \l "_Toc221604307"  Case Study 5: One-Dimensional
Probabilistic Risk Analysis of Exposure to Polychlorinated Biphenyls
(PCBs) via Consumption of Fish from a Contaminated Sediment Site
PAGEREF _Toc221604307 \h  60

  HYPERLINK \l "_Toc221604308"  Case Study 6: Probabilistic Sensitivity
Analysis of Expert Elicitation of Concentration-Response Relationship
Between Particulate Matter (PM2.5) Exposure and Mortality	  PAGEREF
_Toc221604308 \h  64

  HYPERLINK \l "_Toc221604309"  Case Study 7: Environmental Monitoring
and Assessment Program (EMAP): Using Probabilistic Sampling Techniques
To Evaluate the Nation’s Ecological Resources	  PAGEREF _Toc221604309
\h  66

  HYPERLINK \l "_Toc221604310"  Group 3 Case Studies	  PAGEREF
_Toc221604310 \h  66

  HYPERLINK \l "_Toc221604311"  Case Study 8: Two-Dimensional
Probabilistic Risk Analysis of Cryptosporidium in Public Water Supplies,
with Bayesian Approaches to Uncertainty Analysis	  PAGEREF _Toc221604311
\h  68

  HYPERLINK \l "_Toc221604312"  Case Study 9: Two-Dimensional
Probabilistic Model of Children’s Exposure to Arsenic in Chromated
Copper Arsenate (CCA) Pressure-Treated Wood	  PAGEREF _Toc221604312 \h
70

  HYPERLINK \l "_Toc221604313"  Case Study 10: Two-Dimensional
Probabilistic Exposure Assessment of Ozone	  PAGEREF _Toc221604313 \h
72

  HYPERLINK \l "_Toc221604314"  Case Study 11: Analysis of
Microenvironmental Exposures to Particulate Matter (PM2.5) for a
Population Living in Philadelphia, PA	  PAGEREF _Toc221604314 \h  75

  HYPERLINK \l "_Toc221604315"  Case Study 12: Probabilistic Analysis in
Cumulative Risk Assessment of Organophosphorus Pesticides	  PAGEREF
_Toc221604315 \h  77

  HYPERLINK \l "_Toc221604316"  Case Study 13: Probabilistic Ecological
Effects Risk Assessment Models for Evaluating Pesticide Uses	  PAGEREF
_Toc221604316 \h  79

  HYPERLINK \l "_Toc221604317"  Case Study 14: Expert Elicitation of
Concentration-Response Relationship Between Particulate Matter (PM2.5)
Exposure and Mortality	  PAGEREF _Toc221604317 \h  81

  HYPERLINK \l "_Toc221604318"  Case Study 15: Expert Elicitation of
Sea-Level Change Resulting from Global Climate Change	  PAGEREF
_Toc221604318 \h  Error! Bookmark not defined.

  HYPERLINK \l "_Toc221604319"  Case Study 16: Expert Elicitation for
Bayesian Belief Network Model of Stream Ecology	  PAGEREF _Toc221604319
\h  83

  HYPERLINK \l "_Toc221604320"  4. References to Case Studies	  PAGEREF
_Toc221604320 \h  87

  HYPERLINK \l "_Toc221604321"  List of Acronyms and Abbreviations
PAGEREF _Toc221604321 \h  88

Using Probabilistic Methods to Enhance the Role of Risk Analysis in
Decision-Making

Executive Summary

Probabilistic risk assessment (PRA) is a group of techniques that
provide estimates of the range and likelihood of hazard, exposure or
risk, rather than a single point estimate.  Stakeholders, inside and
outside the Agency, have recommended a fuller characterization of risks,
including uncertainties, in protecting more sensitive or vulnerable
populations and life stages.

The goal of this white paper is to explain how EPA can achieve broader
use of probabilistic methods and address uncertainty and variability by
capitalizing on the wide array of tools and methods that comprise PRA.
The information contained in this document is intended for both risk
analysts and managers faced with determining when and how to apply these
tools in their decisions.  This paper begins with a decision-maker’s
perspective, proceeds to a more technical discussion, and finally gives
a number of illustrative examples of actual EPA applications of
probabilistic analyses.

The white paper describes challenges faced by EPA decision makers,
defines and explains the basic principles of probabilistic analysis,
briefly highlights instances where these techniques have been used in
EPA decisions, and describes criteria that may be useful in determining
whether application of probabilistic methods may be useful and/or
applicable to a specific decision.  The white paper also describes
commonly employed methods to address variability and uncertainty,
including those used in the consideration of uncertainty in scenarios,
uncertainty in models, and variability and uncertainty in the inputs and
outputs of models.  A general description is provided of the range of
methods from simple to complex, rapid to more time-consuming, and least
to most resource-intensive, and opportunities for utilization.  More
detailed examples of applications of these methods are provided, in
Appendix D titled “Case Study Examples of the Application of
Probabilistic Risk Analysis in U.S. Environmental Protection Agency
Decision Making.”

This document does not prescribe a specific approach but, rather,
describes the various stages and aspects of an assessment or decision
process in which probabilistic assessment tools may add value.  This
white paper provides answers to common questions regarding PRA,
including key concepts such as scientific and institutional motivations
for use of PRA, and challenges in the application of probabilistic
techniques.  The white paper describes how PRA can both enhance the
Agency’s credibility and improve decision making.

1. Introduction: Relevance of Uncertainty to Decision making: How
Probabilistic Approaches Can Help

1.0. What Is Probabilistic Risk Analysis, and How Does It Address
Variability and Uncertainty?

Probabilistic analyses include techniques that can be applied formally
to address both variability and uncertainty. Probability is used in
sciences, business, economics, and other fields to examine existing data
and estimate the chance of an event, from health effects to rain to
metal fatigue. One can use probability (chance) to quantify the
frequency of occurrence or the degree of belief in information. For
variability, probability distributions are interpreted as representing
the relative frequency of a given state of the system (i.e., that the
data are distributed a certain way), whereas, for uncertainty, they
represent the degree of belief or confidence that a given state of the
system exists (i.e., that we have the appropriate data) (Cullen and
Frey, 1999). PRA often is defined narrowly to mean a statistical or
thought process used to analyze and evaluate the variability of
available data or to look at uncertainty across data sets.

For the purposes of this document, PRA is a term used to describe a
process that uses probability to incorporate the variability in data
sets, and/or the uncertainty in information such as data or models, into
analyses that support environmental risk-based decision making. PRA is
used here broadly to include both quantitative and qualitative methods
for dealing with scenario, model, and input uncertainty. Probabilistic
techniques can be used with other types of analysis, such as
benefit-cost analysis, regulatory impact analysis, and engineering
performance standards and, thus, is used for a variety of applications
and by experts in many disciplines.

1.1. Goals and Intended Audience

The goals of this white paper are to introduce probabilistic analysis
(PRA) and how it can be used to better inform and improve the
decision-making process, and to provide case  studies where it has been
used in human health and ecological analyses at EPA (Appendix D). A
secondary goal of this paper is to bridge communication gaps regarding
PRA among analysts of various disciplines, between these analysts and
Agency decision makers, and affected stakeholders. The white paper is
also intended to serve as a communication tool to help introduce key
concepts and background information on approaches to risk analysis that
incorporate uncertainty and provide a more comprehensive treatment of
variability. Risk analysts, risk managers, and affected stakeholders can
benefit from understanding the potential uses of PRA. PRA and related
approaches can be used to identify further research that can decrease
uncertainty and more thoroughly characterize variability in a risk
assessment. This white paper will explain how PRA is well suited to
enhancing the decision-making processes in EPA by addressing inherent
uncertainties faced by managers involved in that process.

1.2. Overview of This Document

This document reviews EPA’s interest in and experience with addressing
uncertainty and variability using probabilistic methods; identifies key
questions asked or faced by Agency decision makers; shows how
conventional deterministic approaches to risk analysis may not answer
these questions fully; provides examples of applications; and shows how
and why “probabilistic risk analysis” (broadly defined) provides
added value with regard to regulatory decision making by more fully
characterizing risk estimates. For the purposes of this white paper, PRA
and related tools for both human health and ecological assessments
include a range of approaches from statistical tools, such as
sensitivity analysis, to multi-dimensional Monte Carlo models,
geospatial approaches, and expert elicitation. Key points addressed by
this document include definitions and key concepts pertaining to PRA,
the need for PRA, benefits and challenges of PRA, a general conceptual
framework for PRA, conclusions regarding products and insights obtained
from PRA, and examples where EPA has used PRA in human health and
ecological analyses. A glossary and a bibliography are provided.

1.3. What Are Common Challenges Facing EPA Risk Decision makers?

EPA decision makers face scientifically complex problems that are
compounded by varying levels of uncertainty and variability.  In
reality, uncertainty in risk decisions is unavoidable, since we cannot
perfectly model or predict real world situations, but uncertainty can be
reduced or better characterized through knowledge.  Variability is
inherent in natural systems, and therefore cannot be reduced, but can
also be examined and described. Decision makers often want to know who
is at risk and by how much, the tradeoffs between alternative actions or
decisions, and the likely or possible consequences of decisions. To this
end, it is particularly useful to decision makers to understand the
distribution of risk across potentially impacted populations and
ecological systems. It can be important to know the number of
individuals experiencing different magnitudes of risk, the differences
in risk magnitude experienced by individuals in different life stages or
populations, or the probability of an event which may lead to
unacceptable levels of risk. Given the limitations of data, traditional
methods of risk analyses are not well suited to produce such estimates.
Probabilistic analytical methods are capable of addressing these
shortcomings and can contribute to a more thorough recognition of the
impact of data gaps on the projected risk estimates.

A defensible decision process explicitly takes into account
uncertainties and variability and the rationale or factors influencing
how a decision maker addresses these. Factors such as economics, equity,
feasibility, stakeholder input, and other considerations may also be
part of the decision-making process. Decision making typically contend
with several key factors, including multiple, conflicting objectives,
uncertainty, and alternative regulatory options available to a decision
maker. In addition, decision analysis provides a theoretical foundation
for estimating the value of collecting more information to allow for
more informed decisions.  In the face of uncertainty, decision making is
determined not only by science but also by Agency policy.  Where not
prohibited by statute, the relative costs and benefits of regulatory
alternatives may be considered in making decisions.

If uncertainty and variability have not been well characterized or
acknowledged, potential complications arise in the process of
decision-making that seeks to achieve a balance between over- and
under-regulation. Increased uncertainty can make it more difficult to
determine with reasonable confidence the balance point between costs of
regulation and the implications for avoiding damages and producing
benefits. Characterization of these factors, facilitated by
probabilistic analyses, can provide insight in weighing the relative
costs and benefits of varying levels of regulation and also assist in
risk communication activities.

1.4. What Are Key Questions Often Asked by Decision makers?

Determining decision-maker concerns is a critical first step toward
developing a useful and responsive risk assessment. For example, the
appropriate focus and level of detail of the analysis should be
commensurate with decision maker and stakeholder needs as well as
appropriate use of science. Often, analyses are conducted at a level of
detail dictated by the issue being addressed, the breadth and quality of
the available information upon which to base an analysis, and the
significance surrounding this decision. The analytical process tends to
be an iterative one, and, even though a guiding set of questions may
frame the initial analyses, additional questions can arise that further
direct or even reframe the analyses.

Based on a series of discussions with Agency decision makers and risk
assessors, some questions that are typically posed about risk analyses
include the following:

How representative or conservative is the estimate, (e.g., what is the
variability around an estimate)?

What are the major gaps in knowledge, and what are the major assumptions
used in the assessment? How reasonable are the assumptions?

Would my decision be different if the data were different?  Would
additional data collection and research likely lead to a different
decision? How long will it take to collect the information, how much
would it cost, and would the resulting decision be significantly
different?

Will the use of additional resources, such as a probabilistic approach,
impact the decision making in a timely manner (i.e., better characterize
uncertainties, better identify variability, impact timelines, etc.)?

What are the liabilities/consequences of making a decision under the
current level of  knowledge and uncertainty?

What is the percentile of the population to be protected?

How do the different alternative decision choices and the interpretation
of uncertainty and variability impact the target population?

The questions that arise concerning analyses, including PRA, change
depending on the stage and nature of the decision making and analysis,
from planning and scoping through risk management. The utility of
various levels of analysis and levels of sophistication in answering
these questions are illustrated in the case studies section described in
Section 1.8 and presented in Appendix D.

1.5. Why Is the Implementation of Probabilistic Risk Analysis Important?

The principal reason for the inclusion of PRA as an option in the risk
assessor’s toolbox is PRA’s ability to refine and improve the
information leading to decision making by incorporating known
uncertainties. Beginning as early as the 1980s (NRC, 1983), expert
scientific advisory groups have been recommending that risk analyses
include a clear discussion of the uncertainties in risk estimation. The
National Research Council (NRC, 1994) stated the need to describe
uncertainty and to capture variability in risk estimates. The
Presidential/Congressional Commission on Risk Assessment and Risk
Management (PCRARM, 1997) recommended against a requirement or need for
a “bright line,” or single number, level of risk (see Section 2.5
for more on the scientific community’s opinion on use of PRA).
Regulatory science often requires selection of a limit for a
contaminant, yet that limit always contains uncertainty as to how
protective it is. PRA and related tools quantitatively describe the very
real variations in natural systems and living things and how they
respond to stressors and the uncertainty in estimating those responses.
Risk characterization became EPA policy in 1995, and the principles of
transparency, clarity, consistency, and reasonableness are explicated in
the 2000 Risk Characterization Handbook (EPA, 2000). Transparency,
clarity, consistency, and reasonableness criteria require decision
makers to describe and explain the uncertainties, variability, and known
data gaps in the risk analysis and how they affected resulting
decision-making processes (USEPA, 2000, 1992, 1995).

The use of probabilistic methods also has received support from some
decision makers within the Agency, and these methods have been
incorporated into a number of Agency decisions to date. Program offices
such as the Offices of Pesticides, Solid Waste and Emergency Response,
Air and Radiation, and Water, as well as the Office of Research and
Development have utilized probabilistic approaches in different ways and
to varying extents, for both human exposure and ecological risk
analyses. In addition, the Office of Solid Waste and Emergency Response
has provided explicit guidance on the use of probabilistic approaches
for exposure analysis (EPA, 2001). Some program offices have held
training sessions on Monte Carlo simulation (MCS) software frequently
used in probabilistic analyses.

Where it is useful to refine risk estimates, the use of PRA can help in
the characterization and communication of uncertainty, variability, and
the impact of data gaps in risk analyses, for assessors, decision
makers, and stakeholders including the target population.

1.6. How Does EPA Typically Address Scientific Uncertainty and
Communicate Variability?

Environmental assessments can be complex, such as exposures to multiple
chemicals in multiple media for a wide ranging population. The Agency
often has developed simplified approaches to characterizing risks
through the use of point estimates for model variables or parameters.
Such an approach typically produces point estimates of risks (e.g., 10-5
lifetime probability of cancer risk for an individual). These are often
called “deterministic” assessments. As a result of the use of point
estimates for variables in model algorithms, deterministic risk results
usually are reported as what are assumed to be either average or
worst-case estimates and do not contain any quantitative estimate of the
uncertainty in that estimate, nor report to what percentile of the
exposed population the estimate applies. However, the methods typically
used in EPA risk assessments rely on a combination of point values with
potentially varying levels of conservatism and certainty, yielding a
point estimate of exposure that may be at some unknown point in the
range of possible risks.

Because   SEQ CHAPTER \h \r 1 uncertainty is inherent in all risk
assessments, it is important that the risk assessment process handle
uncertainties in a logical way that is transparent and scientifically
defensible, consistent with the Agency’s statutory mission, and
responsive to the needs of decision makers (NRC, 1994). Thus, when data
are missing, EPA often uses several options to provide some bounds on
uncertainty and variability, in an attempt to avoid risk
underestimation; attempting to give a single quantification of how much
confidence there is in the risk estimate may not be informative or
feasible.  For example, in exposure assessment, the practice at EPA is
to collect new data where needed and where time and resources allow.
Alternatives include narrowing the scope of the assessment, using
screening level default assumptions that include upper-end values and/or
central tendency values that are generally combined to generate risk
estimates that fall within the higher end of the population risk range
(USEPA, 2004), using models to estimate missing values, using surrogate
data (e.g., data on a parameter that come from a different region of the
country than the region being assessed), or using professional judgment.
The use of individual assumptions can range from qualitative (e.g.,
assuming one is tied to the residence location and does not move through
time or space) to more quantitative (e.g., using the 95th percentile of
a sample distribution for an ingestion rate). This approach also can be
applied to the practice of hazard identification and dose-response
assessment when data are missing. Identifying the sensitivity of
exposure or risk estimates to key inputs can help in focusing efforts to
reduce uncertainty by collecting more data.

1.7. What Are the Limitations of Relying on Default-Based Deterministic
Approaches?

Deterministic risk analysis often is considered a traditional approach
to risk analysis because of the existence of established guidance and
procedures regarding its use, the ease with which it can be performed,
and its limited data and resource needs. The use of defaults supporting
deterministic risk assessment provides a procedural consistency that
allows for risk assessments to be feasible and tractable. Risk managers
and members of the public tend to be relatively familiar with
deterministic risk assessment, and use of such an approach addresses
assessment-related uncertainties primarily through the incorporation of
predetermined default values and conservative assumptions. It addresses
variability by combining input parameters intended to be representative
of typical or higher end exposure (considered to be conservative
assumptions). The intention is often to provide a margin of safety or to
construct a screening level estimate of high-end exposure and risk
(i.e., an estimate representative of more highly exposed and susceptible
individuals).

Deterministic risk assessment provides an estimation of exposures and
resulting risks that addresses uncertainties and variabilities in a
qualitative manner. The methods typically used in EPA deterministic risk
assessments rely on a combination of point values(some conservative and
some typical(yielding a point estimate of exposure that is at some
unknown point in the range of possible risks. Such an approach is
believed to more likely overestimate than underestimate risks. Although
this conservative bias (more likely to over- than underestimate risks)
aligns with the public health mission of EPA (USEPA, 2004), the degree
of conservatism in these risk estimates (and in any concomitant
decision) cannot be estimated well or communicated (Hattis and
Burmaster, 1994). This results in unquantified uncertainty in risk
statements.

Estimates generated using these methods are unaccompanied by
quantitative information regarding their precision or potential
systematic error and do not account for the distribution of exposures,
effects, and resulting risks across different members of an exposed
population. Although deterministic risk assessments may present
qualitative information regarding the robustness of the estimates, the
impact of data and model limitations on the quality of the results
cannot be quantified. Reliance on deterministically derived estimations
of risk can result in decision making based solely on point estimates
with an unknown degree of conservatism, which can complicate comparison
of risks or management options.   SEQ CHAPTER \h \r 1 The use of
conservative defaults has long been the target of criticism and has led
to the presumption by critics that EPA assessments are overly
conservative and unrealistic. This criticism may reduce the overall
perceived credibility of an Agency decision.

Deterministic risk assessment is not as well suited as PRA for more
complex assessments, including those of aggregate and cumulative
exposures and time-dependent individual exposure, dose, and effects
analyses. Identification and prioritization of contributory sources of
uncertainty can be difficult and time consuming when using deterministic
methods, leading to difficulties in model evaluation and the subsequent
appraisal of risk estimates (Cullen and Frey, 1999). These comprehensive
quantitative analyses of model sensitivities are essential for the
prioritization of key uncertainties(a critical step in identifying steps
for data collection or research to improve exposure or risk estimates.

1.8. What Is EPA’s Experience with the Use of Probabilistic Risk
Analysis?

To assist with the growing number of probabilistic analyses of exposure
data, EPA issued Guiding Principles for Monte Carlo Analysis (EPA,
1997). Probabilistic analysis techniques such as Monte Carlo analysis,
given adequate supporting data and credible assumptions, can be viable
statistical tools for analyzing variability and uncertainty in risk
assessments. The EPA policy for use of probabilistic analysis in risk
assessment, released in 1997, is inclusive of human exposure and
ecological risk assessments, but does not rule out probabilistic health
effects analyses. Subsequently, EPA’s Science Advisory Board (SAB) and
Scientific Advisory Panel have reviewed PRA approaches to risks by EPA
Offices such as Air and Radiation, Pesticides, and others. Several
programs have developed specific guidance on use of PRA, including
Pesticides and Solid Waste and Emergency Response (EPA, 1998, 2001).

To illustrate the practical application of PRA to problems relevant to
the Agency, several example case studies are briefly described here.
Appendix D, Case Study Examples of the Application of Probabilistic Risk
Analysis in U.S. Environmental Protection Agency Regulatory Decision
Making, discusses these and other case studies in greater detail,
including the procedures and outcome. The examples are intended to
illustrate how some of EPA’s programs and offices currently utilize
PRA. They demonstrate how information from probabilistic analyses,
including  sensitivity analysis, MCS, and other techniques, were used in
decision making. Some of the approaches that are profiled can be used
easily in the planning and scoping of risk assessments and risk
management. Other more complex approaches are used to answer more
specific questions and provide a richer description of the risks. Most
show that PRA can improve or expand information generated by
deterministic methods. The case studies illustrate that the Agency
already has applied the science of PRA to ecological risk and human
exposure estimation and has begun using PRA to describe health effects.
Some of the applications have used existing “off the shelf”
software, whereas others have required significant effort and resources.
Once developed, however, some of the more complex models have been used
many times for different assessments. All have stood the test of
internal and external peer review. A list of the case study examples
presented in Appendix D are provided in  Table 1 including
categorizations based on type of assessment (i.e., human health or
ecological risk assessment); PRA tools used in the assessment; and
program office or region responsible for the assessment.  In several
cases, the examples presented represent components of the overall risk
assessment that demonstrate use of multiple PRA techniques.

A few examples that illustrate the variety of PRA uses in EPA are:

Hudson River PCB-Contaminated Sediment Site: Region 2 evaluated the
variability in risks to anglers who consume recreationally caught fish
contaminated with PCBs from sediment contamination in the Hudson River.
(Case Study 5)

EMAP program:  The Office of Research and Development (ORD) developed
and Office of Water (OW) adopted an applied probabilistic sampling
techniques to evaluate nation’s aquatic resources under CWA Section
305(b) (Case Study 7)

Chromated Copper Arsenate Risk Assessment: ORD and the Office of
Pesticide Programs (OPP) conducted a probabilistic exposure assessment
of children’s exposure (addressing both variability and uncertainty)
to arsenic and chromium from contact with CCA-treated wood playsets and
decks. (Case Study 9)

Evaluating Ecological Effects of Pesticide Uses: OPP developed a
probabilistic model which evaluates acute mortality levels in generic
and specific ecological species for user-defined pesticide uses and
exposures.  (Case Study 13)

PM2.5 Health Impacts: The Office of Air and Radiation (OAR) used expert
elicitation to more completely characterize, both qualitatively and
quantitatively, the uncertainties associated with the relationship
between reduction in PM2.5 and benefits of reduced PM2.5-related
mortality.  (Case Study 14)

2. Probabilistic Risk Analysis

2.1. What Are Variability and Uncertainty, and How Are They Relevant to
Decision-making?

The concepts of variability and uncertainty are introduced here, and the
relevance of these concepts to decision making is discussed.

2.1.1. Variability

Variability refers to real differences over time, space, or members of a
population and is a property of the system being studied (e.g., drinking
water rates for each of the many individual adult residents living in a
specific location) (Cullen and Frey, 1999). Variability can arise from
inherently random processes, such as variations in wind speed over time
at a given location or from true variation across members of a
population that, in principle, could be explained but that, in practice,
may not be explainable using currently available models or data (e.g.,
the range of blood lead levels in 6-year-old children following a
specific degree of lead exposure). Of particular interest in human
health risk assessment is inter-individual variability, which typically
refers to differences between members of the same population in either
behavior related to exposure (e.g., dietary consumption rates for
specific food items) or biokinetics related to chemical uptake or toxic
response (e.g., gastrointestinal uptake rates for lead following
intake).

2.1.2. Uncertainty

Uncertainty is the lack of knowledge of the true value of a quantity or
relationships among quantities. For example, there may be a lack of
information regarding the true distribution variability between
individuals for consumption of certain food items. There are a number of
types of uncertainties for both risk analyses and risk management. The
following description of types of uncertainty (drawn from Cullen and
Frey, 1999) addresses uncertainties that arise during risk analyses.
These uncertainties can be separated broadly into three categories: (1)
scenario uncertainty, (2) model uncertainty, and (3) input (or data)
uncertainty. Each of these is explained below.

Scenario uncertainty refers to errors, typically of omission, resulting
from incorrect or incomplete specification of the risk scenario to be
evaluated. The risk scenario refers to a set of assumptions regarding
the situation to be evaluated, such as (a) the specific sources of
chemical emissions or exposure to be evaluated (one industrial facility
or a cluster of varied facilities impacting the same study area), (b)
the specific receptor populations and associated exposure pathways to be
modeled (e.g., indoor inhalation exposure, track-in dust, or consumption
of home-produced dietary items), and (c) the times or activities to be
considered (e.g., exposure only at home, or consideration of workplace
or commuting exposure). Misspecification of the risk scenario can result
in underestimation, overestimation, or other mischaracterization of
risks.  For instance, underestimation may occur because of exclusion of
relevant situations or inclusion of irrelevant situations with respect
to a particular analysis.  Overestimation may occur because of the
inclusion of unrealistic or irrelevant situations (e.g., assuming
continuous exposure to an intermittent airborne contaminant source
rather than accounting for mobility throughout the day.)

Model uncertainty refers to limitations in the mathematical models or
techniques that are developed to represent the system of interest and
often stems from: (a) simplifying assumptions, (b) exclusion of relevant
processes, (c) misspecification of model boundary conditions (i.e.,
range of input parameters), or (d) misapplication of a model developed
for other purposes. Model uncertainty typically arises when the risk
model relies on missing or improperly formulated processes, structures,
or equations.   Refer to the glossary for additional information.

Input or Parameter uncertainty typically refers to errors in
characterizing empirical values used as inputs to the model (e.g.,
engineering, physical, chemical, biological, or behavioral variables).
Input uncertainty can stem from random or systematic errors involved in
measuring a specific phenomenon (e.g., biomarker measurements such as
the concentration of mercury in human hair); from statistical sampling
errors associated with small sample sizes (if the data are based on
samples selected with a random, representative sampling design); from
the use of surrogate data instead of directly relevant data, or the
absence of an empirical basis for characterizing an input (e.g., absence
of measurements for fugitive emissions from an industrial facility); or
from the use of summary measures of central tendency rather than
individual observations. Nonlinear random processes can exhibit a
behavior that, for small changes in input values, produces large
variation in results.

2.2. When Is Probabilistic Risk Analysis Applicable or Useful?

PRA is useful in the following types of situations (Cullen and Frey,
1999).

When a screening level deterministic risk assessment indicates that
risks are possibly higher than a level of concern, and, therefore, a
more refined assessment is needed

When the consequences of using potentially biased point estimates of
risk are unacceptably high

To estimate the value of collecting additional information to reduce
uncertainty

When significant equity issues are raised by inter-individual
variability

To identify promising critical control points and critical levels when
evaluating risk management alternatives

To rank exposure pathways, sites, contaminants, and so on for purposes
of prioritizing model development or further research

PRA typically is not necessary in the following types of situations
(Cullen and Frey, 1999; EPA 1997).

When a screening-level deterministic risk assessment indicates that
risks are negligible, presuming that the assessment is known to be
biased to produce overestimates of risk

When the cost of averting the exposure and risk is smaller than the cost
of probabilistic analysis

When there is little uncertainty or variability in the analysis (This is
a rare situation.)

2.3. How Can Probabilistic Risk Analysis Be Incorporated into
Assessments?

As illustrated in the accompanying case studies (Appendix D),
probabilistic approaches can be incorporated into any stage of a risk
assessment, from problem formulation or planning and scoping to analysis
of alternative risk management decisions. In some situations, PRA can be
used selectively for components of an assessment. It is common in
assessments that some model inputs are known with high confidence (i.e.,
based on site-specific measurements), whereas values for other inputs
are less certain (i.e., based on surrogate data collected for a
different purpose). For example, an exposure modeler may determine that
there is relevant air quality monitoring data but a lack of detailed
information of human activity patterns in different microenvironments.
Thus, an assessment of variability in exposure to airborne pollutants
might be based on direct use of the monitoring data, whereas assessment
of uncertainty and variability in the inhalation exposure component
might be based on statistical analysis of surrogate data or use of
expert judgment. The uncertainties are likely larger for the latter than
the former component of the assessment; thus, efforts to characterize
uncertainties associated with pollutant exposures would focus on the
latter.

2.4. What Are the Scientific Community’s Views on Probabilistic Risk
Analysis, and What Is the Institutional Support for Its Use?

The NRC recently emphasized its long-standing advocacy for probabilistic
risk assessment (NRC, 2007a,b). Dating from its 1983 Risk Assessment in
the Federal Government (NRC, 1983)—which first formalized the risk
assessment paradigm—through various reports released in the late
1980s, all during the 1990s, and through the early 2000s, various NRC
panels have consistently maintained that—because risk analysis
involves substantial uncertainties—these uncertainties should be
evaluated within a risk assessment. These panels noted,

in 1989, that when evaluating total population risk, EPA should consider
the distribution of exposure and sensitivity of response in the
population (NRC, 1989);

in 1991, that when assessing human exposure to air pollutants, EPA
should present model results along with estimated uncertainties (NRC,
1991);

in 1993, that when conducting ecological risk assessments, EPA should
discuss thoroughly uncertainty and variability within the assessment
(NRC, 1993); and,

in 1994, an NRC report, Science and Judgment in Risk Assessment, stated
that “uncertainty analysis is the only way to combat the ‘false
sense of certainty,’ which is caused by a refusal to acknowledge and
[attempt to] quantify the uncertainty in risk predictions” (emphasis
in the original) (NRC, 1994). In 2002, another report suggested that
EPA’s estimation of health benefits were not wholly credible because
EPA failed to deal formally with uncertainties in its analyses (NRC,
2002).

Asked to recommend improvements to the Agency’s human health risk
assessment practices, EPA’s Science Advisory Board echoed the NRC’s
sentiments and urged the Agency to characterize variability and
uncertainty more fully and more systematically and to replace
single-point uncertainty factors with a set of distributions using
probabilistic methods (Parkin and Morgan, 2007). The key principles of
risk assessment cited by the Office of Science and Technology Policy
(OSTP) and the Office of Management and Budget (OMB) include
“explicit” characterization of the uncertainties in risk judgments;
they go on to cite the National Academy of Science’s 2007
recommendation to address “variability of effects across potentially
affected populations” (OSTP/OMB, 2007).

2.5. How Can Probabilistic Risk Assessment Provide More Comprehensive,
Rigorous Scientific Information in Support of Regulatory Decisions?

External stakeholders in the past have used the Administrative Procedure
Act and the Data Quality Act to challenge the Agency for a lack of
transparency and consistency or for not fully analyzing and
characterizing the uncertainties in risk assessments or risk management
decisions (see Fisher et al., 2006). The more complete implementation of
PRA and related approaches to deal with uncertainties in decision-making
lends support to the overall Agency risk-based decision-making process.

The results of any assessment, including PRA, are dependent on the
underlying methods and assumptions. Accompanied by the appropriate
documentation, PRA may communicate a more robust representation of risks
and corresponding uncertainties. This characterization may take shape in
the form of a range of possible estimates as opposed to the more
traditionally presented single-point values. Depending on the needs of
the assessment, ranges can be derived for variability and uncertainty
(or a combination of the two) in both model inputs and resulting
estimations of risk.

2.6. Are There Additional Advantages of Using Probabilistic Risk
Analysis?

PRA quantifies how exposures, effects, and risks differ among
individuals and provides an estimation of the degree of confidence with
which these estimates may be made, given the current uncertainty in
scientific knowledge and available data. A 2007 NRC panel stated that
the objective of PRAs is not to decide “how much evidence is
sufficient” to adopt an alternative but, rather, to describe the
scientific bases of proposed alternatives so that scientific and policy
considerations may be more fully evaluated (NRC, 2007a). EPA’s SAB
similarly noted that PRAs provided more “value of information”
through quantitative assessment of uncertainty, and clarify the science
underlying Agency decisions (EPA, 2007).

The SAB proposed a number advantages Agency decision-makers could reap
from utilization of probabilistic methods (Parkin and Morgan, 2007).

A probabilistic reference dose could help reduce the potentially
inaccurate implication of zero risk below the RfD.

By understanding and explicitly accounting for uncertainties underlying
a decision, EPA can estimate formally the value of gathering more
information. By doing so, two benefits follow: (1) EPA can better
prioritize its research needs by investing in areas that yield the
greatest information value; and (2) when making decisions, the Agency
can eschew the less-than-helpful rationale of “too much scientific
uncertainty.” By candidly acknowledging uncertainty’s ubiquity, EPA
can base a decision on more intellectually robust concepts of comparing
present risks against the costs of gathering more information.

By adopting PRA, EPA sends the appropriate signal to the intellectual
marketplace, thereby encouraging analysts to gather data and to develop
methodologies necessary for assessing uncertainties.

2.7. What Are the Challenges to Implementation of Probabilistic
Analyses?

Currently, EPA is using PRA in a variety of programs to support
decisions, but challenges remain regarding the expanded use of these
tools within the Agency; these include those that follow.

A lack of understanding of the value of PRA to decision making;

The perception, if not reality, that PRA requires additional resources;

Limited resources (staff, time, training, or methods) to conduct PRA;

A lack of clear directive or requirement to utilize PRA in many cases;

Lack of understanding of how to incorporate the results of probabilistic
analyses into decision making and how to establish action levels based
on the scope of the assessment.

The complexity of communicating probabilistic analysis results.

Fuller characterization of the uncertainties in a risk estimate through
use of PRA can lead to more difficult decision making or more
complicated risk communication

The ability to simulate and reanalyze numerous scenarios using various
models and input data could lead to prolonged analyses and delay
decisions

These challenges are discussed in more detail below.

2.8. How Can Probabilistic Risk Analysis Support Specific Regulatory
Decision making?

Decision makers sometimes perceive that the binary nature of regulatory
decisions (e.g. Does an exposure exceed a reference dose or not? Do
emissions comply with Agency standards or not?) precludes the use of a
range of uncertainty, compared with the use of point estimates.
Generally, it is legally necessary to explain the rationale underlying a
particular decision. PRA’s primary purpose is to provide information
so that decisions are based on the best available science; it is not to
necessarily displace legally mandated decisions with a range of
alternatives. By doing a sensitivity analysis of the influence of the
uncertainty on the decision-making process, it can be determined how or
if PRA can help improve the process.

2.9. Does Probabilistic Risk Analysis Require More Resources Than
Default-Based Deterministic Approaches?

PRA can generally be expected to require more resources than standard
Agency default-based deterministic approaches. There is extensive
experience within EPA in conducting and reviewing deterministic risk
assessments. These assessments tend to follow standardized methods that
minimize the effort required to conduct them and to communicate the
results. Probabilistic assessments often entail a more detailed
analysis, and, as a result, there exists a common perception that these
assessments require substantially more resources than do deterministic
approaches.

Appropriately trained staff and the availability of adequate tools,
methods, and guidance are essential for application of PRA. Proper
application of probabilistic methods requires not only software and data
but also guidance and training for both analysts using the tools and for
managers and decision makers tasked with interpreting and communicating
the results. In most circumstances, probabilistic assessments may take
more time and effort to conduct than conventional approaches, primarily
because of the comprehensive inclusion of available information on model
inputs.

An upfront increase in resources needed to conduct a probabilistic
assessment can be expected, but development of standardized approaches
and/or methods can lead to the routine incorporation of PRA in Agency
approaches (e.g., the Office of Pesticide Programs’ use of DEEM, a
probabilistic dietary exposure model). The initial and, in some cases,
ongoing resource cost (e.g., that for development of site-specific
models for site assessments) may be offset by a more informed decision
than a comparable deterministic analysis. Probabilistic methods are
useful for identifying effective risk management options and in
prioritizing additional data collection or research aimed at improving
risk estimation, ultimately resulting in management options that enable
improved environmental protection while, simultaneously, conserving
greater resources.

2.10. Doesn’t Probabilistic Risk Analysis Require More Data Than
Conventional Approaches?

There are differences in opinion within the technical community as to
whether PRA requires more data than other types of analyses. Although
some emphatically believe that PRA requires more data, others argue that
probabilistic assessments make better use of all of the available data
and information. Stahl et al. (2005) discuss when and how much data are
necessary for a decision. PRA can benefit from more data than might be
used in a deterministic risk assessment. For example, where
deterministic risk assessments may employ selected point estimates (such
as mean or 95th percentile values) from available datasets for use in
model inputs, PRA facilitates the use of frequency-weighted data
distributions, allowing for a more comprehensive consideration of the
available data. In many cases, the data that were used to develop the
presumptive 95th percentile can be employed in the development of
probabilistic distributions.

Restriction of PRA to data-rich situations may prevent its application
where it is most useful. Because PRA incorporates information on data
quality, variability, and uncertainty into risk models, the influence of
these factors on the characterization of risk can become a greater focus
of discussion and debate.

A key benefit of using PRA is its ability to reveal limitations, as well
as strengths, of data that often are masked by a deterministic approach.
In doing so, PRA can help inform research agendas, as well as support
regulatory decision making, based on the state of the best available
science. In summary, PRA typically requires more time for developing
input assumptions than a corresponding deterministic risk assessment,
but when incorporated in the relevant steps of the risk assessment
process, PRA can demonstrate real added benefits.  In some cases PRA can
provide additional interpretations that compensate for the additional
efforts.

2.11. Can Probabilistic Risk Analysis Be Used To Screen Risks or Only in
Complex or Refined Assessments?

Probabilistic methods typically are not necessary where traditional
default-based deterministic methods are adequate for screening risks.
Such methods are relatively low cost, are intended to produce
conservatively biased estimates, and are useful for identifying
situations in which risks are so low that no further action is needed.
The application of probabilistic methods can be targeted to situations
in which a screening approach indicates that a risk may be of concern or
when the cost of managing the risk is high, creating a need for
information to help inform risk management decisions. PRA fits directly
into a graduated hierarchical approach to risk analysis. PRA also could
be used to more fully examine the existing default-based methods based
on the current state of information and knowledge to determine if such
methods are truly conservative and adequate for screening.

2.12. Does Probabilistic Risk Analysis Present Unique Challenges to
Model Evaluation?

The concept of “validation” of models used for regulatory decision
making has been a topic of heated discussion. In a recent report on the
use of models in environmental regulatory decision making, NRC
recommended the use of the notion of model “evaluation” rather than
“validation,” suggesting use of a process that encompasses the
entire life cycle of the model and recognizes the spectrum of interested
parties in the application of the model, which often extends beyond the
model builder and decision maker. Such a process can be designed to
ensure that judgment of the model application is based not only on its
predictive value determined from comparison with historical data but
also on its comprehensiveness, rigor in development, transparency, and
interpretability (NRC, 2007b).

Model evaluation is important in all risk assessments. In the case of
PRA, there is an additional question as to the validity of the
assumptions made regarding probability and frequency distributions for
model inputs and their dependencies. Probabilistic information can be
accounted for during evaluation analyses by considering the range of
uncertainty in the model prediction and whether such a range overlaps
with the “true” value based on independent data. Thus, probabilistic
information can aid in characterizing the precision of the model
predictions and whether a prediction is significantly different from a
benchmark of interest.  For example, comparisons of probabilistic model
results and monitoring data were done for multiple models in developing
the cumulative pesticide exposure model.  There are also published
concurrent PRA model evaluations using a Bayesian analysis (Clyde,
2000).

2.13. How Do You Communicate Results of Probabilistic Risk Analysis?

The approaches for reporting results from PRA vary depending on the
assessment objective and the intended audience. Beyond the basic 1997
principles and the policy from the same year, the Risk Assessment
Guidance for Superfund: Volume III also provides some guidance on the
quality and criteria for acceptance as well as communication basics
(EPA, 2001). There have been limited studies of how information from PRA
regarding variability and uncertainty can or should be communicated to
key audiences such as decision makers and stakeholders (e.g., Morgan and
Henrion, 1990; Bloom et al., 1993; Krupnick et al., 2006). Among the
analyst community, there is often an interest in visualization of the
structure of a scenario and model using influence diagrams and depiction
of the variability and uncertainty in model inputs and outputs using
probability distributions in the form of cumulative density functions or
probability

Figure 1. Graphical Description of the Likelihood (Probability of Risk)
of Toxicity

(Fitted data distribution and confidence intervals)

Source: Frey, H.C. (2004)

distribution functions (Figure 1). Sensitivity of the model output to
variability and uncertainty in model inputs can be depicted using
graphical tools.

In some cases, these graphical methods can be useful for those less
familiar with PRA, but in many cases there is a need to translate the
quantitative results into a message that extracts the key insights
without burdening the decision maker with obscure technical details. In
this regard, the use of ranges of values for a particular metric of
decision-making relevance (e.g., range of uncertainty associated with a
particular estimate of risk) may be adequate. The presentation of PRA
results to a decision maker may be conducted best as an interactive
discussion, in which a principal message is conveyed, followed by
exploration of issues such as the source, quality, and degree of
confidence associated with the information. There is a need for
development of recommendations and a communication plan regarding how to
communicate the results of PRA to decision makers and stakeholders,
building on the experience of various programs and regions in this area.

2.14. Are the Results of Probabilistic Risk Analysis Difficult To
Communicate to Decision-makers and Stakeholders?

Research has shown that the ability of decision makers to deal with
concepts of probability and uncertainty is variable. Bloom et al. (1993)
surveyed a group of senior managers at EPA and found that many could
interpret information about uncertainty if it was communicated in an
appropriate manner that was responsive to decision-maker interests,
capabilities, and needs. In a more recent survey of ex-EPA officials,
Krupnick et al. (2006) concluded that most had difficulty understanding
information on uncertainty, and that certain formats used to present
uncertainty information were more effective than others. The findings of
these studies highlight the need for practical strategies for
communication of results of PRA and uncertainty information between risk
analysts and decision makers, as well as between decision makers and
other stakeholders. The Office of Emergency and Remedial Response has
compiled guidance to assist analysts and managers in understanding and
communicating the results of PRA (EPA, 2001).

3. Findings and Recommendations

3.1 Findings: How Probabilistic Risk Analysis and Related Analyses Can
Improve the Decision-making Process at EPA

PRA is an analytical methodology capable of incorporating information
regarding uncertainty and/or variability in analyses to provide insight
regarding the degree of certainty of a risk estimate and how the risk
estimate varies among different members of an exposed population.
Traditional approaches often report risks as “central tendency” or
“high end (e.g., 90th percentile or above),” or “maximum
anticipated exposure” and PRA can be used to more fully describe
uncertainty surrounding such estimates and identify the key contributors
to variability or uncertainty in predicted exposures or risk estimates.
This information then can be used by decision makers to achieve a
science-based level of safety, to weigh alternative risk management
options, or to invest in researching areas which have the greatest
uncertainty and impact on the risk estimates.

Using PRA, one can obtain insight regarding whether one risk management
strategy is more likely to reduce risks compared to another, and by how
much. The methodology facilitates the investigation of potential changes
in decisions that may result from the collection of additional
information that could better characterize variability and potentially
reduce uncertainty and helps determine how expenses incurred by
activities to reduce uncertainty are offset by improved decision-making
capabilities gained from the acquisition of that knowledge. PRA can
facilitate the construction and simultaneous consideration of multiple
model alternatives.  Probabilistic methods offer a number of tools
designed to promote robust management and increased confidence in
decision making through the incorporation of input variability and
uncertainty characterization and prioritization in risk analyses. For
example, sensitivity analyses can be used to identify influential
knowledge gaps involved in the estimation of risk, allowing for improved
transparency and the ability to more clearly communicate or articulate
the most relevant information to decision makers and stakeholders.
Ultimately, PRA can enhance the Agency’s credibility in its approach
to science-based decision making.

3.2. Recommendations for Enhanced Utilization of PRA in EPA

The various tools and methods discussed in this paper can be used at all
stages of risk analysis and also aid the decision-making process by
characterizing inter-individual variability and uncertainties.
Probabilistic analyses and related methods are in use in varying degrees
across the Agency:

The use of Monte Carlo or other probability based techniques to derive a
range of possible outputs from uncertain inputs is a fairly
well-developed approach within EPA.

Although basic guidance exists at EPA on the use and acceptability of
PRA for risk estimation, implementation varies greatly within programs,
offices and regions.

Although highly sophisticated human exposure assessment and ecological
risk applications have been developed, use of PRA models to assess human
health effects and dose-response has been somewhat limited.

Enhanced use of PRA and consistent applications in support of EPA
decision-making requires improved internal capacity for conducting these
assessments, as well as interpreting and communicating such information
in the context of decisions.  Such improvements of internal capacity
could be accomplished through sharing of experiences, knowledge, and
training, improved policy and guidance, and increased availability of
tools and methods.

Some steps to improve implementation include:

Inform risk managers about the advantages and disadvantages in using PRA
techniques in their decision-making process through lectures, webinars,
and communication regarding the techniques and their use in EPA.

Train risk assessors so that they can learn about the various tools
available, their applications, software and review considerations, and
resources for additional information (e.g., experts and support services
within the Agency).

Meetings and discussions of PRA techniques and their application with
both managers and assessors will aid in providing greater consistency
and transparency to EPA’s risk assessment process and in developing
EPA’s internal capacity.

Demonstrate through informational opportunities and resource libraries
the various tools and methods that can be used at all stages of risk
analysis and also aid the decision-making process by characterizing
inter-individual variability and uncertainties.

Promote the sharing of experience, knowledge, models, and best practices
via meetings of risk assessors and risk managers; electronic exchanges,
such as the EPA Portal Environmental Science Connector; and more
detailed discussions regarding the case studies.

Provide easily available, flexible, modular training for all levels of
experience to familiarize employees to the menu of tools and their
capacities.

Provide introductory as well as advanced training open to all offices.

Provide live and recorded seminars and Webinars for introductory and
supplemental education, as well as periodic, centralized hands-on
training sessions on how to utilize software programs.

Risk assessors, risk managers, and decision makers need to be provided
the information and training necessary so that they can better utilize
these tools. Education and experience will generate familiarity with
these tools that will then lead analysts and decision makers to better
understand the techniques and consider more fully utilizing these
techniques.

3.3 Guidance and Policy

Additional guidance can be developed to help analysts and decision
makers decide which statistical tools to use and when to use them, and
how probabilistic information can help to inform the basis of those
decisions. Both deterministic and probabilistic approaches and other
statistical methods may be useful at any stage of the risk analysis and
decision-making process, from planning and scoping to characterizing and
communicating uncertainty. Such bodies as the EPA’s Science Policy
Council can play a role in directing guidance development to help
implement probabilistic and related tools.  Examples of guidance needed
include:

Probabilistic approaches to evaluating health effects data

Probabilistic approaches to ecological risk assessment

Integrating probabilistic exposure and risk estimates and communicating
uncertainty and variability.

3.4 Challenges

In general terms, while PRA techniques are currently available which
would help inform EPA decision-making process, research, as well as
guidance, is needed to further improve these methods for more complete
implementation of PRA in human health and ecological risk assessment.
Some examples include:

Although highly sophisticated human exposure assessment and ecological
risk applications have been developed, use of PRA models to evaluate
toxicity data has been very limited. Scientific, technical, and science
policy discussions are needed in this area.

There is no consensus on any one well-accepted general methodology for
dealing with model uncertainty, although there are various examples of
efforts to do so. Thus, additional research on formal methods for
treating model uncertainties will be valuable.

As noted in Appendix A.3, there are significant challenges to properly
account for variability and uncertainty when multiple models are coupled
together to represent the source-to-outcome continuum (e.g., the
OPP-Environmental Fate and Effects Division’s aquatic and terrestrial
models). Moreover, the coupling of multiple models (e.g., emissions, air
quality, exposure, dose, effect) may need to involve inputs and
corresponding uncertainties that are incorporated into more than one
model, potentially resulting in complex dependencies (e.g., ambient
temperature affects emission rates, air quality, and human activity that
influence total emissions and exposures).

There may be mismatches in the temporal and spatial resolution of each
model, which confound the ability to propagate variability and
uncertainty from one model to another. For some models, the key
uncertainties may be associated with inputs, whereas, for other models,
the key uncertainties may be associated with structure or
parameterization alternatives

Appendix A: An Overview of Some of the Techniques Used in Probabilistic
Risk Analysis

A.1. What Is the General Conceptual Approach in Probabilistic Risk
Analysis?

PRA includes several major steps, which parallel the accepted
environmental health risk assessment process. These include (1) problem
and/or decision criteria identification, (2) getting information, (3)
interpreting the information, (4) selecting and applying models and
methods for quantifying variability and/or uncertainty, (5) quantifying
inter-individual or population variability and uncertainty in metrics
relevant to decision-making, (6) sensitivity analysis to identify key
sources of variability and uncertainty, and (7) reporting of results.

Problem identification deals with identifying the assessment end points,
or issues, that are relevant to the decision-making process, as well as
to other stakeholders, and that can be addressed in a scientific
assessment process. Following problem identification, information is
needed from stakeholders and experts regarding the scenarios to
evaluate. Based on the scenarios and assessment endpoints, the analysts
select or develop models, which in turn leads to identification of model
input data requirements and acquisition of data or other information
(e.g., expert judgment encoded as the result of a formal elicitation
process) from which to quantify inputs to the models. The data or other
information for model inputs is interpreted in the process of developing
probability distributions to represent variability, uncertainty, or both
for a particular input. Thus, the steps (1) through (4) listed above are
highly interactive and iterative in that the data input requirements and
how information is to be interpreted depend on the model formulation,
which depends on the scenario, which in turn depends on the assessment
objective. The assessment objective may have to be refined depending on
the availability of information.

Once a scenario, model, and inputs are specified, the model output is
estimated. A common approach is to use Monte Carlo or other
probabilistic methods for generating samples from the probability
distributions of each model input, run the model based on one random
value from each probabilistic input, and produce one corresponding
estimate of the model outputs. This process is repeated typically
hundreds or thousands of times to create a synthetic statistical sample
of model outputs. These output data are interpreted as a probability
distribution of the output of interest. Sensitivity analysis can be
performed to determine which model input distributions are most highly
associated with the range of variation in the model outputs. The results
may be reported in a wide variety of forms depending on the intended
audience, ranging from qualitative summaries to tables, graphs, and
diagrams.

A.2. What Are the Multiple Types of Probabilistic Risk Analyses, and How
Are They Used?

There are multiple levels for conducting risk assessments. Graduated
approaches to analysis are widely recognized (e.g., EPA, 1997; EPA,
2001; WHO, 2007). The idea of a graduated approach is to choose a level
of detail and refinement for an analysis that is appropriate to the
assessment objective, data quality, information available, and
importance of the decision (e.g., resource implications).

Detailed introductions to PRA methodology are available elsewhere, such
as Ang and Tang (1984), Cullen and Frey (1999), EPA (2001), Morgan and
Henrion (1990) and EPA, 2001. A few key aspects of PRA methodology are
briefly mentioned here. However, readers who seek more detail should
consult these references and see the bibliography for additional
references.

The deterministic risk assessment approaches described in Section 1.6
are examples of lower levels in a graduated approach to analysis, in
which risk at the lower levels of analysis is assessed by conservative,
bounding assumptions. If the risk estimate is found to be very low
despite use of conservative assumptions, then there exists a great deal
of certainty that the actual risks to the population of interest for the
given scenario are below levels of concern and, thus, that no further
intervention is required, assuming the model specification is correct.
However, when a conservative deterministic risk assessment indicates
that a risk may be high, it is possible that the risk estimate is
biased, and the actual risk may be lower. In such a situation, depending
on the resource implications of risk management, it may be appropriate
to proceed with a more refined, or higher level, analysis. If the cost
of intervention is less than the cost of further analysis, then it may
be appropriate to simply proceed to the risk management decision as a
preventive measure that is also expedient. In some deterministic
assessments, for instance, for ecological risks, the assumptions are not
well assured of conservatism and the estimated  risks might be biased to
appear lower than the unseen actual risk

A more refined analysis could involve applications of deterministic risk
assessment methods but with alternative sets of assumptions intended to
characterize central tendency and reasonable upper bounds of exposure,
effects, and risk estimates, such that the estimates could be for an
actual individual in the population of interest (rather than a
hypothetical maximally exposed individual). However, such analyses are
not likely to provide quantification regarding the proportion of the
population at or below a particular exposure or risk level of concern,
uncertainties for any given percentile of the exposed population, nor
priorities among input assumptions with respect to their contributions
to variability and uncertainty in the estimates.

To more fully answer the questions often asked by decision-makers, the
analysis can be further refined by incorporating quantitative
comparisons of alternative modeling strategies (to represent structural
uncertainties associated with scenarios or models), by quantifying
ranges of variability and uncertainty in model outputs, and by providing
the corresponding ranges for model outputs of interest. When performing
probabilistic analyses such as these, choices are made regarding whether
to focus on quantification of variability only, uncertainty only, both
variability and uncertainty co-mingled (representing a randomly selected
individual), or variability and uncertainty distinguished (e.g., in a
two-dimensional depiction of probability bands for estimates of
inter-individual variability) (see Figure 2). The simultaneous but
distinct propagation of variability and uncertainty in a two dimensional
framework enables quantification of uncertainty in the risk for any
percentile of the population.  For example, one could estimate the range
of uncertainty in the risk faced by the median member of the population
or the 95th percentile member of the population. Such information can be
used by a decision maker (for example) to gauge the confidence that
should be placed in any particular estimate of risk, as well as to
determine whether additional data collection or information might be
useful to reduce the uncertainty in the estimates. The OPP assessment of
chromated copper arsenate treated wood (see Appendix D) used such an
approach.

Figure 2: Diagrammatic comparison between three alternative
probabilistic approaches for the same exposure assessment. In option 1,
only variability is quantified. In option 2, both variability and
uncertainty are propagated together. In option 3, variability and
uncertainty are propagated separately. MC = Monte Carlo. 1D = one
dimensional; 2D = two dimensional.   Source: WHO (2008)

When conducting an analysis for the first time, it may not be known or
clear, prior to analysis, which components of the model or which model
inputs contribute the most to the estimate risk or its variability and
uncertainty. However, as a result of completing an analysis, the analyst
often gains insight into both strengths and weaknesses of the models and
input information. Probabilistic analysis and sensitivity analysis can
be used together to identify the key sources of quantified uncertainty
in the model outputs to inform decisions regarding priorities for
additional data collection. Ideally, time should be allowed for
collecting such information and refining the analysis to arrive at a
more representative and robust estimate of variability and uncertainty
in risk. Thus, the notion of iteration in developing and improving an
analysis is widely recommended.

The notion of iteration can be applied broadly to the risk assessment
framework. For example, a first effort to perform an analysis may lead
to insight that the assessment questions might be impossible to address,
or that there are additional assessment questions that may be equally or
more important. Thus, iteration can include reconsideration of the
initial assessment questions and the corresponding implications for
definition of scenarios, selection of models, and priorities for
obtaining data for model inputs. Alternatively, in a time-limited
decision environment, such probabilistic and sensitivity analyses may
offer insight into the effect of risk management options on risk
estimates.

A.3. What Are Some Specific Aspects of and Issues Related to Methodology
for Probabilistic Risk Analysis?

This section briefly touches on a few key aspects of PRA, model
development, and associated uncertainties. Detailed introductions to PRA
methodology are available elsewhere, such as Ang and Tang (1984), Cullen
and Frey (1999), EPA (2001), and Morgan and Henrion (1990). Readers who
seek more detail should consult these references and see the
bibliography for additional references.

A.3.1. Developing a Probabilistic Risk Analysis Model

There are a number of key issues that should be considered in developing
a PRA model. Some of these are outlined below.

Structural Uncertainty in Scenarios

A potentially key source of uncertainty in an analysis is the scenario,
which includes specification of pollutant sources, transport pathways,
exposure routes, timing and locations, geographic extent, and related
issues. As yet, there appears to be no formalized methodology for
dealing quantitatively with uncertainty and variability in scenarios.
Decisions regarding what to include or exclude from a scenario could be
recast as hypotheses regarding which agents, pathways,
microenvironments, and so on contribute significantly to the overall
exposure and risk of interest. In practice, however, the use of
qualitative methods tends to be more common, given the absence of a
formal quantitative methodology.

Coupled Models

For source-to-outcome risk assessments, it is often necessary to work
with multiple models, each of which represents a different component of
a scenario. For example, there may be separate models for emissions, air
quality, exposure, dose, and effects. Such models may have different
spatial and temporal scales. When conducting an integrated assessment,
there may be significant challenges and barriers to coupling such models
into one coherent framework. Sometimes, the coupling is done dynamically
in a software environment. In other cases, the output of one model might
be processed manually to prepare the information for input to the next
model. Furthermore, there may be feedbacks between components of the
scenario (e.g., poor air quality might affect human activity, which, in
turn, could affect both emissions and exposures) that are incompletely
captured or not included at all. Thus, the coupling of multiple models
can be a potentially significant source of structural uncertainty.

A.3.2. Conducting the Probabilistic Analysis

Quantifying Variability and Uncertainty in Model Inputs and Parameters

Once the models are selected or developed to simulate a scenario of
interest, attention typically turns to development of input data for the
model. There is a substantial amount of literature regarding the
application of statistical methods for quantifying variability and
uncertainty in model inputs and parameters based on empirical data
(e.g,. Ang and Tang, 1984; Cullen and Frey, 1999; Morgan and Henrion,
1990; EPA 2001). For example, a commonly used method for quantifying
variability in a model input is to obtain a sample of data, select a
type of parametric probability distribution model to fit to the data
(e.g., normal, lognormal, or other form), estimate the parameters of the
distribution based on the data, critique the goodness-of-fit using
graphical (e.g., probability plot) and statistical methods (e.g.,
Anderson-Darling, Chi-Square, or Kolmogorov-Smirnov tests), and choose a
preferred fitted distribution. This methodology can be adjusted to deal
with various types of data, such as data that are samples from mixtures
of distributions or that contain nondetected (censored) values.
Uncertainties can be estimated based on confidence intervals for
statistics of interest, such as mean values, or the parameters of
frequency distributions for variability.  Various texts and guidance
documents, both Agency and programmatic, describe these approaches,
including the Guiding Principles for Monte Carlo Analysis (EPA, 1997)
and the internet site learner.org.

The most commonly used method for estimating a probability distribution
in the output of a model, based on probability distributions specified
for model inputs, is Monte Carlo Simulation  (MCS) (Cullen and Frey,
1999; Morgan and Henrion, 1990). MCS is popular because it is very
flexible. MCS can be used with a wide variety of different types of
probability distributions as well as different types of models. The main
challenge for MCS is that it requires repetitive model calculations to
construct a set of pseudo-random numbers for model inputs and the
corresponding estimates for model outputs of interest. There are
alternatives to MCS that are similar but more computationally efficient,
such as Latin Hypercube Sampling (LHS). Techniques are available for
simulating correlations between inputs in both MCS and LHS. For models
with very simple functional forms, it may be possible to use exact or
approximate analytical calculations, but, in practice, such situations
are encountered infrequently.

There may be situations in which the data do not conform to a
well-defined probability distribution. For such situations, Markov Chain
Monte Carlo is an algorithm that samples the data iteratively and
randomly to estimate a so-called “likelihood function” (i.e., the
probability distribution and parameter estimates that provide the most
likely explanation of the data). The likelihood function is a key
component of   HYPERLINK  \l "_Glossary"  Bayesian  inference and,
therefore, serves as the basis for some of the analytical approaches to
variability and uncertainty described below.

The use of empirical data presumes that the data are a representative,
random sample. However, if there are known biases or other data quality
problems, or if there is a scarcity or absence of relevant data, then
reliance on available empirical data is likely to lead to misleading
inferences in the analysis. Alternatively, estimates of variability and
uncertainty can be encoded, using formal protocols, based on elicitation
of expert judgment (e.g., Morgan and Henrion, 1990). Elicitation of
expert judgment for subjective probability distributions is used in
situations where there are insufficient data to support a statistical
analysis of uncertainty but in which there is sufficient knowledge on
the part of experts to make an inference regarding uncertainty. For
example, EPA has recently conducted an expert elicitation study on the
concentration-response relationship between annual average ambient PM2.5
exposures and annual mortality (IEC, 2006; see also Case Studies 6 and
14). Subjective probability distributions that are based on expert
judgment can be “updated” with new data as they become available
using Bayesian statistical methods.

Structural Uncertainty in Models

There may be situations in which it proves useful to evaluate not just
the uncertainties in inputs and parameter values, but also uncertainties
regarding whether a model adequately captures, in a hypothesized,
mathematical, structured form, the relationship under investigation. A
qualitative approach to evaluating the structural uncertainty in a model
is to describe critical assumptions within a model, the documentation of
a model, or model quality. Quantitative approaches to evaluating
structural uncertainty in models are manifold. These include
parameterization of a general model that can be reduced to alternative
functional forms (e.g., Morgan and Henrion, 1990), enumeration of
alternative models in a probability tree (e.g., Evans et al., 1994),
comparing alternative models by evaluating likelihood functions (e.g.,
Royall, 1997; Burnham and Anderson, 2002), pooling results of model
alternatives using Bayesian updating (e.g., Hoeting et al., 1999), or
testing the causal relationships within alternative models using
Bayesian Networks (Pearl 2000).

Sensitivity Analysis: Identifying the Most Important Model Inputs

Sensitivity analysis is complementary to probabilistic methods. There
are many types of sensitivity analysis methods, including, for example,
simple techniques that involve changing the value of one input at a time
and assessing the effect on an output and statistical methods that
evaluate which of many simultaneously varying inputs contributes the
most to the variance of the model output. Sensitivity analysis can
answer the following key questions.

What is the impact of changes in input values on model output?

How can variation in output values be apportioned among model inputs?

What are the ranges of inputs associated with best or worst outcomes?

What are the key controllable sources of variability?

What are the critical limits (e.g., emission reduction target for a risk
management strategy)?

What are the key contributors to the output uncertainty?

Thus, sensitivity analysis can be used to inform decision making
regarding research priorities and risk management.

Probabilistic methods typically focus on the forward propagation of
uncertainty or variability in the input to a model with respect to
uncertainty or variability in a model output. However, once a
probabilistic analysis is completed, sensitivity analysis typically
takes the perspective of looking backwards to evaluate how much of the
variation in the model output is attributable to individual model inputs
(e.g., Frey and Patil, 2002; Mokhtari et al., 2006; Saltelli et al.,
2004).

Iteration

There are two major types of iteration in risk assessment modeling. One
is iterative refinement of the type of analysis, perhaps starting with a
relatively simple deterministic risk assessment as a screening step in
an initial level of analysis and proceeding to more refined types of
assessments as needed in subsequent levels of analysis. Examples of more
refined levels of assessment include application of sensitivity analysis
to deterministic risk assessment; the use of probabilistic methods to
quantify variability only, uncertainty only, or co-mingled variability
and uncertainty (to represent a randomly selected individual); or the
use of two-dimensional probabilistic methods for distinguishing and
simultaneously characterizing both variability and uncertainty.

The other type of iteration occurs within a particular level and
includes iterative efforts to formulate a model, obtain data, and
evaluate the model to prioritize data needs. For example, a model may
require a large number of input assumptions. To prioritize efforts of
specifying distributions for variability and uncertainty for model
inputs, it is useful to determine which model inputs are most
influential with respect to the assessment end point. Therefore,
sensitivity can be used based on preliminary assessments of ranges or
distributions for each model input to determine which inputs are the
most important to the assessment. Refined efforts to characterize
distributions then can be prioritized to the most important inputs.

Appendix B: Glossary

Analysis. Examination of anything complex to understand its nature or to
determine its essential features (WHO IPCS Risk Assessment Terminology)

Assessment. The analysis and transformation of data into policy-relevant
information that can assist decision making and action.

Assessment end point. 1. Quantitative or qualitative expression of a
specific factor or metric with which a risk may be associated, as
determined through an appropriate risk assessment. 2. An explicit
expression of the environmental value that is to be protected,
operationally defined by an ecological entity and its attributes. For
example, salmon are valued ecological entities; reproduction and age
class structure are some of their important attributes. Together, salmon
“reproduction and age class structure” form an assessment end point.

Bayesian probability.  An approach to probability, representing a
personal degree of belief that something will occur..

Critical control point. A controllable variable that can be adjusted to
reduce exposure and risk. For example, a critical control point might be
the emission rate from a particular emission source. The concept of
critical control point is from the hazard assessment and critical
control point concept for risk management that is used in space and food
safety applications, among others.

Critical limit. A numerical value of a critical control point at or
below which risk is considered to be acceptable.

Ecological risk assessment. An ecological risk assessment evaluates the
potential adverse effects that human activities have on the plants and
animals that make up ecosystems. The risk assessment process provides a
way to develop, organize, and present scientific information, so that it
is relevant to environmental decisions. When conducted for a particular
place, such as a watershed, the ecological risk assessment process can
be used to identify vulnerable and valued resources, prioritize data
collection activity, and link human activities with their potential
effects.

Ecosystem. The interacting system of a biological community (plants and
animals) and its nonliving environment.

Expert Elicitation.  Expert elicitation (EE) is a systematic process of
formalizing and quantifying, typically in probabilistic terms, expert
judgments about uncertain quantities.

Environment. The sum of all external conditions affecting the life,
development, and survival of an organism.

Frequentist (or frequency) probability. A view of probability that
concerns itself with the frequency of events in a long series of trials,
or is based upon a data set.

Inputs. Quantities that are input to a model.

Model. 1. A set of constraints restricting the possible joint values of
several quantities. 2. A hypothesis or system of belief regarding how a
system works or responds to changes in its inputs. 3. A mathematical
function with parameters that can be adjusted so the function closely
describes a set of empirical data. A mechanistic model usually reflects
observed or hypothesized biological or physical mechanisms and has model
parameters with real-world interpretation. In contrast, statistical or
empirical models selected for particular numerical properties are best
fits to data; model parameters may or may not have real-world
interpretation. When data quality is otherwise equivalent, extrapolation
from mechanistic models (e.g., biologically based dose-response models)
often carries higher confidence than extrapolation using empirical
models (e.g., logistic models).

Modeling. 1. Development of a mathematical or physical representation of
a system or theory that accounts for all or some of its known
properties. Models often are used to test the effect of changes of
components on the overall performance of the system. 2. Use of
mathematical equations to simulate and predict real events and
processes. 3. Development or application of conceptual or graphical
methods to depict the structure and organization among major elements of
the system to be modeled.

Model uncertainty (sources of).

Model structure. Reflects competing sets of conceptual, scientific, or
technical assumptions available to develop a model for a particular
phenomenon. An example of model structure uncertainty is the use of
epidemiologically derived concentration-response functions for modeling
cancer risk in humans versus the use of toxicological (animal)-based
functions employing human-to-animal extrapolation factors.

Model detail. Reflects simplifying assumptions used to make modeling
tractable. For example, complex nonlinear behavior of chemical uptake
from the gastrointestinal system into the blood stream may be replaced
by a rather simplified linear model, especially for specific exposure
(intake) ranges.

Extrapolation. Use of models outside of the parameter space used in
their derivation may result in erroneous predictions. For example, a
threshold for health effects may exist at exposure levels below those
covered by a particular epidemiological study. If that study is used in
modeling health effects at those lower levels (and it is assumed that
the level of response seen in the study holds for lower levels of
exposure), then disease incidence may be overestimated.

Resolution. Selection of spatial or temporal resolution (i.e., grid
size) typically reflects a balance between a desired level of precision
and resources required to model the system. If the grid size selected is
too small, then key patterns of behavior reflected in the smaller step
size may be missed altogether, or the behavior of the system may be
misrepresented. For example, efforts to capture realistic high-end (near
upper-bound) risks to farmers around an incinerator may necessitate a
geographical-information-system-based modeling framework precise enough
to model exposures for individual farms. If a less resolved exposure
model is used, then risk to the most exposed farm may be underpredicted.

Model boundaries. Decisions regarding the time, space, number of
chemicals, etc., used in guiding modeling of the system. Risks can be
understated or overstated if the model boundary is misspecified. For
example, if a study area is defined to be too large and includes a
significant number of low-exposure areas, then a population-level risk
distribution can be diluted by including less exposed individuals, which
can, in turn, result in a risk-based decision that does not protect
sufficiently the most exposed individuals in the study area.

Parameter. 1. A variable, measurable property whose value is a
determinant of the characteristics of a system (e.g., Temperature,
pressure, and density are parameters of the atmosphere.). 2. A constant
or variable term in a function that determines the specific form of the
function but not its general nature, as “a” in f(x) = ax, where
“a” determines only the slope of the line described by f(x). 3. A
variable entering into the mathematical form of any probability
distribution model such that the possible values of the variable
correspond to different distributions.

Probability. 1. Frequentist approach/ The frequency with which samples
are obtained within a specified range or for a specified category (e.g.,
the probability that an average individual with a particular mean dose
will develop an illness). 2. Bayesian approach. The degree of belief
regarding the different possible values of a quantity.

Probabilistic risk analysis. Application of a computational method,
often based on a randomized sampling of available data or information,
to produce a probability distribution to more fully describe the data
than selecting a single point in the distribution, e.g., the mean.

Risk. 1. Risk includes consideration of exposure to the possibility of
an adverse outcome, the frequency with which one or more types of
adverse outcomes may occur, and the severity or consequences of the
adverse outcomes if such occur. 2. The potential for realization of
unwanted, adverse consequences to human life, health, property, or the
environment. 3. The probability of adverse effects resulting from
exposure to an environmental agent or mixture of agents. 4. The combined
answers to (1) What can go wrong? (2) How likely is it? and (3) What are
the consequences?

Risk analysis. 1. A process for identifying, characterizing,
controlling, and communicating risks in situations where an organism,
system, subpopulation, or population could be exposed to a hazard. Risk
analysis is a process that includes risk assessment, risk management,
and risk communication (WHO). 2. A detailed examination, including risk
assessment, risk evaluation, and risk management alternatives, performed
to understand the nature of unwanted, negative consequences to human
life, health, property, or the environment; an analytical process to
provide information regarding undesirable events; the process of
quantification of the probabilities and expected consequences for
identified risks.

Risk assessment. 1. A process intended to calculate or estimate the risk
to a given target organism, system, subpopulation, or population,
including the identification of attendant uncertainties following
exposure to a particular agent, taking into account the inherent
characteristics of the agent of concern, as well as the characteristics
of the specific target system (WHO). 2. The evaluation of scientific
information on the hazardous properties of environmental agents (hazard
characterization), the dose-response relationship (dose-response
assessment), and the extent of human exposure to those agents (exposure
assessment) (NRC, 1983). The product of the risk assessment is a
statement regarding the probability that populations or individuals so
exposed will be harmed and to what degree (risk characterization)
(USEPA, 2000). 3. Qualitative and quantitative evaluation of the risk
posed to human health or the environment by the actual or potential
presence or use of specific pollutants.

Risk-informed decision making. An approach to decision making in which
insights from probabilistic risk analyses are considered with other
insights and factors.

Risk management. A decision making process that takes into account
environmental laws, regulations, political, social, economic,
engineering, and scientific information, including a risk assessment, to
weigh policy alternatives associated with a hazard.

Scenario. 1. An outline or model of an expected or supposed sequence of
events. 2. A set of facts, assumptions, and inferences about how
exposure takes place and regarding how exposures translate into adverse
effects that aides the analyst in evaluating, estimating, or quantifying
exposures and risks. Scenarios might include identification of
pollutants, pathways, exposure routes, and modes of action, among
others.

Sensitivity analysis. A study of how the variation in data inputs
(including inputs to models) affect the outputs of a model or choice
among potential decision options.

Levels. Refers to various hierarchical levels of complexity and
refinement for different types of modeling approaches that can be used
in risk assessment. A deterministic risk assessment with conservative
assumptions is an example of a lower level type of analysis that can be
used to determine whether exposures and risks are below levels of
concern. Examples of progressively higher levels include the use of
deterministic risk assessment coupled with sensitivity analysis, the use
of probabilistic techniques to characterize either variability or
uncertainty only, and the use of two-dimensional probabilistic
techniques to distinguish between but simultaneously characterize both
variability and uncertainty.

Two-dimensional probabilistic analysis. A modeling approach in which
inter-individual variability in exposure and risk is characterized using
frequency distributions, and in which uncertainty in the estimates of
statistics of the frequency distributions (e.g., the mean, median,
standard deviation, percentiles) are characterized using probability
distributions.

Uncertainty. Occurs because of a lack of knowledge. It is not the same
as variability. For example, a risk assessor may be very certain that
different people drink different amounts of water but may be uncertain
about how much variability there is in water intakes within the
population. Uncertainty often can be reduced by collecting more and
better data, whereas variability is an inherent property of the
population being evaluated. Variability can be better characterized with
more data but it cannot be reduced or eliminated. Efforts to clearly
distinguish between variability and uncertainty are important for both
risk assessment and risk characterization, although they both may be
incorporated into an assessment.[I agree with Harvey’s comment about
this sentence.]

Uncertainty analysis. A detailed examination of the systematic and
random errors of a measurement or estimate; an analytical process to
provide information regarding uncertainty.

Value of information. A quantitative measure of the value of knowing the
outcome of an uncertain variable prior to making a decision. Decision
theory provides a means for calculating the value of both perfect and
imperfect information. The former value, informally known as the value
of clairvoyance, is an upper bound for the latter. Obtaining meaningful
value-of-information measurements requires an awareness of important
restrictions (concerning the nature of free will) on the validity of
this kind of information.

Variability. Refers to true heterogeneity or diversity, as exemplified
in natural variation . For example, among a population that drinks water
from the same source and with the same contaminant concentration, the
risks from consuming the water may vary. This may result from
differences in exposure (i.e., different people drinking different
amounts of water and having different body weights, different exposure
frequencies, and different exposure durations), as well as differences
in response (e.g., genetic differences in resistance to a chemical
dose). Those inherent differences are referred to as variability.
Differences among individuals in a population are referred to as
inter-individual variability, differences for one individual over time
is referred to as intra-individual variability.Appendix C: References

Ang, A. H-S., and W.H. Tang. 1984. Probability Concepts in Engineering
Planning and Design, Volume 2: Decision, Risk, and Reliability. John
Wiley & Sons, New York.

Bloom, D.L., D.M. Byrne, and J.M. Andreson. 1993. Communicating Risk to
Senior EPA Policy-Makers: A Focus Group Study. Prepared by Bloom
Research and the Office of Air Quality Planning and Standards. U.S.
Environmental Protection Agency, Research Triangle Park, NC.

Burnham, K.P., and D.R. Anderson. 2002. Model Selection and Inference: A
Practical Information-Theoretic Approach (2nd ed.). Springer-Verlag, New
York.

Cullen, A.C., and H.C. Frey. 1999. Probabilistic Exposure Assessment: A
Handbook for Dealing with Variability and Uncertainty in Models and
Inputs. Plenum Press, New York.

Clyde, M. et al.  2000.  Effects of Ambient Fine and Coarse Particles on
Mortality in Phoenix, Arizona.  Duke University, Durham, N.C.

ON MORTALITY IN PHOENIX, ARIZONA1

EPA, 1995.  1995.  Policy for risk characterization.  Science Policy
Council, Washington, DC.  (http://www.epa.gov/osp/spc/rcpolicy.htm).

EPA. 1997a. Guiding Principles for Monte Carlo Analysis.
EPA/630/R-97/001, Risk Assessment Forum. U.S. Environmental Protection
Agency, Washington.

EPA. 1998. Guidance for Submission of Probabilistic Human Health
Exposure Assessments to the Office of Pesticide Programs. Office of
Pesticide Programs. U.S. Environmental Protection Agency, Draft,
11/4/98,   HYPERLINK
"http://www.epa.gov/fedrgstr/EPA-PEST/1998/November/Day-05/6021.pdf"
http://www.epa.gov/fedrgstr/EPA-PEST/1998/November/Day-05/6021.pdf
(accessed August 8, 2007).

EPA. 2000. EPA Science Policy Council Handbook, Risk Characterization.
EPA 100-B-00-002.

EPA. 2001. Risk Assessment Guidance for Superfund: Volume III - Part A,
Process for Conducting Probabilistic Risk Assessment. EPA 540-R-02-002,
Office of Emergency and Remedial Response, U.S. Environmental Protection
Agency, Washington,   HYPERLINK
"http://www.epa.gov/oswer/riskassessment/rags3adt/"
http://www.epa.gov/oswer/riskassessment/rags3adt/  (accessed August 8,
2007).

EPA. 2007. Letter from M.G. Morgan and R.T. Parkin (SAB) to S. Johnson
(EPA), February 28, 2007. EPA-SAB-07-003.

EPA. 2008. Case Study Examples of the Application of Probabilistic Risk
Analysis in U.S. Environmental Protection Agency Regulatory
Decision-Making (in review). Risk Assessment Forum, U.S. Environmental
Protection Agency, Washington.

Evans, J.S., J.D. Graham, G.M. Gray, and R.L. Sielken. 1994. A
Distributional Approach to Characterizing Low-Dose Cancer Risk. Risk
Analysis, 14(1):25-34.

Fisher, L., P. Pascual, and W. Wagner. 2006. Mapping the Role of Models
in US and EU Risk Regulation: A Legal Background Paper.

Frey, H.C. 2004. Presentation at the EPA Colloquium on Probabilistic
Risk Assessment, April 2004 (unpublished).

Frey, H.C., and S.R. Patil. 2002. Identification and Review of
Sensitivity Analysis Methods. Risk Analysis, 22(3):553-578.

Hattis, D.B., and D.E. Burmaster. 1994. “Assessment of Variability and
Uncertainty Distributions for Practical Risk Analyses,” Risk Analysis,
14(5):713-730.

Hoeting, J.A.; D. Madigan, A.E. Raftery, and C.T. Volinsky. 1999.
Bayesian Model Averaging: A Tutorial. Statistical Science,
14(4):382-417.

IEC. 2006. Expanded Expert Judgment Assessment of the
Concentration-Response Relationship Between PM2.5 Exposure and
Mortality, Final Report. Prepared by Industrial Economics, Incorporated
for Office of Air Quality Planning and Standards, U.S. Environmental
Protection Agency, Research Triangle Park, NC,   HYPERLINK
"http://www.epa.gov/ttn/ecas/regdata/Uncertainty/pm_ee_report.pdf"
http://www.epa.gov/ttn/ecas/regdata/Uncertainty/pm_ee_report.pdf .

Krupnick, A., R. Morgenstern, M. Batz, P. Nelson, D. Burtraw, J. Shih,
and M. McWilliams. 2006. Not a Sure Thing: Making Regulatory Choices
Under Uncertainty. Resources for the Future, Washington.

Mokhtari, A., H.C. Frey, and J. Zheng. 2006. Evaluation and
Recommendation of Sensitivity Analysis Methods for Application to
Stochastic Human Exposure and Dose Simulation (SHEDS) Models. Journal of
Exposure Science and Environmental Epidemiology, 16(6):491-506.

Morgan, M.G., and M. Henrion. 1990. Uncertainty: A Guide to Dealing with
Uncertainty in Quantitative Risk and Policy Analysis. Cambridge
University Press, New York.

NRC. 1983. Risk Assessment in the Federal Government: Managing the
Process. Committee on the Institutional Means for Assessment of Risks to
Public Health, National Academy Press, Washington.

NRC. 1989. Risk Assessment in the Federal Government: Managing the
Process. National Research Council, National Academy Press, Washington.

NRC. 1991. Rethinking the Ozone Problem in Urban and Regional Air
Pollution. National Research Council, National Academy Press,
Washington.

NRC. 1993. Issues in Risk Assessment. National Research Council,
National Academy Press, Washington.

NRC. 1994. Science and Judgment in Risk Assessment. National Research
Council, National Academy Press, Washington.

NRC. 2002. Estimating the Public Health Benefits of Proposed Air
Pollution Regulations. National Research Council, National Academy
Press, Washington.

NRC. 2007a. Scientific Review of the Proposed Risk Assessment Bulletin
from the Office of Management and Budget. National Research Council,
National Academy Press, Washington.

NRC. 2007b. Models in Environmental Regulatory Decision Making. National
Research Council, National Academy Press, Washington.

OSTP/OMB. 2007. Memorandum (M-07-24) for the Heads of Executive
Departments and Agencies; from Susan E. Dudley, Administrator, Office of
Information and Regulatory Affairs, Office of Management and Budget, to
Sharon L. Hays, Associate Director and Deputy Director for Science,
Office of Science and Technology Policy: Updated Principles for Risk
Analysis, September 19, 2007.

Parkin, R.T., and M.G. Morgan. 2007. Consultation on Enhancing Risk
Assessment Practices and Updating EPA’s Exposure Guidelines.
EPA-SAB-07-003, Science Advisory Board, U.S. Environmental Protection
Agency, Washington.

PCRARM. 1997. Framework for Environmental Health Risk Management Final
Report, Volume 1. Presidential/Congressional Commission on Risk
Assessment and Risk Management,   HYPERLINK
"http://www.riskworld.com/nreports/1997/risk-rpt/pdf/EPAJAN.PDF"
http://www.riskworld.com/nreports/1997/risk-rpt/pdf/EPAJAN.PDF .

Pearl, J. 2000. Causality: models, Reasoning, and Inference. Cambridge
University Press, New York.

Royall, R.M. 1997. Statistical Evidence: A Likelihood Paradigm. Chapman
& Hall, London.

Saltelli A, S. Tarantola, F. Campolongo, et al. 2004. Sensitivity
Analysis in Practice. John Wiley & Sons, New York.

Stahl, C.H., and A.J. Cimorelli. 2005. How Much Uncertainty Is Too Much
and How Do We Know? A Case Example of the Assessment of Ozone Monitor
Network Options.   HYPERLINK "javascript:void(0);"  Risk Analysis, Vol.
25, No. 5, pp. 1109-1120, October 2005 .

WHO. 2008. Guidance on Characterizing and Communicating Uncertainty in
Exposure Assessment (draft). The International Programme on Chemical
Safety, World Health Organisation, Geneva.

Bibliography

Probabilistic Risk Analysis Methodology(General

Cullen, A.C., and H.C. Frey. 1999. The Use of Probabilistic Techniques
in Exposure Assessment: A Handbook for Dealing with Variability and
Uncertainty in Models and Inputs. Plenum, New York.

EPA. 1997. Guiding Principles for Monte Carlo Analysis.
EPA/630/R-97/001, U.S. Environmental Protection Agency, Washington,
March 1997.

Frey, H.C., D. Crawford-Brown, J. Zheng, and D. Loughlin. 2003.
Hierarchy of Methods to Characterize Uncertainty: State of Science of
Methods for Describing and Quantifying Uncertainty, Draft. Prepared for
E.H. Pechan and Associates, Inc., Springfield, VA, for submittal to U.S.
Environmental Protection Agency, Research Triangle Park, NC, August 12,
2003.

Frey, C., J. Penman, L. Hanle, S. Monni, and S. Ogle. 2006.
Uncertainties, Chapter 3 in Volume 1, General Guidance and Reporting,
2006 IPCC Guidelines for National Greenhouse Gas Inventories. National
Greenhouse Gas Inventories Programme, Inter-Governmental Panel on
Climate Change, Technical Support Unit, Kanagawa, Japan.

Loughlin, D., H.C. Frey, K. Hanisak, and A. Eyth. 2003. Implementation
Requirements for the Development of a Sensitivity/Uncertainty Analysis
Tool for MIMS, Draft. Prepared by Carolina Environmental Program and
N.C. State University for U.S. Environmental Protection Agency, Research
Triangle Park, NC, May 6, 2003.

Morgan, M.G., and M. Henrion. 1990. Uncertainty: A Guide to Dealing with
Uncertainty in Quantitative Risk and Policy Analysis. Cambridge
University Press, New York.

NARSTO. 2005. Improving Emission Inventories for Effective Air Quality
Management Across North America, A NARSTO Assessment. NARSTO-05-001.

NRC. 1994. Science and Judgment in Risk Assessment. National Research
Council, National Academy Press, Washington.

Probabilistic Risk Analysis and Decision Making

Bloom, D.L., D.M. Byrne, and J.M. Andreson. 1993. Communicating Risk to
Senior EPA Policy-Makers: A Focus Group Study. Prepared by Bloom
Research and the Office of Air Quality Planning and Standards, U.S.
Environmental Protection Agency, Research Triangle Park, NC.

EPA. 1997.   HYPERLINK "http://www.epa.gov/OSA/spc/pdfs/probcovr.pdf_"
Memo  from Fred Hansen entitled Use of Probabilistic Techniques
(including Monte Carlo Analysis) in Risk Assessment, dated May 15, 1997.
U.S. Environmental Protection Agency, Washington.

Krupnick et al. 2006. Not a Sure Thing: Making Regulatory Choices Under
Uncertainty. Resources for the Future, Washington.

McGarity, T.O., and W.E. Wagner. 2003. Legal Aspects of the Regulatory
Use of Environmental Modeling. Environmental Law Reporter, Volume 33,
10751-10774.

NRC. 1996. Understanding Risk: Informing Decisions in a Democratic
Society. National Research Council, National Academy Press, Washington.

Pascual, P., N. Steiber, and E. Sunderland. 2003. Draft Guidance on the
Development, Evaluation, and Application of Regulatory Environmental
Models. Prepared by the Council for Regulatory Environmental Modeling,
Office of Science Policy, U.S. Environmental Protection Agency,
Washington, November.

Thompson, K.M., and J.D. Graham. 1996. Going Beyond the Single Number:
Using Probabilistic Risk Assessment to Improve Risk Management. Human
and Ecological Risk Assessment, 2(4):1008-1034.

Probabilistic Risk Analysis Methodology(Specific Aspects

Abdel-Aziz, A., and H.C. Frey. 2003b. Development of Hourly
Probabilistic Utility NOx Emission Inventories Using Time Series
Techniques: Part I-Univariate Approach. Atmospheric Environment,
37:5379-5389.

Abdel-Aziz, A., and H.C. Frey. 2003c. Development of Hourly
Probabilistic Utility NOx Emission Inventories Using Time Series
Techniques: Part II-Multivariate Approach. Atmospheric Environment,
37:5391-5401.

Abdel-Aziz, A., and H.C. Frey. 2004. Propagation of Uncertainty in
Hourly Utility NOx Emissions Through a Photochemical Grid Air Quality
Model: A Case Study for the Charlotte, NC Modeling Domain. Environmental
Science and Technology, 38(7):2153-2160.

Casman, E.A., M.G. Morgan, and H. Dowlatabadi. 1999. Mixed Levels of
Uncertainty in Complex Policy Models. Risk Analysis, 19(1):33-42.

Crawford-Brown and Brown. 1987. A Framework for Assessing the
Rationality of Judgments in Carcinogenicity Hazard Identification. Risk:
Health, Safety and Environment, Fall, 307-337.

Draper, D. 1995. Assessment and Propagation of Model Uncertainty.
Journal of Royal Statistical Society 57:45-97.

Evans, J.S., J.D. Graham, G.M. Gray, and R.L. Sielken. 1994. A
Distributional Approach to Characterizing Low-Dose Cancer Risk. Risk
Analysis, 14(1):25-34.

Frey, H.C., and D.E. Burmaster. 1999. Methods for Characterizing
Variability and Uncertainty: Comparison of Bootstrap Simulation and
Likelihood-Based Approaches. Risk Analysis, 19(1):109-130 (February).

Frey, H.C., and D.S. Rhodes. 1996. Characterizing, Simulating, and
Analyzing Variability and Uncertainty: An Illustration of Methods Using
an Air Toxics Emissions Example. Human and Ecological Risk Assessment:
an International Journal, 2(4):762-797 (December).

Frey, H.C., and Y. Zhao. 2004. Quantification of Variability and
Uncertainty for Air Toxic Emission Inventories With Censored Emission
Factor Data. Environmental Science and Technology, 38(22):6094-6100.

Zheng, J., and H.C. Frey. 2004. Quantification of Variability and
Uncertainty Using Mixture Distributions: Evaluation of Sample Size,
Mixing Weights and Separation between Components. Risk Analysis,
24(3):553-571 (June).

Zheng, J., and H.C. Frey. 2005. Quantitative Analysis of Variability and
Uncertainty with Known Measurement Error: Methodology and Case Study.
Risk Analysis, 25(3):663-676.

Sensitivity Analysis

Agro, K.E., C.A. Bradley, and N. Mittmann. 1997. Sensitivity Analysis in
Health Economic and Pharmacoeconomic Studies - an Appraisal of the
Literature. Pharmacoeconomics, 11(1):75-88.

Cukier, R.I., H.B. Levine, and K.E. Shuler. 1978. Nonlinear Sensitivity
Analysis of Multi-parameter Model Systems. Journal of Chemical Physics,
26(1):1-42.

Frey, H.C., and R. Patil. 2002. Identification and Review of Sensitivity
Analysis Methods. Risk Analysis, 22(3):553-577.

Frey, H.C., A. Mokhtari, and J. Zheng. 2004. Recommended Practice
Regarding Selection, Application, and Interpretation of Sensitivity
Analysis Methods Applied to Food Safety Process Risk Models. Prepared by
N.C. State University for U.S. Department of Agriculture, Washington,
January 30, 2004.  HYPERLINK "http://www.ce.ncsu.edu/risk"   Available
at   HYPERLINK "http://www.ncsu.edu/risk"  http://www.ncsu.edu/risk  .

Helton, J.C., and F.J. Davis. 2002. Illustration of Sampling-Based
Methods for Sensitivity Analysis. Reliability Engineering and System
Safety, 81(1):23-69.

Mokhtari, A., and H.C. Frey. 2005a. Recommended Practice Regarding
Selection of Sensitivity Analysis Methods Applied to Microbial Food
Safety Process Risk Models. Human and Ecological Risk Assessment,
11(3):591-605.

Mokhtari, A., H.C. Frey, J. Zheng. 2006. Evaluation and Recommendation
of Sensitivity Analysis Methods for Application to EPA-Stochastic Human
Exposure and Dose Simulation (SHEDS) Models. Journal of Exposure Science
and Environmental Epidemiology, 16(6):491-506.

Saltelli, A., K. Chan, and M. Scott. 2000. Sensitivity Analysis,
Probability and Statistics Series. John Wiley & Sons, New York.

Sobol, I.M. 1993. Sensitivity Estimates for Nonlinear Mathematical
Models. Mathematical Modeling and Computation, 1(4):407-414.

Xue, J., V.G. Zartarian, H. Özkaynak, W. Dang, G. Glen, L. Smith, and
C. Stallings. 2006. A Probabilistic Arsenic Exposure Assessment for
Children Who Contact Chromated Copper Arsenate (CCA)-Treated Playsets
and Decks, Part 2: Sensitivity and Uncertainty Analyses. Risk Analysis,
26(2):533-541.

Case Study Examples of Probabilistic Risk Analysis(EPA (see also the PRA
Case Studies White Paper)

Babendreier, J.E., and K.L. Castleton. 2005. Investigating Uncertainty
and Sensitivity in Integrated Multimedia Environmental Models: Tools for
FRAMES-3MRA. Environmental Modeling and Software, 20:1043-1055.

Blancato, J.N., F.W. Power, R.N. Brown, and C.C. Dary. 2004. Exposure
Related Dose Estimating Model (ERDEM) for Assessing Human Exposure and
Dose, EPA/600/R-04/060, U.S. Environmental Protection Agency, Research
Triangle Park, NC, December 2004.

Burke, J. 2005. EPA Stochastic Human Exposure and Dose Simulation for
Particulate Matter (SHEDS-PM), User’s Guide. U.S. Environmental
Protection Agency, Research Triangle Park, NC.

Frey, H.C. 2003. Evaluation of an Approximate Analytical Procedure for
Calculating Uncertainty in the Greenhouse Gas Version of the Multi-Scale
Motor Vehicle and Equipment Emissions System. Prepared for Office of
Transportation and Air Quality, U.S. Environmental Protection Agency,
Ann Arbor, MI, May 30, 2003.

Frey, H.C., and Y. Zhao. 2003. Development of Probabilistic Emission
Inventories of Benzene, Formaldehyde and Chromium for the Houston
Domain. Prepared by N.C. State University for U.S. Environmental
Protection Agency, Research Triangle Park, NC, September 2003.

Frey, H.C., R. Bharvirkar, and J. Zheng. 1999. Quantitative Analysis of
Variability and Uncertainty in Emissions Estimation. Prepared by N.C.
State University for U.S. Environmental Protection Agency, Research
Triangle Park, NC, July 1999.

Frey, H.C., J. Zheng, Y. Zhao, S. Li, and Y. Zhu. 2002. Technical
Documentation of the AuvTool Software for Analysis of Variability and
Uncertainty. Prepared by N.C. State University for Office of Research
and Development, U.S. Environmental Protection Agency, Research Triangle
Park, NC, February 2002.

Zartarian, V.G., J. Xue, H.A. Ozkaynak, W. Dang, G. Glen, L. Smith, and
C. Stallings. 2005. A Probabilistic Exposure Assessment for Children Who
Contact CCA-Treated Playsets and Decks Using the Stochastic Human
Exposure and Dose Simulation Model for the Wood Preservative Scenario
(SHEDS-WOOD), Final Report. U.S. Environmental Protection Agency,
Washington, EPA/600/X-05/009. Available at   HYPERLINK
"http://www.epa.gov/heasd/sheds/cca_treated.htm"
http://www.epa.gov/heasd/sheds/cca_treated.htm .

Zartarian, V.G., J. Xue, H. Özkaynak, W. Dang, G. Glen, L. Smith, and
C. Stallings. 2006. A Probabilistic Arsenic Exposure Assessment for
Children Who Contact Chromated Copper Arsenate (CCA)-Treated Playsets
and Decks, Part 1: Model Methodology, Variability Results, and Model
Evaluation. Risk Analysis, 26(2):515-532.

Zheng, J., and H.C. Frey. 2002. AuvTool User’s Guide. Prepared by N.C.
State University for Office of Research and Development, U.S.
Environmental Protection Agency, Research Triangle Park, NC, February
2002.

Case Study Examples of Probabilistic Risk Analysis(Other

Bloyd, Cary, et al. 1996. Tracking and Analysis Framework (TAF) Model
Documentation and User’s Guide, 1996. Argonne National Laboratory,
ANL/DIS/TM-36 (December).

Frey, H.C., and S. Li. 2003. Quantification of Variability and
Uncertainty in AP-42 Emission Factors: Case Studies for Natural
Gas-Fueled Engines. Journal of the Air & Waste Management Association,
53(12):1436-1447 (December).

Hanna, S.R., Z. Lu, H.C. Frey, N. Wheeler, J. Vukovich, S. Arunachalam,
M. Fernau, and D.A. Hansen. 2001. Uncertainties in Predicted Ozone
Concentrations due to Input Uncertainties for the UAM-V Photochemical
Grid Model Applied to the July 1995 OTAG Domain. Atmospheric
Environment, 35(5):891-903.

Munns, W.R., Jr., H.A. Walker, and J.F. Paul. 1989. An Ecological Risk
Assessment Framework for Examining the Impacts of Oceanic Disposal
Status and Trends Program. In: Oceans ‘89, Volume 2: Ocean Pollution,
Marine Technology Society and Oceanic Society of the Institute of
Electrical and Electronics Engineers, IEEE Publication No. 89CH2780-5,
pp. 664-669.

Munns, W.R., Jr., H.A. Walker, J.F. Paul, and J.H. Gentile. 1996. A
Prospective Assessment of Ecological Risks to Upper Water Column
Populations from Ocean Disposal at the 106-Mile Dumpsite. Journal of
Marine Environmental Engineering, 3:279-297.

Nocito, J.A., H.A. Walker, J.F. Paul, and C. Menzie. 1988. Application
of a Risk Assessment for Marine Disposal of Sewage Sludge at Midshelf
and Offshelf Sites. In: Proceedings of the Seventh International Ocean
Disposal Symposium, Environment Canada, pp. 644-663.