New Course Offering – Working with Risk and Uncertainty

Improving Uncertainty Assessments that Underpin Our Analyses

More and more companies in the oil and gas industry are making use of probabilistic models and risk analysis. Models are becoming more powerful and in combination with new forecasting techniques and computer simulations can now perform probabilistic analyses with ease. One area that has not received much attention, though, is one of the most important – quantifying the uncertainty that underlies these analyses. Whether estimating costs, schedule, commercial or subsurface uncertainty, too often, the range and chance factor assessments that are needed are rushed in the interest of time, even though we all know about “garbage in – garbage out”. When rushed, the assessments often overlook the grounding of the experts in the terminology of uncertainty, and in the fundamentals of quantitatively expressing our strength of belief about an uncertainty.

Decision Frameworks has developed a new training course focused specifically on providing a solid foundation for those who work with uncertainty and risk in any capacity. Designed to be suitable for team members, analysts, and practitioners alike, the course can be tailored to fit the particular needs of each audience. The Working with Risk and Uncertainty course will be offered as an open enrolment course, as well as be introduced as new modules in some of our leading internal course offerings.

Participants will learn the fundamental terminology of probability and statistics and the importance of quantifying and calibrating our judgements about the future. Through simple yet effective exercises, they will learn to make calibrated range and confidence estimates that are consistent and consistently correct. Attendees will also learn how to avoid some of the more common mistakes that are made when combining uncertain variables. Simple models will be built to illustrate the concepts and to provide practice working with uncertainty and risk.

The course will be offered in a variety of modules to address different needs. There will be a half day version for Subject Matter Experts, a full day version for those who use ranges and chance factors in simple models, and two and three day workshops for those who need to understand interactions and dependencies among variables in more complex models. With the shorter workshops, participants will be able to provide quantitative, accurate descriptions of their view of unknown quantities or the likelihood of future events. In the longer versions, participants will explore assessments of variable interactions and their applications in more complex problems.

Who should attend?

• Uncertainty estimators
• Engineers
• Geologists
• Geophysicists
• Economists
• Managers

Monte Carlo vs. Decision Trees – Round One

By Jan Schulze, Vice President, Software Development

I’d like to begin this article by stating that although this topic is a discussion centered on methodology, it can become a very emotional subject. Disciples on either side of the debate can get very charged up on the merits of one approach over the other.

So, I’d like to emphasize that I was raised to believe in the virtues of Monte Carlo analysis techniques for all things probabilistic. I saw great utility with this approach and even wrote my own Monte Carlo simulator. I came late to Decision Tree analysis, and now have a much deeper appreciation for its merits, believing that each approach has its place.Over the next few newsletters, I intend to point out the strengths and weakness of both approaches, beginning, in this issue, with a quantitative comparison.

A Two Dimensional Plot of the Solution Space
To begin, let’s use the two-dimensional plot shown in Figure A to represent the full set of potential outcomes for a given opportunity, one outcome of which is displayed on the plot. Instead of relying on one potential outcome, one of our tasks, as analysts, is to describe the full risk profile of the opportunity. Monte Carlo and Decision Tree techniques typically do this in two different ways. Let’s explore.


Figure A - Two Dimensional Problem Space

A Monte Carlo Approach
First, let’s look at Monte Carlo. A Monte Carlo model will generate new potential outcomes in the solution space at random and, if we have enough time and computation power, and if we generate a large number of trials, we will uncover many possible results and explore the extreme corners of the plot. This can be very useful if we are considering “black swan” events (low probability – high impact occurrences). With a valid, representative model, this can provide powerful insights and identify areas of potential concern.


Figure B - Monte Carlo Model

A Decision Tree Approach
By comparison, a Decision Tree model generates solutions in a controlled way. Instead of defining complete distributions, uncertainty ranges are typically represented as three branches on a tree, the P10, P50, and P90 outcomes. When we combine uncertainties together, we might create a plot as seen in Figure C. The tree combinations cover the decision space more symmetrically.


Figure C – Decision Tree Model

Comparing the Two
Now, let us delve into the problem space a little more. If we run the same number of Monte Carlo trials as Decision Tree nodes, we might end up with a plot that would look something like Figure D, where the Monte Carlo analysis is unlikely to explore the space as thoroughly.

If we assume that the opportunity is characterized with five key uncertainties, each with three possible outcomes, we end up with 243 combinations (3*3*3*3*3 = 243). Assuming we use a 30% / 40% / 30% distribution for the probability of each outcome, the chance of all three uncertainties being low at the same time becomes quite small (30% * 30% * 30% * 30% *30% = 0.243%). The Decision Tree dots at the edges of the plot represent these extreme outcomes. Are they close enough to the axes?

The answer depends on the questions you are trying to answer about the opportunity. If you’re primary concern is with the expected reserves distribution, this mapping is probably okay; but if you’re concerned about the remote chance of a well blow-out, this decision tree mapping may not map the extremes well enough.


Figure D – Monte Carlo vs. Decision Tree
If we now look at the probability of any of these extreme events occurring, then we are saying that there is a roughly 1 in 400 chance of seeing one of these results (30% + 30% + 30% + 30% +30% = 0.243%). In other words, we will need to run a minimum of 400 perfectly placed Monte Carlo trials to be able to achieve a coverage that matches that of a Decision Tree. But Monte Carlo analysis will select points randomly; the coverage is unlikely to be “perfect”. To compensate, we need to increase the number of trials, so that we don’t end up with skewed results.

For instance, if we use the analogy that Monte Carlo analysis is like the game “pin the tail on the donkey”, we could end up with a skewed result at the end of the game where we’ve pinned the tail on the right front hoof more times than not, creating not only a painful donkey but an inaccurate view of where the tail could actually be.

The Issue of Sampling and Iterations
How much sampling is required in a Monte Carlo run to ensure so that we get the same coverage of the result space as a Decision Tree? I’ve worked the problem manually, by repeating a Monte Carlo analysis and each time increasing the number of trials, until the results matched that of the Decision Tree (that is, until the distribution of results matched AND until the Monte Carlo analysis produced points that matched or exceeded the points at the extremes as the Decision Tree).

I was only able to match results after I increased the number of Monte Carlo trials to one hundred times more than that of the Decision Tree. For Monte Carlo analysis to match the results of the five-uncertainty Decision Tree example (with 243 combinations, or outcomes), we would need to run 400 x 100 = 40,000 trials.

But how many times would we run 40,000 Monte Carlo trials on five-uncertainty evaluation? This might not even be viable computationally with a complex model. 500 trials, yes. 1000, yes. 5000, maybe. But 40,000? Are we routinely running enough trials to truly understand the results space or are we falsely chasing precision and erroneously feeling that we are getting a better result simply because we have expended more effort to achieve it?

When you compare the two methods now, we see that Decision Tree analysis provides broad coverage of the problem space with a near symmetrical distribution of points, while Monte Carlo analysis can give us the same coverage if we are prepared to run it enough times. Granted at the end of that number of trials, the granularity we will have is much greater than we get with a Decision Tree but, if we can arrive at the same conclusions with 94% less computational effort, why wouldn’t we take it?

What if we are only interested in finding the expected value (or risk weighted value) of the results space? How many Monte Carlo runs does it take to match the results of the Decision Tree? Much fewer. In fact, with a large number of uncertainties, it’s likely that the Monte Carlo evaluation will converge on the expected value with fewer trials than that of a Decision Tree because the Monte Carlo points will tend to cluster around the average. But, remember, our mission as analysts is to map out the full risk profile of the potential results. For this, you’ll need to run a much larger number of iterations.

The Right Tool for the Problem You are Trying to Solve
It is for these reasons that I believe decision trees lend so much value to the typical decision problems we see in the petroleum, pharmaceutical and other sectors we support and why I undertook the challenge of building a new Decision Tree tool.

As ever, which tool is the right tool depends on the problem you are trying to solve. In my view, one of the major reasons Monte Carlo analysis tools have tended to be most popular is because they have been historically easier to use.

Right click on an input in your spreadsheet model, enter a couple numbers and presto, it’s done. Entering distributions on ten uncertainties is easy. By comparison, walking through that same ten-level decision tree with 59,049 end node branches might become painful (if your software tool can even do it). Yet, for all the reasons, I’ve stated above, Decision Trees can be very valuable, insightful, efficient and (depending upon the decision problem) better than Monte Carlo analysis.

As a software developer, I took this as a challenge. Bridge the gap between the ease of use of a Monte Carlo tool and the unique attributes of Decision Tree analysis, one might have a brand new powerful analytic tool. TreeTop™ is my attempt to address this challenge; I will be very interested to hear your thoughts on my efforts.

I appreciate your comments on this article and look forward to more dialogue in this area.