Thursday, December 16, 2010

INFORMS Annual Meeting - Gumball 2010 Blog

From Sunday, November 7 through Wednesday, November 10, 2010, we participated in the 2010 INFORMS Annual Meeting at the Conference Center in Austin, Texas We had a booth, ran a pre-meeting seminar and showcased our latest decision framing (DTrio) and decision tree software (TreeTop) at the booth and in a Tuesday morning demonstration.

During the conference, we ran another in a series of tests based around the “Wisdom of Crowds”, James Surowiecki’s best-selling book. In it he contends that we put too much emphasis on expert opinion and if you solicit a group of individuals in an independent manner and collect data from their responses, you may be very pleasantly surprised by the results.

This year’s experiment saw an oddly shaped glass container standing at the corner of our booth filled with red, blue, green and yellow gumballs, as along with an unknown number of rubber squishy owls. Visitors were asked to guess the number of gumballs in the container. They were first asked for their P10 and P90 estimates. In other words, they should be 80% confident that the actual number of gumballs should fit between their range. That should be pretty easy to do, right?

At the bottom of the ballot, they were asked for their best guess on the exact number of gumballs in the container. In an effort to reward the best estimator in this experiment, we offered an iPod to the individual with the closest guess.

So without another wasted breath, here are the results:

Number of estimates
78
Range of estimates
45-11075
Average of all estimates
1914
Actual number of gumballs
1371
P50 estimate of gumballs (calculated)
1350
Closest Best Estimate
1400
Winner
Justin Mao-Jones



Figure A: Distribution of Gumball Estimates
(Shown in Decision Frameworks new decision tree tool – TreeTop™)

Let’s take a look at the results. Firstly, two people guessed that there were 1400 gumballs in the container. Justin was judged to be the winner because he handed his ballot in the earliest (We knew that small print on the back side of the ballot on how the winner would be determined would come in handy one day ;>0 ). Upon hearing the news, he mentioned donating it to a worthy cause - which he named as himself. To the winner go the spoils! Good job.

Secondly, let’s step back and take a look at what the data is telling us. We see we had a fairly wide range of estimates 45 – 11,075. What is noteworthy is that the EV (expected Value) was skewed so by the “fat tails” on the high side. There was an equal number of estimates above and below the closest answer, so those who thought they saw more gumballs ready thought they saw lots more gumballs, thereby skewing the average to the upside. Does anyone have any evidence that shows when people are confronted with a large number of objects that they tend to disproportionally overestimate than underestimate the correct number?

As stated above, when you order the estimates from lowest to highest, Justin’s estimate was interestingly at the exact mid-point. His was the 39th out of 78 sequentially ordered estimates. His was the mid-point estimate and it was also closest. Does this support Surowiecki’s assertions about the wisdom of crowds? Though this is not the statistical average of all estimates, it is smack-dab in the middle of everyone’s wisdom. A gumball for your thoughts on this little tidbit of information…

Here’s another interesting finding from looking at the data. If you remember, you were asked when you submitted your estimate to write down a bounded range with your P10 and P90 limits. When we look at the results, only 34.1% of you had a range that was wide enough to contain the actual number. The means that roughly 2 out of 3 people were not able to create a reasonable estimate, an 80% confidence range around this uncertainty. This confirms other experiences we see in our daily consulting and training courses. People seem to have an inherent blind spot to creating broad enough ranges. Perhaps because people are often rewarded for the exactness/precision of their answers, there is a perceived weakness if too broad a range is given as an answer. Alternatively, maybe gumballs are just too tricky to count.

Regardless, what it shows is that proper care needs to be taken when it comes around to gathering range estimates. Do your best to de-bias and create more realistic extremes. There are several techniques we use to do this including structured interviews with expert interview templates. We can teach you if you are unfamiliar with these tools and techniques.

There is much more that bigger brains than mine might glean from this data. If you would like a copy of the raw excel file, shoot me a note and I’ll send you a copy.

If this has sparked your interest or you would like to know more about the training and tools Decision Frameworks provides those who are focused on improving the quality of decision making within their organizations, please click here.
.
Thanks for everyone who participated in this contest. We hope to see many of you before the next INFORMS Conference. And remember great decision making is not divined - it is derived!

Merry Christmas and Happy Holidays.

Jeremy Walker
Decision Frameworks
jeremy@decisionframeworks.com
713-647-9736

Sunday, October 31, 2010

New Course Offering – Working with Risk and Uncertainty

Improving Uncertainty Assessments that Underpin Our Analyses


More and more companies in the oil and gas industry are making use of probabilistic models and risk analysis. Models are becoming more powerful and in combination with new forecasting techniques and computer simulations can now perform probabilistic analyses with ease. One area that has not received much attention, though, is one of the most important – quantifying the uncertainty that underlies these analyses. Whether estimating costs, schedule, commercial or subsurface uncertainty, too often, the range and chance factor assessments that are needed are rushed in the interest of time, even though we all know about “garbage in – garbage out”. When rushed, the assessments often overlook the grounding of the experts in the terminology of uncertainty, and in the fundamentals of quantitatively expressing our strength of belief about an uncertainty.


Decision Frameworks has developed a new training course focused specifically on providing a solid foundation for those who work with uncertainty and risk in any capacity. Designed to be suitable for team members, analysts, and practitioners alike, the course can be tailored to fit the particular needs of each audience. The Working with Risk and Uncertainty course will be offered as an open enrolment course, as well as be introduced as new modules in some of our leading internal course offerings.


Participants will learn the fundamental terminology of probability and statistics and the importance of quantifying and calibrating our judgements about the future. Through simple yet effective exercises, they will learn to make calibrated range and confidence estimates that are consistent and consistently correct. Attendees will also learn how to avoid some of the more common mistakes that are made when combining uncertain variables. Simple models will be built to illustrate the concepts and to provide practice working with uncertainty and risk.


The course will be offered in a variety of modules to address different needs. There will be a half day version for Subject Matter Experts, a full day version for those who use ranges and chance factors in simple models, and two and three day workshops for those who need to understand interactions and dependencies among variables in more complex models. With the shorter workshops, participants will be able to provide quantitative, accurate descriptions of their view of unknown quantities or the likelihood of future events. In the longer versions, participants will explore assessments of variable interactions and their applications in more complex problems.


Who should attend?

  • Uncertainty estimators
  • Engineers
  • Geologists
  • Geophysicists
  • Economists
  • Managers

Wednesday, October 20, 2010

Monte Carlo vs. Decision Trees – Round One

By Jan Schulze, Vice President, Software Development

I’d like to begin this article by stating that although this topic is a discussion centered on methodology, it can become a very emotional subject. Disciples on either side of the debate can get very charged up on the merits of one approach over the other.

So, I’d like to emphasize that I was raised to believe in the virtues of Monte Carlo analysis techniques for all things probabilistic. I saw great utility with this approach and even wrote my own Monte Carlo simulator. I came late to Decision Tree analysis, and now have a much deeper appreciation for its merits, believing that each approach has its place.Over the next few newsletters, I intend to point out the strengths and weakness of both approaches, beginning, in this issue, with a quantitative comparison.

A Two Dimensional Plot of the Solution Space

To begin, let’s use the two-dimensional plot shown in Figure A to represent the full set of potential outcomes for a given opportunity, one outcome of which is displayed on the plot. Instead of relying on one potential outcome, one of our tasks, as analysts, is to describe the full risk profile of the opportunity. Monte Carlo and Decision Tree techniques typically do this in two different ways. Let’s explore.


Figure A - Two Dimensional Problem Space

A Monte Carlo Approach
First, let’s look at Monte Carlo. A Monte Carlo model will generate new potential outcomes in the solution space at random and, if we have enough time and computation power, and if we generate a large number of trials, we will uncover many possible results and explore the extreme corners of the plot. This can be very useful if we are considering “black swan” events (low probability – high impact occurrences). With a valid, representative model, this can provide powerful insights and identify areas of potential concern.


Figure B - Monte Carlo Model

A Decision Tree Approach
By comparison, a Decision Tree model generates solutions in a controlled way. Instead of defining complete distributions, uncertainty ranges are typically represented as three branches on a tree, the P10, P50, and P90 outcomes. When we combine uncertainties together, we might create a plot as seen in Figure C. The tree combinations cover the decision space more symmetrically.


Figure C – Decision Tree Model

Comparing the Two
Now, let us delve into the problem space a little more. If we run the same number of Monte Carlo trials as Decision Tree nodes, we might end up with a plot that would look something like Figure D, where the Monte Carlo analysis is unlikely to explore the space as thoroughly.

If we assume that the opportunity is characterized with five key uncertainties, each with three possible outcomes, we end up with 243 combinations (3*3*3*3*3 = 243). Assuming we use a 30% / 40% / 30% distribution for the probability of each outcome, the chance of all three uncertainties being low at the same time becomes quite small (30% * 30% * 30% * 30% *30% = 0.243%). The Decision Tree dots at the edges of the plot represent these extreme outcomes. Are they close enough to the axes?

The answer depends on the questions you are trying to answer about the opportunity. If you’re primary concern is with the expected reserves distribution, this mapping is probably okay; but if you’re concerned about the remote chance of a well blow-out, this decision tree mapping may not map the extremes well enough.


Figure D – Monte Carlo vs. Decision Tree
If we now look at the probability of any of these extreme events occurring, then we are saying that there is a roughly 1 in 400 chance of seeing one of these results (30% + 30% + 30% + 30% +30% = 0.243%). In other words, we will need to run a minimum of 400 perfectly placed Monte Carlo trials to be able to achieve a coverage that matches that of a Decision Tree. But Monte Carlo analysis will select points randomly; the coverage is unlikely to be “perfect”. To compensate, we need to increase the number of trials, so that we don’t end up with skewed results.

For instance, if we use the analogy that Monte Carlo analysis is like the game “pin the tail on the donkey”, we could end up with a skewed result at the end of the game where we’ve pinned the tail on the right front hoof more times than not, creating not only a painful donkey but an inaccurate view of where the tail could actually be.

The Issue of Sampling and Iterations
How much sampling is required in a Monte Carlo run to ensure so that we get the same coverage of the result space as a Decision Tree? I’ve worked the problem manually, by repeating a Monte Carlo analysis and each time increasing the number of trials, until the results matched that of the Decision Tree (that is, until the distribution of results matched AND until the Monte Carlo analysis produced points that matched or exceeded the points at the extremes as the Decision Tree).

I was only able to match results after I increased the number of Monte Carlo trials to one hundred times more than that of the Decision Tree. For Monte Carlo analysis to match the results of the five-uncertainty Decision Tree example (with 243 combinations, or outcomes), we would need to run 400 x 100 = 40,000 trials.

But how many times would we run 40,000 Monte Carlo trials on five-uncertainty evaluation? This might not even be viable computationally with a complex model. 500 trials, yes. 1000, yes. 5000, maybe. But 40,000? Are we routinely running enough trials to truly understand the results space or are we falsely chasing precision and erroneously feeling that we are getting a better result simply because we have expended more effort to achieve it?

When you compare the two methods now, we see that Decision Tree analysis provides broad coverage of the problem space with a near symmetrical distribution of points, while Monte Carlo analysis can give us the same coverage if we are prepared to run it enough times. Granted at the end of that number of trials, the granularity we will have is much greater than we get with a Decision Tree but, if we can arrive at the same conclusions with 94% less computational effort, why wouldn’t we take it?

What if we are only interested in finding the expected value (or risk weighted value) of the results space? How many Monte Carlo runs does it take to match the results of the Decision Tree? Much fewer. In fact, with a large number of uncertainties, it’s likely that the Monte Carlo evaluation will converge on the expected value with fewer trials than that of a Decision Tree because the Monte Carlo points will tend to cluster around the average. But, remember, our mission as analysts is to map out the full risk profile of the potential results. For this, you’ll need to run a much larger number of iterations.

The Right Tool for the Problem You are Trying to Solve
It is for these reasons that I believe decision trees lend so much value to the typical decision problems we see in the petroleum, pharmaceutical and other sectors we support and why I undertook the challenge of building a new Decision Tree tool.

As ever, which tool is the right tool depends on the problem you are trying to solve. In my view, one of the major reasons Monte Carlo analysis tools have tended to be most popular is because they have been historically easier to use.

Right click on an input in your spreadsheet model, enter a couple numbers and presto, it’s done. Entering distributions on ten uncertainties is easy. By comparison, walking through that same ten-level decision tree with 59,049 end node branches might become painful (if your software tool can even do it). Yet, for all the reasons, I’ve stated above, Decision Trees can be very valuable, insightful, efficient and (depending upon the decision problem) better than Monte Carlo analysis.

As a software developer, I took this as a challenge. Bridge the gap between the ease of use of a Monte Carlo tool and the unique attributes of Decision Tree analysis, one might have a brand new powerful analytic tool. TreeTop™ is my attempt to address this challenge; I will be very interested to hear your thoughts on my efforts.

I appreciate your comments on this article and look forward to more dialogue in this area.

Wednesday, February 10, 2010

Why Trees





By Kent Burkholder


Decision tree analysis has been around for years. It’s rare to find someone who’s never heard of it. Yet from my experience, the tool is vastly underutilized in many companies. Why?

Garbage in, Garbage out
In pushing back on probabilistic analysis people sometimes mutter, “Garbage in, garbage out.” Industry’s track record in assessing the risks and uncertainties has not been stellar and, if we can’t trust the inputs, the outputs must be equally suspect. So goes the argument.

It doesn’t have to be this way. Research has shown that by using expert interviewing techniques to consider a broader range of influences and possible outcomes, our estimates become much better. Like anything, if we practice the techniques, and track and review the results, we improve.


It’s So Subjective
It is. But I would also argue that practically every number in every economic evaluation is subjective, including the future tax and royalty rates (just ask the E&P companies who operate in Alberta!). Somehow we are OK when we put our “best estimate” of all the forecasts into an NPV evaluation but, as soon as we put ranges to those estimates, they become “subjective” (in a bad way). All the numbers are subjective and I mean that in good way, in that the numbers reflect the integration of the relevant, objective data with our years of industry experience and knowhow, combined with our judgment of what could happen in the future.


Gut Feel
Every stakeholder brings a different perspective to a given project, borne of their individual experiences, natural biases, and role within the project. Everyone has a different gut feel how each uncertainty will play out: reserves, capital costs, or whatever. As such, it is difficult to collectively agree on THE ONE scenario to represent all the potential outcomes for the project.

Probabilistic analysis is not burdened by this problem. When we agree the range that might be seen for each key input (uncertainty), the ranges ultimately reflect all the relevant opinions on the team, optimistic and pessimistic. Research has shown that uncertainty assessments become much better when we consider multiple points of view. It’s a great team tool.

Weird Science
Sometimes our intuition on probabilities is quite good. Sometimes it’s not. Consider the following example… Ben Johnston was the first to cross the finish line in the 100 meter sprint in the 1988 Olympic Games. The next day he failed the drug test and was disqualified. Given the following assumptions:

  • 10% of all athletes in 1988 took performance enhancing drugs.
  • The test is 98% accurate in correctly assessing that an athlete has taken drugs, and
  • The test is 93% accurate in correctly assessing that an athlete is clean.

What is the chance that Ben Johnston is guilty after having failed the first test? Greater than 90%, right? Wrong.

Let’s walk through the math to figure it out. First assume that 1000 athletes attend the games. Based on our estimates, how many of the 1000 take performance enhancing drugs? 10%, or 100 people. 900 are clean. If we test all the 100 drug takers, how many will get caught? 98%, or 98 people out of 100. If we test all the 900 clean athletes, how many will be falsely accused? 7% (100%-93%), or 63 athletes. If we tested all 1000 athletes, how many would fail the first test? 98+63, or 161. Therefore given that any randomly selected athlete fails the first test, what is the chance that they are truly taking performance enhancing drugs? 98 / 161, or 60.9%.

The math is not complicated. By writing it down and walking through it logically, we have a check on whether our brain is playing tricks on us. As a team, we are documenting our assumptions for all to see and allowing multiple sets of eyes to look for flaws in our logic.

What if?
Ultimately we are assessing uncertainties to determine how they might influence our decisions, to answer those “what if” questions that gnaw at decision makers. We already run sensitivities to understand how the value might change if prices are low or high but what happens when all of those uncertainties are changing at the same time? Decision trees can be used to test a much wider view of the entire decision space than straight sensitivity analysis and they do so in a structured way.

Consider the diagrams below. The decision tree shows the nine combinations that result from three inputs of both price and reserves. These same combinations are shown as the orange diamonds in a two-dimensional “map” representation to the right. The blue background space is made up of the myriads of potential combinations of price and reserves. The orange diamonds are the points that we have selected to evaluate. You can see that the diamonds cover the decision space quite reasonably… and with only nine points!


Click to enlarge image



Simple Does It, Sometimes
Decision trees can add value even with little hard data. Discussions amongst experts even around simple trees can yield surprising insights that might be masked in a larger evaluation. As such, it’s a great idea to try to dissect a problem as much as you can into components to better understand how each contributes to the overall decision problem. Take the decision tree below, as an example. A team was trying to decide how best to design an upcoming development well. They were concerned about the potential for a water zone below the main oil producing zone and they anticipated needing to decide where to perforate the well bore. If the water zone existed and vertical permeability in the reservoir rock was high (i.e., water flowed easily upwards), they would avoid the water by perforating only the top of the oil interval, keeping as far away from the water as they could. If the vertical permeability was low (i.e., water couldn’t flow upwards), they would perforate the whole oil interval and get the best rates and recovery.

They mapped the problem out in the simple Oracle tree below. With a brief conversation around the probabilities and potential values, they realized that perforating the top of the interval did not erode that much value even when water was not going to be a problem. Besides, they could always go back into the well later to optimize the perforations.


Click to enlarge image





Real Options
Because of their simple structure and simple math, decision trees are generally considered to be easy to understand and interpret. A short explanation probability and risk-weighted value is all that’s required and, once complete, the audience follows along and contributes.

Consider the tree below. We are debating piloting a new enhanced oil recovery (EOR) technology, the results of which help us decide whether or not we would opt for a full-field implementation. The key uncertainty is recovery factor (RF). The tree is structured in time according to how we anticipate the decisions to be made and uncertainties resolved:
We first decide whether or not to run the pilot;
If we pilot, we’ll get the pilot results, and update our interpretation of how effective the technology might be in a full-field implementation;
We’ll then decide whether or not to commercialize; and
If we commercialize, we’ll get the actual results.

Click to enlarge image





When we interpret RF=25% from the pilot and we decide to commercialize, there remains a 26% chance of a highly negative result. Are we comfortable with this? Probably not. Is there something we could do to limit this exposure or improve the interpretation reliability? Possibly. We might extend or expand the pilot. These insights fall out of decision tree analysis just by walking down the branches and answering those “what if” questions. Invariably we uncover even better strategies than we considered initially.

Add Water and Serve
As an industry, we already run NPV analysis on our key investment decisions, complete with sensitivities. To layer decision trees on top of our current evaluations is not a big leap. Teams will gain greater clarity on the options available and the ranges of outcomes, decision makers will gain greater clarity on the risks and rewards, and ultimately we’ll all make better decisions.