November 15, 2011

CPDEP - Gumball 2011 Blog

Anyone know Kent Rinehart? Well, his parents and family must be very proud of him, he has achieved big things in his life, no bigger than being able to win our Gumball Challenge. As many of you who have frequently attended the annual CPDEP forum know, every year we run a little contest to see how “awesome” you are at estimating the number of gumballs in an oddly shaped container. This year was no different than past conferences , though there may have been a few more “squeezie Owls” present, which could have been a distraction.

At this year’s forum, those who chose to play the game were asked for their P10 – P90 estimates of the number of gumballs in the container. Then, they were asked to provide an estimate of how many were actually in the jar. Their ballots were recorded, the gumballs were counted and the results were reviewed.

Bottom line, Kent was up to the challenge and used his impressive gumball counting skills to come closest to the actual number. His estimate of 1320 was less than 1% away from the actual number of 1330. When he was asked about his gumball counting abilities he refused to reveal his techniques only saying “I’m multi-talented”. This impressive feat does not go unrewarded, as he has won himself the brand new Kindle Fire.

Now why do we run this game? Well, we are big believers in the Wisdom of Crowds, a book written by James Surowiecki. In it, he suggests that we put too much emphasis on one expert’s opinion and that if you solicit a group of individuals, with some relevant knowledge, and collect data from their responses, you may be very pleasantly surprised by the result of more accurate forecasts. In the case where there is a finite answer (i.e., one where a knowable answer may lie within a definable range, i.e., the number of gumballs in a container, the actual weight of a bull or the population of Milwaukee, etc.), the average of the estimates may be as close or closer to the actual number than any single best estimate.

So for those of you who would like to look through the numbers, here you go:

Number of estimates: 104
Range of estimates: 72-50000

Average/mean of all estimates: 2679
Actual number of gumballs: 1330
Kent’s estimate: 1320
Calculated Mean over just the P10-P90 range of the estimates: 1311

Number of people whose estimate was below the actual result: 60
Number of people whose estimate was above the actual result: 43
Number of P10-P90 estimates wide enough to include the actual result: 30.8%
Number of P10-P90 estimates which missed on the low side: 26.9%
Number of P10-P90 estimates which missed on the high side: 42.3%













Figure A: Distribution of Gumball Estimates
(Shown in Decision Frameworks new decision tree tool – TreeTop™)

So here are some early observations:

1) Chevron needs to revisit its employee optical plan – with estimates between 72 and 50,000, we either had people trying to game the results or was something in what they were drinking?

2) As a result, there were a number of outliers provided in the estimates. You can see from the graph below that there was a big skew to the upside. It doesn’t take very many people to guess extremely high values before the graph gets distorted. In this case, we had more people underestimate the number of gumballs than overestimate, but those that overestimated, way overestimated.

3) Surowiecki’s test failed as the average of all the estimates was way off. In fact, the average of all estimates was more than double the actual number of gumballs. In other tests, we’ve seen this to be much closer.

4) Interestingly (and we’ve done this before with this game), if we take out the outliers by only running an average over the P10-P90 range of all estimates, we see that the average is much closer to the actual result. In this case, if we take out 21 estimates (i.e., the majority of the outliers), focus only on the P10-P90 range of all the estimates, and then average the results, we end up at 1311. Now this is quite thought provoking because there is only one estimate closer than this, and that is Kent’s estimate. You can create your own conclusions from this exercise. All we can say is this result is pretty consistent with other tests and trials of this nature which we’ve run using gumballs.

5) Let’s turn our attention to people’s ability to come up with an effective range. When we look at the ability of people to provide a good P10-P90 estimate (one that would include the resulting number of gumballs), we see that only 30% of the people were capable of providing a solid inclusive range. This means that 70% of people were too narrow with their estimates and missed including the result. Is this unusual? No, especially without some expert interviewing techniques.

One of the big failings we see is that people tend to be too narrow in their estimates. There was one person who had a range as narrow as 36 gumballs. You’d have to be pretty good to be that precise and in this case the individual was off by a factor of 5. How do you avoid this bias? Through effective expert interviewing techniques which teach those providing the estimates to start their thinking from the outer edges of what they know and work inward.

If you have interest in learning more about expert interviewing you can check out one of our free webinars in 2012 and or attend one of our 3-day courses. Additionally, there are great expert interview templates in DTrio which is available to be downloaded from the GIL Option Panel.

In summary, we hope you will stop by Kent’s office and congratulate him on his impressive abilities. We also hope to see many of you again through project work and/or in one of our upcoming courses. Remember, great decisions are derived not divined!

Cheers,
Jeremy Walker
jeremy@decisionframeworks.com

INFORMS Annual Meeting - Gumball 2010 Blog

From Sunday, November 7 through Wednesday, November 10, 2010, we participated in the 2010 INFORMS Annual Meeting at the Conference Center in Austin, Texas We had a booth, ran a pre-meeting seminar and showcased our latest decision framing (DTrio) and decision tree software (TreeTop) at the booth and in a Tuesday morning demonstration.

During the conference, we ran another in a series of tests based around the “Wisdom of Crowds”, James Surowiecki’s best-selling book. In it he contends that we put too much emphasis on expert opinion and if you solicit a group of individuals in an independent manner and collect data from their responses, you may be very pleasantly surprised by the results.

This year’s experiment saw an oddly shaped glass container standing at the corner of our booth filled with red, blue, green and yellow gumballs, as along with an unknown number of rubber squishy owls. Visitors were asked to guess the number of gumballs in the container. They were first asked for their P10 and P90 estimates. In other words, they should be 80% confident that the actual number of gumballs should fit between their range. That should be pretty easy to do, right?

At the bottom of the ballot, they were asked for their best guess on the exact number of gumballs in the container. In an effort to reward the best estimator in this experiment, we offered an iPod to the individual with the closest guess.

So without another wasted breath, here are the results:

Number of estimates
78
Range of estimates
45-11075
Average of all estimates
1914
Actual number of gumballs
1371
P50 estimate of gumballs (calculated)
1350
Closest Best Estimate
1400
Winner
Justin Mao-Jones

Figure A: Distribution of Gumball Estimates
(Shown in Decision Frameworks new decision tree tool – TreeTop™)

Let’s take a look at the results. Firstly, two people guessed that there were 1400 gumballs in the container. Justin was judged to be the winner because he handed his ballot in the earliest (We knew that small print on the back side of the ballot on how the winner would be determined would come in handy one day ;>0 ). Upon hearing the news, he mentioned donating it to a worthy cause - which he named as himself. To the winner go the spoils! Good job.

Secondly, let’s step back and take a look at what the data is telling us. We see we had a fairly wide range of estimates 45 – 11,075. What is noteworthy is that the EV (expected Value) was skewed so by the “fat tails” on the high side. There was an equal number of estimates above and below the closest answer, so those who thought they saw more gumballs ready thought they saw lots more gumballs, thereby skewing the average to the upside. Does anyone have any evidence that shows when people are confronted with a large number of objects that they tend to disproportionally overestimate than underestimate the correct number?

As stated above, when you order the estimates from lowest to highest, Justin’s estimate was interestingly at the exact mid-point. His was the 39th out of 78 sequentially ordered estimates. His was the mid-point estimate and it was also closest. Does this support Surowiecki’s assertions about the wisdom of crowds? Though this is not the statistical average of all estimates, it is smack-dab in the middle of everyone’s wisdom. A gumball for your thoughts on this little tidbit of information…

Here’s another interesting finding from looking at the data. If you remember, you were asked when you submitted your estimate to write down a bounded range with your P10 and P90 limits. When we look at the results, only 34.1% of you had a range that was wide enough to contain the actual number. The means that roughly 2 out of 3 people were not able to create a reasonable estimate, an 80% confidence range around this uncertainty. This confirms other experiences we see in our daily consulting and training courses. People seem to have an inherent blind spot to creating broad enough ranges. Perhaps because people are often rewarded for the exactness/precision of their answers, there is a perceived weakness if too broad a range is given as an answer. Alternatively, maybe gumballs are just too tricky to count.

Regardless, what it shows is that proper care needs to be taken when it comes around to gathering range estimates. Do your best to de-bias and create more realistic extremes. There are several techniques we use to do this including structured interviews with expert interview templates. We can teach you if you are unfamiliar with these tools and techniques.

There is much more that bigger brains than mine might glean from this data. If you would like a copy of the raw excel file, shoot me a note and I’ll send you a copy.

If this has sparked your interest or you would like to know more about the training and tools Decision Frameworks provides those who are focused on improving the quality of decision making within their organizations, please click here.
.
Thanks for everyone who participated in this contest. We hope to see many of you before the next INFORMS Conference. And remember great decision making is not divined - it is derived!

Merry Christmas and Happy Holidays.

Jeremy Walker
Decision Frameworks
jeremy@decisionframeworks.com
713-647-9736

New Course Offering – Working with Risk and Uncertainty

Improving Uncertainty Assessments that Underpin Our Analyses

More and more companies in the oil and gas industry are making use of probabilistic models and risk analysis. Models are becoming more powerful and in combination with new forecasting techniques and computer simulations can now perform probabilistic analyses with ease. One area that has not received much attention, though, is one of the most important – quantifying the uncertainty that underlies these analyses. Whether estimating costs, schedule, commercial or subsurface uncertainty, too often, the range and chance factor assessments that are needed are rushed in the interest of time, even though we all know about “garbage in – garbage out”. When rushed, the assessments often overlook the grounding of the experts in the terminology of uncertainty, and in the fundamentals of quantitatively expressing our strength of belief about an uncertainty.

Decision Frameworks has developed a new training course focused specifically on providing a solid foundation for those who work with uncertainty and risk in any capacity. Designed to be suitable for team members, analysts, and practitioners alike, the course can be tailored to fit the particular needs of each audience. The Working with Risk and Uncertainty course will be offered as an open enrolment course, as well as be introduced as new modules in some of our leading internal course offerings.

Participants will learn the fundamental terminology of probability and statistics and the importance of quantifying and calibrating our judgements about the future. Through simple yet effective exercises, they will learn to make calibrated range and confidence estimates that are consistent and consistently correct. Attendees will also learn how to avoid some of the more common mistakes that are made when combining uncertain variables. Simple models will be built to illustrate the concepts and to provide practice working with uncertainty and risk.

The course will be offered in a variety of modules to address different needs. There will be a half day version for Subject Matter Experts, a full day version for those who use ranges and chance factors in simple models, and two and three day workshops for those who need to understand interactions and dependencies among variables in more complex models. With the shorter workshops, participants will be able to provide quantitative, accurate descriptions of their view of unknown quantities or the likelihood of future events. In the longer versions, participants will explore assessments of variable interactions and their applications in more complex problems.

Who should attend?

• Uncertainty estimators
• Engineers
• Geologists
• Geophysicists
• Economists
• Managers

Monte Carlo vs. Decision Trees – Round One

By Jan Schulze, Vice President, Software Development

I’d like to begin this article by stating that although this topic is a discussion centered on methodology, it can become a very emotional subject. Disciples on either side of the debate can get very charged up on the merits of one approach over the other.

So, I’d like to emphasize that I was raised to believe in the virtues of Monte Carlo analysis techniques for all things probabilistic. I saw great utility with this approach and even wrote my own Monte Carlo simulator. I came late to Decision Tree analysis, and now have a much deeper appreciation for its merits, believing that each approach has its place.Over the next few newsletters, I intend to point out the strengths and weakness of both approaches, beginning, in this issue, with a quantitative comparison.

A Two Dimensional Plot of the Solution Space
To begin, let’s use the two-dimensional plot shown in Figure A to represent the full set of potential outcomes for a given opportunity, one outcome of which is displayed on the plot. Instead of relying on one potential outcome, one of our tasks, as analysts, is to describe the full risk profile of the opportunity. Monte Carlo and Decision Tree techniques typically do this in two different ways. Let’s explore.


Figure A - Two Dimensional Problem Space

A Monte Carlo Approach
First, let’s look at Monte Carlo. A Monte Carlo model will generate new potential outcomes in the solution space at random and, if we have enough time and computation power, and if we generate a large number of trials, we will uncover many possible results and explore the extreme corners of the plot. This can be very useful if we are considering “black swan” events (low probability – high impact occurrences). With a valid, representative model, this can provide powerful insights and identify areas of potential concern.


Figure B - Monte Carlo Model

A Decision Tree Approach
By comparison, a Decision Tree model generates solutions in a controlled way. Instead of defining complete distributions, uncertainty ranges are typically represented as three branches on a tree, the P10, P50, and P90 outcomes. When we combine uncertainties together, we might create a plot as seen in Figure C. The tree combinations cover the decision space more symmetrically.


Figure C – Decision Tree Model

Comparing the Two
Now, let us delve into the problem space a little more. If we run the same number of Monte Carlo trials as Decision Tree nodes, we might end up with a plot that would look something like Figure D, where the Monte Carlo analysis is unlikely to explore the space as thoroughly.

If we assume that the opportunity is characterized with five key uncertainties, each with three possible outcomes, we end up with 243 combinations (3*3*3*3*3 = 243). Assuming we use a 30% / 40% / 30% distribution for the probability of each outcome, the chance of all three uncertainties being low at the same time becomes quite small (30% * 30% * 30% * 30% *30% = 0.243%). The Decision Tree dots at the edges of the plot represent these extreme outcomes. Are they close enough to the axes?

The answer depends on the questions you are trying to answer about the opportunity. If you’re primary concern is with the expected reserves distribution, this mapping is probably okay; but if you’re concerned about the remote chance of a well blow-out, this decision tree mapping may not map the extremes well enough.


Figure D – Monte Carlo vs. Decision Tree
If we now look at the probability of any of these extreme events occurring, then we are saying that there is a roughly 1 in 400 chance of seeing one of these results (30% + 30% + 30% + 30% +30% = 0.243%). In other words, we will need to run a minimum of 400 perfectly placed Monte Carlo trials to be able to achieve a coverage that matches that of a Decision Tree. But Monte Carlo analysis will select points randomly; the coverage is unlikely to be “perfect”. To compensate, we need to increase the number of trials, so that we don’t end up with skewed results.

For instance, if we use the analogy that Monte Carlo analysis is like the game “pin the tail on the donkey”, we could end up with a skewed result at the end of the game where we’ve pinned the tail on the right front hoof more times than not, creating not only a painful donkey but an inaccurate view of where the tail could actually be.

The Issue of Sampling and Iterations
How much sampling is required in a Monte Carlo run to ensure so that we get the same coverage of the result space as a Decision Tree? I’ve worked the problem manually, by repeating a Monte Carlo analysis and each time increasing the number of trials, until the results matched that of the Decision Tree (that is, until the distribution of results matched AND until the Monte Carlo analysis produced points that matched or exceeded the points at the extremes as the Decision Tree).

I was only able to match results after I increased the number of Monte Carlo trials to one hundred times more than that of the Decision Tree. For Monte Carlo analysis to match the results of the five-uncertainty Decision Tree example (with 243 combinations, or outcomes), we would need to run 400 x 100 = 40,000 trials.

But how many times would we run 40,000 Monte Carlo trials on five-uncertainty evaluation? This might not even be viable computationally with a complex model. 500 trials, yes. 1000, yes. 5000, maybe. But 40,000? Are we routinely running enough trials to truly understand the results space or are we falsely chasing precision and erroneously feeling that we are getting a better result simply because we have expended more effort to achieve it?

When you compare the two methods now, we see that Decision Tree analysis provides broad coverage of the problem space with a near symmetrical distribution of points, while Monte Carlo analysis can give us the same coverage if we are prepared to run it enough times. Granted at the end of that number of trials, the granularity we will have is much greater than we get with a Decision Tree but, if we can arrive at the same conclusions with 94% less computational effort, why wouldn’t we take it?

What if we are only interested in finding the expected value (or risk weighted value) of the results space? How many Monte Carlo runs does it take to match the results of the Decision Tree? Much fewer. In fact, with a large number of uncertainties, it’s likely that the Monte Carlo evaluation will converge on the expected value with fewer trials than that of a Decision Tree because the Monte Carlo points will tend to cluster around the average. But, remember, our mission as analysts is to map out the full risk profile of the potential results. For this, you’ll need to run a much larger number of iterations.

The Right Tool for the Problem You are Trying to Solve
It is for these reasons that I believe decision trees lend so much value to the typical decision problems we see in the petroleum, pharmaceutical and other sectors we support and why I undertook the challenge of building a new Decision Tree tool.

As ever, which tool is the right tool depends on the problem you are trying to solve. In my view, one of the major reasons Monte Carlo analysis tools have tended to be most popular is because they have been historically easier to use.

Right click on an input in your spreadsheet model, enter a couple numbers and presto, it’s done. Entering distributions on ten uncertainties is easy. By comparison, walking through that same ten-level decision tree with 59,049 end node branches might become painful (if your software tool can even do it). Yet, for all the reasons, I’ve stated above, Decision Trees can be very valuable, insightful, efficient and (depending upon the decision problem) better than Monte Carlo analysis.

As a software developer, I took this as a challenge. Bridge the gap between the ease of use of a Monte Carlo tool and the unique attributes of Decision Tree analysis, one might have a brand new powerful analytic tool. TreeTop™ is my attempt to address this challenge; I will be very interested to hear your thoughts on my efforts.

I appreciate your comments on this article and look forward to more dialogue in this area.

Why Trees

By Kent Burkholder


Decision tree analysis has been around for years. It’s rare to find someone who’s never heard of it. Yet from my experience, the tool is vastly underutilized in many companies. Why?

Garbage in, Garbage out
In pushing back on probabilistic analysis people sometimes mutter, “Garbage in, garbage out.” Industry’s track record in assessing the risks and uncertainties has not been stellar and, if we can’t trust the inputs, the outputs must be equally suspect. So goes the argument.

It doesn’t have to be this way. Research has shown that by using expert interviewing techniques to consider a broader range of influences and possible outcomes, our estimates become much better. Like anything, if we practice the techniques, and track and review the results, we improve.


It’s So Subjective
It is. But I would also argue that practically every number in every economic evaluation is subjective, including the future tax and royalty rates (just ask the E&P companies who operate in Alberta!). Somehow we are OK when we put our “best estimate” of all the forecasts into an NPV evaluation but, as soon as we put ranges to those estimates, they become “subjective” (in a bad way). All the numbers are subjective and I mean that in good way, in that the numbers reflect the integration of the relevant, objective data with our years of industry experience and knowhow, combined with our judgment of what could happen in the future.


Gut Feel
Every stakeholder brings a different perspective to a given project, borne of their individual experiences, natural biases, and role within the project. Everyone has a different gut feel how each uncertainty will play out: reserves, capital costs, or whatever. As such, it is difficult to collectively agree on THE ONE scenario to represent all the potential outcomes for the project.

Probabilistic analysis is not burdened by this problem. When we agree the range that might be seen for each key input (uncertainty), the ranges ultimately reflect all the relevant opinions on the team, optimistic and pessimistic. Research has shown that uncertainty assessments become much better when we consider multiple points of view. It’s a great team tool.

Weird Science
Sometimes our intuition on probabilities is quite good. Sometimes it’s not. Consider the following example… Ben Johnston was the first to cross the finish line in the 100 meter sprint in the 1988 Olympic Games. The next day he failed the drug test and was disqualified. Given the following assumptions:

• 10% of all athletes in 1988 took performance enhancing drugs.
• The test is 98% accurate in correctly assessing that an athlete has taken drugs, and
• The test is 93% accurate in correctly assessing that an athlete is clean.
  

What is the chance that Ben Johnston is guilty after having failed the first test? Greater than 90%, right? Wrong.

Let’s walk through the math to figure it out. First assume that 1000 athletes attend the games. Based on our estimates, how many of the 1000 take performance enhancing drugs? 10%, or 100 people. 900 are clean. If we test all the 100 drug takers, how many will get caught? 98%, or 98 people out of 100. If we test all the 900 clean athletes, how many will be falsely accused? 7% (100%-93%), or 63 athletes. If we tested all 1000 athletes, how many would fail the first test? 98+63, or 161. Therefore given that any randomly selected athlete fails the first test, what is the chance that they are truly taking performance enhancing drugs? 98 / 161, or 60.9%.

The math is not complicated. By writing it down and walking through it logically, we have a check on whether our brain is playing tricks on us. As a team, we are documenting our assumptions for all to see and allowing multiple sets of eyes to look for flaws in our logic.

What if?
Ultimately we are assessing uncertainties to determine how they might influence our decisions, to answer those “what if” questions that gnaw at decision makers. We already run sensitivities to understand how the value might change if prices are low or high but what happens when all of those uncertainties are changing at the same time? Decision trees can be used to test a much wider view of the entire decision space than straight sensitivity analysis and they do so in a structured way.

Consider the diagrams below. The decision tree shows the nine combinations that result from three inputs of both price and reserves. These same combinations are shown as the orange diamonds in a two-dimensional “map” representation to the right. The blue background space is made up of the myriads of potential combinations of price and reserves. The orange diamonds are the points that we have selected to evaluate. You can see that the diamonds cover the decision space quite reasonably… and with only nine points!

Click to enlarge image



Simple Does It, Sometimes
Decision trees can add value even with little hard data. Discussions amongst experts even around simple trees can yield surprising insights that might be masked in a larger evaluation. As such, it’s a great idea to try to dissect a problem as much as you can into components to better understand how each contributes to the overall decision problem. Take the decision tree below, as an example. A team was trying to decide how best to design an upcoming development well. They were concerned about the potential for a water zone below the main oil producing zone and they anticipated needing to decide where to perforate the well bore. If the water zone existed and vertical permeability in the reservoir rock was high (i.e., water flowed easily upwards), they would avoid the water by perforating only the top of the oil interval, keeping as far away from the water as they could. If the vertical permeability was low (i.e., water couldn’t flow upwards), they would perforate the whole oil interval and get the best rates and recovery.

They mapped the problem out in the simple Oracle tree below. With a brief conversation around the probabilities and potential values, they realized that perforating the top of the interval did not erode that much value even when water was not going to be a problem. Besides, they could always go back into the well later to optimize the perforations.

Click to enlarge image





Real Options
Because of their simple structure and simple math, decision trees are generally considered to be easy to understand and interpret. A short explanation probability and risk-weighted value is all that’s required and, once complete, the audience follows along and contributes.

Consider the tree below. We are debating piloting a new enhanced oil recovery (EOR) technology, the results of which help us decide whether or not we would opt for a full-field implementation. The key uncertainty is recovery factor (RF). The tree is structured in time according to how we anticipate the decisions to be made and uncertainties resolved:
We first decide whether or not to run the pilot;
If we pilot, we’ll get the pilot results, and update our interpretation of how effective the technology might be in a full-field implementation;
We’ll then decide whether or not to commercialize; and
If we commercialize, we’ll get the actual results.

Click to enlarge image

When we interpret RF=25% from the pilot and we decide to commercialize, there remains a 26% chance of a highly negative result. Are we comfortable with this? Probably not. Is there something we could do to limit this exposure or improve the interpretation reliability? Possibly. We might extend or expand the pilot. These insights fall out of decision tree analysis just by walking down the branches and answering those “what if” questions. Invariably we uncover even better strategies than we considered initially.

Add Water and Serve
As an industry, we already run NPV analysis on our key investment decisions, complete with sensitivities. To layer decision trees on top of our current evaluations is not a big leap. Teams will gain greater clarity on the options available and the ranges of outcomes, decision makers will gain greater clarity on the risks and rewards, and ultimately we’ll all make better decisions.

Introductions

For my first blog post to the Decision Frameworks blog, I thought I should probably introduce myself. My name is Jan and my main responsibility is the software development side of Decision Frameworks.

I love to develop software, and so I am in the admirable position to spend a great deal of time doing what I love every day. Most of our software development is undertaken in C# and is geared specifically towards the Windows platform. I have always been very interested in graphical systems and user interaction design issues – both of which are topics that are core to our current development tasks. But I will delve into more detail on those in future blog posts.
In my spare time I read a lot, ranging from rather technical computer science and mathematics books over business books (preferably business biographies) to fiction. On the fiction side I quite enjoy authors like Dale Brown, Tom Clancy, James Patterson, Chris Ryan, etc.

I have to admit to a bike fetish. I own three race bikes, a mountain bike, and various other contraptions on two wheels. The area that I live in (in Germany) is quite nice for biking, since I can choose quite freely whether I am up for an uphill battle or prefer a relaxed ride with almost no climbing at all.

On the subject of sports, I also run from time to time. I have never really enjoyed running – and can’t imagine that I ever will. But I find it a time-effective way to burn some calories in an attempt to stay in shape. Running also gives me the time to listen to the latest pod casts by other geeks out there – or sometimes I just use the time to churn through some of the tougher software development problems we face.

In addition to biking and reading, I am also in a perennial search for the best affordable home entertainment system. The next addition to that end of things is most likely going to be a Mac Mini. But I reserve the right to change my mind on that one.

I am also starting to develop an interest in digital photography. We own a Nikon D80 and enjoy taking pictures with it. We catalogue and edit the images using Adobe Photoshop Lightroom – which has turned out to be a great piece of software for the task. We’re still pretty early on the learning curve both in terms of actually taking photos as well as the software – but it looks like there’s fun ride ahead of us.

That’s probably enough information on me to get started. You’ll get to know me better in future blog posts. You can expect blog posts about software development, usually quite specifically about what we’re working on. Or, in fact, pretty much anything that comes to mind. I am working on a piece around project scheduling, as well as something around the next wave of .coms.

If you’re more interested in a CV or Bio, you can also look me up in the section about me on our website.

Stay tuned!

Gumball Game Epilogue

Friday, May 18th

Epilogue

As a follow-up to the CPDEP forum gumball quiz, I took matter into my own hands to see if I could create another opportunity to test Surowiecki’s theories….in this case, the results were much different.

I am the Cub Leader at Pack 641 in Houston and we had a year-end gathering at my house for the eldest boys (called Webelos) and leaders tonight. Families were invited and we had about 45 people show up. The ages of the group ranged from about 5 years of age to senior statesman status. People were approached to estimate the number of gumballs in a glass container. This container was smaller and more cylindrical than the one at CPDEP. Of all the attendees, 37 took a shot at estimating the total….some parents were scared away from guessing when they learned that the winner took home the gumballs…(I mean, what else are we going to do with 1337 gumballs?...my wife is concerned I may be on a first named basis with our dentist in about a year’s time if they remain within this household for another week).

When I charted out the results, I saw something that was very interesting.

There were 282 gumballs in the container, the average guess was 257….there were only two people closer, one guessed 275 and another guessed 295. Clearly, this showed that there was some wisdom here. There was only a 2/37 chance that any one person’s estimate would be more correct than the average of all the estimates. Surowiecki should be proud – in this instance, his theory holds up pretty well.

What this says about a conference full of decision analysts versus a pack of cub scouts…I will not comment on.

- Jeremy

Gumball Game Winner Commentary

The Jar and the Gumballs - CPDEP Forum

This game was spawned from James Surowiecki’s best selling book, “The Wisdom of Crowds”, which proposes a vastly different view to the popular belief that collective decision making may be flawed at times. Unlike Surowiecki, there are those who believe that group interactions tend to make people dumb or crazy, or both. Friedrich Nietzsche, summed up his views on group decision making when he stated, “Madness is the exception in individuals but the rule in groups.”

Surowiecki asserts that this statement is not true, contending that groups make excellent decisions when they are able “to maintain order and coherence”. He also asserts that “… groups benefit from members talking to and learning from each other, but too much communication, paradoxically, can actually make the group also be unmanageable and inefficient”. “The idea of the wisdom of crowds is not that a group will always give you the right answer but that on average it will consistently come up with a better answer than any individual could provide.”

With this in mind, we thought that this year’s CPDEP Forum would be the perfect opportunity to run a little test of Surowiecki’s theory. So, we bought some gumballs (red, blue, green and yellow) and a big glass container…and ran the game.

The Results

How many gumballs were in the container? 1337 Who was closest to estimating the correct number? Two people guessed 1350 …which was within 13 gumballs of the correct answer. To break the tie, we consulted the second question on the survey which read, “What do you think the mean of all of the estimates will be?” The winner is Fred Abbott who works in MCP Legal in Houston. In speaking with him about his estimate, he explained that he used his keen legal mind and some precise calculations to figure out his estimate – then use used his dad’s badge number from when he worked at Aramco ???? – ah yes, it is as good to be lucky as to be skilled. Please abuse him and his good fortune.
What was the mean of all of the estimates? 2160 What does this tell us? Well, for one thing it shows that in this instance “The Wisdom of Crowds” failed to prove out. There were 49 of the 78 people who played the game who were closer to the right answer than the average guess. The average of the estimates was 2160. The correct answer was 1337. The difference between the two was 823; 49 people provided estimates that were closer to the correct answer than was the average of all the estimates.

There are a few reasons why the test may have failed. Author, James Surowiecki, claims “…that a group will not always give you the right answer but that on average [italics and bolding added] it will consistently come up with a better answer than any individual could provide.” Perhaps this test was an anomaly and we should run more tests to find out? We plan to do this, using our website as the vehicle for the guessing to allow for broad participation….so, watch this space.

Maybe we didn’t have a large enough number of participants. 78 people submitted estimates. This may not be a large enough sample size. Perhaps if more people were to participate in the game, the extremely large estimates submitted would balance out and the average of the guesses would be closer to the real number of gumballs. (Ten people guessed 4000 or higher; two of those well-calibrated souls guessed 10,000, certainly skewing the mean of the estimates.)

Perhaps we had some very insightful spirits who figured out that if they guessed very inaccurately, then they could skew the average guess to being in their favor. It was never clearly explained that we would use the “mean” of all the guesses as the tie-breaker. Perhaps people reasoned that they could tilt the game in their favor with a large number estimate that would skew the mean to the high side.

After all, seven of the ten high guessers chose estimates of the mean that were below their own estimate AND Decision Frameworks’ presentation was sandwiched between two booths of Game Theory consultants. So, go figure, was their some game theory at play? Or, is there always some game theory at play?

The shape of the container might have had something to do with the over-estimation. It was shaped like a giant beer glass with a wide throat, a narrow neck and a broad base… a bit hourglass in shape. If people simply counted the gumballs they could see in the open top, and then multiplied the sum by the number of rows deep, they would have over-estimated the total.

Lastly, we tend to be better at estimating things we are more familiar with. For example, if you’ve ever rolled coins before, after a couple times, you become pretty good at grabbing a group of around 50 pennies to make a roll (adding or subtracting a few to make the roll complete)….but what do 100 gumballs look like? How many are really in that gumball machine at the car wash? In this case, this may be the first time someone actually saw 1337 gumballs.

In summary, there may be a variety of reasons that the gumball game fell short of proving the utility of the wisdom of crowds for estimation of uncertainty… because there was definitely wisdom in this crowd. (It was, after all, a decision quality oriented forum.) Interestingly, 50 of the 78 people had “means” lower than their estimates. Perhaps they already sensed that they were over-estimating the answer?

We don’t want to make any definitive conclusions because it was only one test; we’d like to run another couple of games and see if the results change.

Thanks to everyone who participated and congratulations to the winner (who has been sent an email announcing his sound estimating capabilities), and we will send you a note when another public estimating game ensues.

- Jeremy