Gumball Game Epilogue

Friday, May 18th

Epilogue

As a follow-up to the CPDEP forum gumball quiz, I took matter into my own hands to see if I could create another opportunity to test Surowiecki’s theories….in this case, the results were much different.

I am the Cub Leader at Pack 641 in Houston and we had a year-end gathering at my house for the eldest boys (called Webelos) and leaders tonight. Families were invited and we had about 45 people show up. The ages of the group ranged from about 5 years of age to senior statesman status. People were approached to estimate the number of gumballs in a glass container. This container was smaller and more cylindrical than the one at CPDEP. Of all the attendees, 37 took a shot at estimating the total….some parents were scared away from guessing when they learned that the winner took home the gumballs…(I mean, what else are we going to do with 1337 gumballs?...my wife is concerned I may be on a first named basis with our dentist in about a year’s time if they remain within this household for another week).

When I charted out the results, I saw something that was very interesting.

There were 282 gumballs in the container, the average guess was 257….there were only two people closer, one guessed 275 and another guessed 295. Clearly, this showed that there was some wisdom here. There was only a 2/37 chance that any one person’s estimate would be more correct than the average of all the estimates. Surowiecki should be proud – in this instance, his theory holds up pretty well.

What this says about a conference full of decision analysts versus a pack of cub scouts…I will not comment on.

- Jeremy

Gumball Game Winner Commentary

The Jar and the Gumballs - CPDEP Forum

This game was spawned from James Surowiecki’s best selling book, “The Wisdom of Crowds”, which proposes a vastly different view to the popular belief that collective decision making may be flawed at times. Unlike Surowiecki, there are those who believe that group interactions tend to make people dumb or crazy, or both. Friedrich Nietzsche, summed up his views on group decision making when he stated, “Madness is the exception in individuals but the rule in groups.”

Surowiecki asserts that this statement is not true, contending that groups make excellent decisions when they are able “to maintain order and coherence”. He also asserts that “… groups benefit from members talking to and learning from each other, but too much communication, paradoxically, can actually make the group also be unmanageable and inefficient”. “The idea of the wisdom of crowds is not that a group will always give you the right answer but that on average it will consistently come up with a better answer than any individual could provide.”

With this in mind, we thought that this year’s CPDEP Forum would be the perfect opportunity to run a little test of Surowiecki’s theory. So, we bought some gumballs (red, blue, green and yellow) and a big glass container…and ran the game.

The Results

How many gumballs were in the container? 1337 Who was closest to estimating the correct number? Two people guessed 1350 …which was within 13 gumballs of the correct answer. To break the tie, we consulted the second question on the survey which read, “What do you think the mean of all of the estimates will be?” The winner is Fred Abbott who works in MCP Legal in Houston. In speaking with him about his estimate, he explained that he used his keen legal mind and some precise calculations to figure out his estimate – then use used his dad’s badge number from when he worked at Aramco ???? – ah yes, it is as good to be lucky as to be skilled. Please abuse him and his good fortune.
What was the mean of all of the estimates? 2160 What does this tell us? Well, for one thing it shows that in this instance “The Wisdom of Crowds” failed to prove out. There were 49 of the 78 people who played the game who were closer to the right answer than the average guess. The average of the estimates was 2160. The correct answer was 1337. The difference between the two was 823; 49 people provided estimates that were closer to the correct answer than was the average of all the estimates.

There are a few reasons why the test may have failed. Author, James Surowiecki, claims “…that a group will not always give you the right answer but that on average [italics and bolding added] it will consistently come up with a better answer than any individual could provide.” Perhaps this test was an anomaly and we should run more tests to find out? We plan to do this, using our website as the vehicle for the guessing to allow for broad participation….so, watch this space.

Maybe we didn’t have a large enough number of participants. 78 people submitted estimates. This may not be a large enough sample size. Perhaps if more people were to participate in the game, the extremely large estimates submitted would balance out and the average of the guesses would be closer to the real number of gumballs. (Ten people guessed 4000 or higher; two of those well-calibrated souls guessed 10,000, certainly skewing the mean of the estimates.)

Perhaps we had some very insightful spirits who figured out that if they guessed very inaccurately, then they could skew the average guess to being in their favor. It was never clearly explained that we would use the “mean” of all the guesses as the tie-breaker. Perhaps people reasoned that they could tilt the game in their favor with a large number estimate that would skew the mean to the high side.

After all, seven of the ten high guessers chose estimates of the mean that were below their own estimate AND Decision Frameworks’ presentation was sandwiched between two booths of Game Theory consultants. So, go figure, was their some game theory at play? Or, is there always some game theory at play?

The shape of the container might have had something to do with the over-estimation. It was shaped like a giant beer glass with a wide throat, a narrow neck and a broad base… a bit hourglass in shape. If people simply counted the gumballs they could see in the open top, and then multiplied the sum by the number of rows deep, they would have over-estimated the total.

Lastly, we tend to be better at estimating things we are more familiar with. For example, if you’ve ever rolled coins before, after a couple times, you become pretty good at grabbing a group of around 50 pennies to make a roll (adding or subtracting a few to make the roll complete)….but what do 100 gumballs look like? How many are really in that gumball machine at the car wash? In this case, this may be the first time someone actually saw 1337 gumballs.

In summary, there may be a variety of reasons that the gumball game fell short of proving the utility of the wisdom of crowds for estimation of uncertainty… because there was definitely wisdom in this crowd. (It was, after all, a decision quality oriented forum.) Interestingly, 50 of the 78 people had “means” lower than their estimates. Perhaps they already sensed that they were over-estimating the answer?

We don’t want to make any definitive conclusions because it was only one test; we’d like to run another couple of games and see if the results change.

Thanks to everyone who participated and congratulations to the winner (who has been sent an email announcing his sound estimating capabilities), and we will send you a note when another public estimating game ensues.

- Jeremy