Gumball Game Winner Commentary

The Jar and the Gumballs - CPDEP Forum

This game was spawned from James Surowiecki’s best selling book, “The Wisdom of Crowds”, which proposes a vastly different view to the popular belief that collective decision making may be flawed at times. Unlike Surowiecki, there are those who believe that group interactions tend to make people dumb or crazy, or both. Friedrich Nietzsche, summed up his views on group decision making when he stated, “Madness is the exception in individuals but the rule in groups.”

Surowiecki asserts that this statement is not true, contending that groups make excellent decisions when they are able “to maintain order and coherence”. He also asserts that “… groups benefit from members talking to and learning from each other, but too much communication, paradoxically, can actually make the group also be unmanageable and inefficient”. “The idea of the wisdom of crowds is not that a group will always give you the right answer but that on average it will consistently come up with a better answer than any individual could provide.”

With this in mind, we thought that this year’s CPDEP Forum would be the perfect opportunity to run a little test of Surowiecki’s theory. So, we bought some gumballs (red, blue, green and yellow) and a big glass container…and ran the game.

The Results

How many gumballs were in the container? 1337 Who was closest to estimating the correct number? Two people guessed 1350 …which was within 13 gumballs of the correct answer. To break the tie, we consulted the second question on the survey which read, “What do you think the mean of all of the estimates will be?” The winner is Fred Abbott who works in MCP Legal in Houston. In speaking with him about his estimate, he explained that he used his keen legal mind and some precise calculations to figure out his estimate – then use used his dad’s badge number from when he worked at Aramco ???? – ah yes, it is as good to be lucky as to be skilled. Please abuse him and his good fortune.
What was the mean of all of the estimates? 2160 What does this tell us? Well, for one thing it shows that in this instance “The Wisdom of Crowds” failed to prove out. There were 49 of the 78 people who played the game who were closer to the right answer than the average guess. The average of the estimates was 2160. The correct answer was 1337. The difference between the two was 823; 49 people provided estimates that were closer to the correct answer than was the average of all the estimates.

There are a few reasons why the test may have failed. Author, James Surowiecki, claims “…that a group will not always give you the right answer but that on average [italics and bolding added] it will consistently come up with a better answer than any individual could provide.” Perhaps this test was an anomaly and we should run more tests to find out? We plan to do this, using our website as the vehicle for the guessing to allow for broad participation….so, watch this space.

Maybe we didn’t have a large enough number of participants. 78 people submitted estimates. This may not be a large enough sample size. Perhaps if more people were to participate in the game, the extremely large estimates submitted would balance out and the average of the guesses would be closer to the real number of gumballs. (Ten people guessed 4000 or higher; two of those well-calibrated souls guessed 10,000, certainly skewing the mean of the estimates.)

Perhaps we had some very insightful spirits who figured out that if they guessed very inaccurately, then they could skew the average guess to being in their favor. It was never clearly explained that we would use the “mean” of all the guesses as the tie-breaker. Perhaps people reasoned that they could tilt the game in their favor with a large number estimate that would skew the mean to the high side.

After all, seven of the ten high guessers chose estimates of the mean that were below their own estimate AND Decision Frameworks’ presentation was sandwiched between two booths of Game Theory consultants. So, go figure, was their some game theory at play? Or, is there always some game theory at play?

The shape of the container might have had something to do with the over-estimation. It was shaped like a giant beer glass with a wide throat, a narrow neck and a broad base… a bit hourglass in shape. If people simply counted the gumballs they could see in the open top, and then multiplied the sum by the number of rows deep, they would have over-estimated the total.

Lastly, we tend to be better at estimating things we are more familiar with. For example, if you’ve ever rolled coins before, after a couple times, you become pretty good at grabbing a group of around 50 pennies to make a roll (adding or subtracting a few to make the roll complete)….but what do 100 gumballs look like? How many are really in that gumball machine at the car wash? In this case, this may be the first time someone actually saw 1337 gumballs.

In summary, there may be a variety of reasons that the gumball game fell short of proving the utility of the wisdom of crowds for estimation of uncertainty… because there was definitely wisdom in this crowd. (It was, after all, a decision quality oriented forum.) Interestingly, 50 of the 78 people had “means” lower than their estimates. Perhaps they already sensed that they were over-estimating the answer?

We don’t want to make any definitive conclusions because it was only one test; we’d like to run another couple of games and see if the results change.

Thanks to everyone who participated and congratulations to the winner (who has been sent an email announcing his sound estimating capabilities), and we will send you a note when another public estimating game ensues.

- Jeremy