We are gamblers, you and me. With our very first breath, we begin assessing our surroundings and playing the odds for survival and pleasure. We strive to amass data on what works to our benefit, and what doesn’t. Unexpected outcomes (aka “outliers”) might be pleasant surprises or not-so-pleasant disappointments – either way, most of us seek the middle ground, regressing toward the mean (aka “reversion to mediocrity”).
Although we place our trust in what will work based on our assembled knowledge and experience, this is not always wise. Often our well-honed intuition fails, leaving us dumbfounded. One quintessential example is the familiar Birthday Paradox.
If 23 or more people are gathered together – like in a classroom or at a cocktail party – the odds are better than 50-50 that two will have the same birthday. Given that there are 365 (not counting leap year) possible birthdays, how is this even possible? At a very large party, with 366 people in attendance, then at least 2 would share a birthday – that much is clear – but only with some clever math can we prove the aforementioned counterintuitive odds. See the link above for the math.
For nearly 30 years beginning in 1963, TV personality Monty Hall served as the host for the game show Let’s Make a Deal. In the eponymous Monty Hall Problem, contestants were faced with three closed doors, two of which concealed goats, and one which hid a new car. Compared with the Birthday Paradox, this test of intuition has a higher financial payoff (unless you gambled big on the birthday thing), while also serving as proof that additional information can confound the issue. The math seems simple enough – you have a 1/3 chance of choosing the door with the car. But there’s more...
The excitement builds once a door is chosen. Before opening it, Monty opens one of the other two, revealing a goat. He then offers the option of switching doors or staying with the original choice. At first glance, it would seem that with two doors remaining, one concealing a car, your chances are 50-50 no matter what you do. This is intuitive, but wrong. U.C. Berkeley Professor Steve Selvin solved the problem, known as a veridical paradox (a proposition that is in fact true despite its air of absurdity), in a 1975 letter to the journal American Statistician. The detailed explanation can be found here, but the bottom line is that there is a 1/3 chance your first choice was right, and thus a 2/3 chance your first choice was wrong. Monty’s little diversion has no effect on this – remember you made your choice based on 3 doors, not 2. Although counterintuitive, your odds are clearly better (66% vs 33%) if you switch doors.
Given that we are constantly gambling and reverting to mediocrity, you would think that anything that improves our outcomes would be welcome. The math that replaces our unreliable intuition in the two prior examples should be of great interest to anyone wishing to improve their odds. If only it weren’t the much-maligned subject of Probability and Statistics that we must rely on.
In spite of its reputation (as a student I was required to take a class we referred to as Improbability and Sadistics), there are some truly useful tools that can improve our chances and override our intuition when necessary. Consider the lesser-known story of Arthur Guinness.
On the last day of the year 1759, Arthur signed a lease on St. Jame’s Gate Brewery in Dublin, Ireland. The terms were a bit extreme - £45 a year for 9,000 years. He had learned the craft of brewing from his father Richard and was to become the first in a long line of Guinness Master Brewers. It goes without saying that Arthur had a confidence and a vision that set him apart from other entrepreneurs.
Having personally toured the Dublin brewery, I can attest to the scientific attention to details. Each tour ends with a pint of Guinness, which somehow tasted better than any I’ve had anywhere else in the world. Our barkeep informed us that the perfect pour lasts precisely 119.5 seconds – a clue to just how obsessed these guys are with numbers. Mathematician Michael Edward Ash, who became a brewer there, tinkered for years to perfect a chemical technique using nitrogen gas to create the velvety head for which Guinness is famous (“nitro brew” has since become a thing in coffee and other beers).
But the true influence of Guinness goes beyond foamy beers. After almost 150 years of brewing, they were by far the largest brewery in the world. Still, quality control was based largely on intuition. Eye-balling and smell testing were the foundation of the process. Global expansion demanded more, and they soon busied themselves seeking answers to the questions: what do our customers really want?, what is the perfect saccharine level in malt extract?, where do the best varieties of barley grow? ...
One particular issue was that of hop flowers, which act as a natural preservative and help impart the bitter flavor Guinness is known for. Soft resin content was a good metric, and a value of around 8% was known to be desirable. But you can’t measure every hop flower, and you don’t want a single outlier to bias your decisions. How can you know that you are checking a large enough sample for valid results? This question of statistical significance was not unique to beer brewing, but no one had considered the impact of small sample sizes until Guinness experimental brewer William Sealy Gosset invented the t-test, with which he solved many problems at the brewery. Bill published his results under the pseudonym “Student” not because students drink a lot of beer, but because he didn’t want to alert competitors to his research.
Mathematician John D. Cook blogged in 2008 that it was no surprise that the t-test originated at a brewery as opposed to a winery. Brewers are relentless in their quest for consistency, while winemakers tout their variety (vintage, terroir, harvest times, climate changes ...) – each bottle has its own unique story. The aphorism “If you can’t fix it, feature it” comes to mind.
In spite of its far-reaching impact, there is no mention of the Student t-test in the Guinness World Records book – perhaps there was concern about the appearance of nepotism.
Gamblers that we are, we owe a debt of gratitude to Bill Gosset because after all, no one wants to risk drinking an outlier.
Author Profile - Paul W. Smith - leader, educator, technologist, writer - has a lifelong interest in the countless ways that technology changes the course of our journey through life. In addition to being a regular contributor to NetworkDataPedia, he maintains the website Technology for the Journey and occasionally writes for Blogcritics. Paul has over 50 years of experience in research and advanced development for companies ranging from small startups to industry leaders. His other passion is teaching - he is a former Adjunct Professor of Mechanical Engineering at the Colorado School of Mines. Paul holds a doctorate in Applied Mechanics from the California Institute of Technology, as well as Bachelor’s and Master’s Degrees in Mechanical Engineering from the University of California, Santa Barbara.
