Are Randomness and Uncertainty fundamental and pervasive?
Posted on 20 April 2011 by cjf
The view that randomness impacts and shapes our lives in profound ways has been gaining traction since 2002 when Daniel Kahneman won the Nobel prize in Economics for his work with Amos Tversky in characterizing human weaknesses when facing uncertainty. My thinking on the subject was first awakened by reading Nassim Nicholas Taleb‘s book Fooled by Randomness which will give anyone who imagines they can think “rationally” a healthy dose of humble pie. A more helpful discussion can be found in Jonah Lehrer‘s How We Decide which pays heed to our brain’s strengths while acknowledging our weaknesses. As I relayed in a post on the brain, mind and thinking, Lehrer recommends thinking about your thinking process to strengthen its decision-making function. Recently I finished reading Leonard Mlodinow‘s The Drunkard’s Walk: How Randomness Rules our Lives which provides an accessible, historically detailed, and elementary introduction to the sciences of randomness and uncertainty and shows how they rule our lives.
These books have started to change my thinking about the nature of reality itself: I see now that randomness and uncertainty have an essential role to play. Interestingly, I shunned probability and statistics, the sciences of randomness and uncertainty, in college because I was steeped in Euclid, logic, and Buckminster Fuller‘s “generalized principles” in Synergetics. I wanted to design destiny with deliberate application of knowledge … to worship at the altar of scientific determinism. Fortunately, Bucky taught me to “dare to be naïve” so I have been open to the new evidence about randomness. Now I suspect that Bucky and I were a little off about this subtle subject. It isn’t surprising, probability and statistics are among the newer branches of mathematics having developed mostly after the calculus was well established. They have not had enough time to pervade our collective consciousness.
Do you think the world is fundamentally deterministic or random? What influences have shaped your thinking and biases about the subjects of randomness, uncertainty, probability, and statistics? Do you think the increasing focus on the role of randomness and uncertainty in our lives is an important trend?
Randomness Rules Our Lives
Is Mlodinow’s thesis that randomness rules our lives really so convincing? Evidently so. Mlodinow finds dramatic evidence of randomness in our economic lives. He retells the poignant story of Sherry Lansing who led Paramount Pictures to huge successes in seven consecutive phenomenal years. Then after three years of bad results, she left the company. Did Paramount let her go too quickly? Evidently so because the pipline she left behind was full of new hits that restored Paramount’s revenue and market share. Shouldn’t seven years of success earn the right to forgive a few bad years? What if another great leader happened to have their three consecutive bad years at the beginning of their tenure? Do we replace them before their ship comes in? Mlodinow cites many other examples including the fact that “And to Think That I Saw It on Mulberry Street” was rejected by publishers some 27 times before Dr. Seuss’ career launched. Mlodinow also shows that student grades are often random and independent of their skill and knowledge.
Should we insist that our students, our schools, and our business leaders perform, perform, and perform with no “bad” years allowed? Do you believe that performance results are somewhat random? We invest a lot in exam and executive performance. Given the evidence, is that wise?
One part of Kahneman’s Nobel-prize winning work addressed the conjunction fallacy. Let A, B, and C be statements represented by a colored circle in the venn diagram to the right. The only case in which they can be simultaneously true is in the small area where all three colors overlap. So it is much less likely (less area) for three statements to be simultaneously true than for any one of them to be true. However, when someone weaves a story filled with a lot of concrete details, it seems more vivid and hence more believable than the statements considered separately: that’s the conjunction fallacy. Evidence of people falling for this fallacy has been documented widely even in medicine and the court room. We humans are easily duped by a good story!
It is surprising that the Nobel prize for the work showing how “blind” humans are to the elementary logic of the conjunction fallacy was only awarded one decade ago! Humanity has only just yesterday identified this basic weakness in our cognitive function! Add to the conjunction fallacy the many other fallacies and biases that Taleb, Lehrer, and Mlodinow show us to be subject to and one can see that Emanuel Lasker who was world chess champion for 27 years got it right: “In life we are all duffers”!
What is the significance of our weakness in understanding uncertainty? Do these weaknesses of the human mind subject us to the ravages of randomness? Are they a consequence of an inherent randomness in reality? Or do they simply lead to the appearance of randomness?
Our weakness extends to our sensory organs and perception as well. Mlodinow notes
Human perception … is not a direct consequence of reality but rather an act of imagination. Perception requires imagination because the data people encounter in their lives are never complete and always equivocal.
Mlodinow illustrates the problem by explaining that the human visual system sends “the brain a shaky, badly pixelated picture with a hole in it” (due to the relative weakness of our vision outside the fovea and the blind spot). In addition to conjunction bias, the sharp shooter effect, the hot-hand fallacy, availability bias, confirmation bias, and more, it becomes evident that “When we look closely, we find that many of the assumptions of modern society are based … on shared illusions.” And his conclusion
It is important in our own lives to take the long view and understand that streaks and other patterns that don’t appear random can indeed happen by pure chance. It is also important, when assessing others, to recognize that among a large group of people it would be very odd if one of them didn’t experience a long streak of successes or failures.
What shared illusions do we hold? How often are our lives subject to pure chance events? How important is serendipity? Do you believe that a long series of failures or successes is just the result of luck? When is it luck and when is it skill? How can we tell the difference?
The problem of randomness is deeper still: even machine-enhanced human sensing and measurement are fundamentally random! In Walter Lewin’s excellent video introducing physics and measurement in MIT OCW’s Physics I course, he says “Any measurement that you make without any knowledge of the uncertainty is meaningless.” Understanding uncertainty is at the heart of scientific measurement. No physics experiment ever found an exact match between theory and the laws of nature: data points always appear at random! Then add in effects like Heisenberg’s uncertainty principle and we see that randomness and uncertainty are vital elements of experience: they are pervasive.
In view of the elementary role of uncertainty in our perceptual and physical experience, what can we say about reality? What is reality if experience is so imprecise, fuzzy, uncertain, and fallible?
In chapter III of D’Arcy Wentworth Thompson’s great magnum opus On Growth and Form many of the issues involved with the law of error are eloquently discussed. Mlodinow’s chapter 7 covers the same material at a more introductory level and with a wider range of vivid examples. But the two authors reach different conclusions. Thomspon says what I perceive to be society’s orthodoxy:
Buckminster Fuller seems to agree with Thompson that randomness is an illusion:
When further meticulously studied and magnified, this superficial seeming randomness proves to be our flying squadrons [...] enjoying a vast number of intricately orderly team maneuvers but with never a pilot in sight. The whole is flown by remote control with fantastic feedback and local automation, all governed by an eternally complex integrity of complementary, interaccommodative principles.
The famous quote “God does not play dice” (a paraphrase of an actual Einstein quote) reinforces that some of our greatest scientists think of the world as essentially deterministic.
In contrast, Mlodinow asserts
the triumph of a great principle: that much of the order we perceive in nature belies an invisible underlying disorder and hence can be understood only through the rules of randomness.
Inspired in part by the work of Kahnemann and Tversky and books like The Drunkard’s Walk, society is, it seems to me, undergoing a deep philosophical transformation as the dawning awareness that randomness is fundamental seeps into our consciousness. In addition to the books by Taleb, Lehrer, and Mlodinow, witness that the importance of randomness is asserted in many of the responses to The Edge‘s Question 2011: What Scientific Concept Would Improve Everybody’s Cognitive Toolkit. In particular, see these six short pieces: Uncertainty (Lawrence Krauss), Randomness (Charles Seife), Possibility Spaces (W. Daniel Hillis), Probability Distributions (John Allen Paulos), The World is Unpredictable (Rudy Rucker), and The Uselessness of Certainty (Carlo Rovelli).
What do you think? Is experience basically random? Uncertain? Deterministic? Both? Neither? How do you parse experience?
Exploring the relationships between reality and randomness
Reality it seems to me is concrete experience. Experience is “what happened” in contradistinction to our story or interpretation about what happened. But we are subject to forgetfulness, “invented” recollections, and our “creative” imagination may even get the concrete details wrong (how would we know?). Illusions and delusions fool us. Even when we are extraordinarily careful about observing data as in scientific measurement, each datum varies noticeably and randomly from the next. If raw experience is inundated with randomness and uncertainty, it is evidently essential. Mlodinow’s thesis that randomness rules our lives seems clearly justified!
Buckminster Fuller often pointed out that from time-to-time our minds can find patterns that are common in all experience. Then we say we have found a “truth” and in time it may even be recognized as a generalized principle. We are pretty confident about these scientific truths: they work reliably in experiment and in engineering practice. They make sense and they interaccommodate. They are represented by exquisite mathematics. They have been thoroughly vetted in a vigorous debate with alternative theories. From this perspective, randomness and uncertainty are seen as “challenges with messy data” that need to be overcome to reveal the order in nature. Indeed, if a datum differs from what a generalized principle would suggest, we examine our measurement to find the cause of the error. That is, we trust the science and question the errant data. Universe, it seems, rigorously follows the laws of nature … it is deterministic!
But science depends on the tools of statistics to determine the validity of its data and to develop incisive understanding of the interrelationships in the variables under study. Statistics is the tool through which scientists parse experience to find the order in the chaos. Probability and statistics are the technology used to overcome the inherent biases that blind us. They quantify uncertainty and show again that randomness is fundamental in the sciences. It is pivotal in the framework used to identify the mathematical relations that form the laws of nature.
Might the assumption of uncertainty simply be an analytical tool and not the essence of ontology (the nature of being and existence)? Or is uncertainty at the heart of the matter and “determinism” merely the rare statistical relationship with a 95% confidence interval?
Perhaps, both perspectives are illusions induced by the mathematical framework used to parse reality: probability theory vs. deductive reasoning. Reality could be simply the interrelationships in experience per se. The ontological distinction might simply be a story … an interpretation … philosophical sugar. Perhaps the uncertainty-determinism duality is just another example of the fundamental both-neither-ness inherent in Universe? [Credit to Tom Miller for introducing me to the concept of "both neither" at the Synergetics Collaborative's third Summer Workshop at SUNY Oswego in July 2005.]
How do you interpret the “tension” between uncertainty and determinism? What ontology do you favor? Do you lean toward determinism or uncertainty or both-neither-ness or something else? Why?
Some OER (Open Educational Resources) for learning more about probability and statistics
Mlodinow’s excellent story-telling approach makes The Drunkard’s Walk an easy and entertaining way to learn the story of randomness. If you want to go into more depth by studying on-line, I recommend the introductory Probability and Statistics course at Carnegie Mellon’s OLI (Open Learning Initiative). The course is self-paced and includes exercises to test out and practice using the concepts to build intuition and facility in using the material. The OLI Statistics course supports solving problems on the computer using several statistical packages including my favorite, the R project for statistical computing. Like most statistics resources that I have seen, the course explains how to “do statistics” and sometimes fails to give justifications for the methods. The course lacks video lectures and requires free registration.
Video courses are better at relaying context for the concepts and for surveying a field. Building skills or striving for mastery of the subject requires more disciplined practice of the material by either working the exercises or building an application of personal interest (a method that is currently underutilized in formal education). I found several video courses available (including ones at UCLA, Stanford, Berkeley, Iowa State, and IIT Kharagpur). Of course, Khan Academy is great for supplementing the understanding of specific concepts). The video course Sets, Counting, and Probability at Harvard looks very promising (I hope to find time to check it out … eventually — for now I’m working to finish the OLI course).
Do you know of any other good OER materials on probability and statistics? Can anyone review one of the video courses?