Noise: A Flaw in Human Judgment

An informational cascade is defined as the phenomenon that occurs during a decision-making process when events such as an influential person’s early involvement cause others in the team to copycat their stance on a particular issue. Although informational cascades produce an illusion of agreement, the authors view them as a form of noise, as a unanimous decision has been reached less from real agreement than from people subordinating their own judgment to that of a person they respect. Studies have shown that informational cascades are prevalent in multiple types of organizations, especially during complex decision making. This occurs because some people do not have a strong opinion on the matter being discussed, while others want to “stay in the group’s good graces” (124).

Informational cascades can become problematic when they stem the flow of debate and lead to poor decision-making. The authors, by way of the Good Decision Project, demonstrate that it is better to relinquish the illusion of early agreement in favor of independent assessment of a situation and debate. The aggregation of different perspectives, rather than the imposition of one perspective, produces better forecasting, especially in situations where the future is unknown—for example, in hiring practices.

The authors and this study guide use the word judge interchangeably in different ways. While the conventional definition of a public officer appointed to decide cases in a court of law applies, so does the idea of a judge as a decision-maker in any walk of life. Thus, in the sphere of employment, an interviewing panel that decides between candidates can also be said to be composed of judges.

The authors define level noise as the consistent variation in the disposition of different judges. For example, defense lawyers differentiate between harsher-than-average “hanging judges” and more lenient “bleeding-heart judges” (91). In the sphere of medicine, level noise might occur in the disposition of one doctor to insist that all patients over 60 have cancer screenings, and another who never does this. The authors term the disparity between the two consistent judgments “level errors” (91). Level noise can be quantified in legal judges when the mean of their sentences for a particular crime is taken and then measured against that of other judges. The authors argue that the variability “has nothing to do with justice” and instead reflects judges’ life experiences, backgrounds, and biases (91). There is also level noise between judges who think that the goal of justice is rehabilitation and those who think it is “deterrence or incapacitation” (92). As it is often random luck which judge a particular defendant will see, level noise perpetuates unfairness in the justice system. However, it can be measured and controlled more easily than other types of noise.

The authors define noise as “unwanted variability” in judgments (21). For example, while two people convicted of identical crimes would ideally receive identical sentences, they might not because the approach of two different judges or the same judge on a different day might vary. Noise is distinguished from bias, a form of preference that is “systematically off target”—for example, bias may be embodied by a consistently harsh judge. It might occur in the form of occasion noise where factors on the day in question influence decisions, or in the form of pattern noise where a judge has an idiosyncratic response to a particular type of case. The divergent sources of noise mean that only a statistical approach is needed to identify and measure it. While bias, particularly against specified demographics, earns more media attention, the authors argue that noise is an equally important source of error in judgment, and that taking steps to reduce it will inevitably result in a fairer world.

The authors define a noise audit as a study “designed to measure how much disagreement there is among professionals considering the same cases within an organization” (17). Noise audits are important because if noise is not observed and measured, then it cannot be addressed, and organizations will continue to make the same mistakes. The authors consider this so important that they provide an appendix on “how to conduct a noise audit” from the perspective of a consultant hired to examine the type of noise that occurs in the professional judgments of the organization (438). The noise audit involves a whole ensemble of characters, including clients, judges, and project managers.

While noise audits can be time-consuming and costly, the authors argue that they are worth the trouble, given the disruption that noise causes in organizations.

The authors define occasion noise as “the variability in judgments of the same case by the same person or group on different occasions” (17). Sources of occasion noise can be the weather, the time of day, or a local sports team’s performance on the weekend prior to the decision. Such factors, which are irrelevant to the case at hand, or the judge’s values and general disposition, can affect serious decisions. The authors show how the astonishing impact of occasion noise can influence many types of decisions, from the fact that patients are more likely to be prescribed cancer screenings in the morning to the fact that college admissions officers pay greater attention to academic virtues on cloudier days and non-academic ones on sunnier days. Overall, the presence of occasion noise highlights the fallibility of human judgments and the perils of not taking steps to minimize the impact of irrelevant factors in decision making.

Pattern noise is defined as the systematic deviation from a habitual pattern owing to the presence of mitigating factors. For example, in the law, a normally lenient judge may show above-average harshness when it comes to sentencing driving offenses. The official term for pattern noise is “judge-by-case” variability (94). These pattern errors have less to do with the case at hand than with the judges’ life experiences. The authors draw upon statistical evidence to show that patterns are “not mere chance: we would expect them to recur if the judge saw the same case again” (94). However, the unpredictability of pattern noise makes it harder to measure than level noise. Still, pattern noise is pervasive and contributes to disparities in sentencing at least as much as level noise.

However, in some cases, pattern noise can be more important than level noise in determining unwanted variability, as in Sendhil Mullainathan’s study of judges’ bail decisions where human judges were compared to noise-resistant algorithms. The study found that 79 percent of the variance in judges’ opinions was pattern noise, far exceeding the amount of level noise between them.

System 1 thinking is fast and intuitive. It jumps to conclusions and uses simple problem-solving techniques called heuristics that allow people to make quick judgments and function in day-to-day life without thinking too much. Daniel Kahneman described System 1 thinking in his 2011 book Thinking, Fast and Slow, where he differentiated it from slow, more deliberative System 2 thinking.

In Noise, System 1 thinking is shown to be deeply susceptible to noise and therefore the enemy of good judgment. It is often accompanied by conclusion bias or prejudgment, when a person fixates on the outcome they would most like and makes decisions accordingly.

System 1 thinking is also used when individuals simplify difficult questions by substituting them with much easier ones. For example, if a reader used System 1 thinking when the authors challenged them to determine Julie’s college GPA from the age at which she learned to read, the reader would have asked themselves “how impressive was Julie’s achievement as a four-year-old reader?” (211). They would have then tried to match an appropriate GPA to their impression.