Noise: A flaw in human judgment by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein

Noise: A Flaw in Human Judgment by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein argues that wherever human judgment operates, there is noise, and far more of it than organizations believe. The authors define noise as unwanted variability in judgments that should ideally be identical, proving through the Mean Squared Error equation ( MSE = Bias² + Noise² ) that random scatter degrades accuracy just as much as systematic directional bias. Drawing on empirical studies across federal criminal sentencing, commercial insurance underwriting, medical diagnosis, forensic science, corporate hiring, and geopolitical forecasting, the book decomposes system noise into level noise, stable pattern noise, and transient occasion noise. Rather than relying on intuition or post-hoc debiasing, Kahneman, Sibony, and Sunstein urge leaders to quantify internal disagreement through formal noise audits and adopt preventative decision hygiene protocols, including independent aggregation, linear sequential unmasking, case-anchored comparative scales, and the Mediating Assessments Protocol.

Noise: A flaw in human judgment by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein

Who this book is for

Editorial analysis of target readers and domain limits based on the scope of Noise.

Read it if

  • You lead or design decision systems in underwriting, law, medicine, hiring, or strategy where interchangeable professionals evaluate comparable cases.
  • You want statistical methods to audit judgment scatter across an organization even when objective ground truth cannot be known for years.
  • You chair committees or boards and need structured decision hygiene routines to prevent informational cascades, halo effects, and group polarization.

You can skip it if

  • You focus exclusively on venture investing, proprietary market trading, or creative discovery where contrarian bet variation is desirable diversity rather than system error.
  • Your evaluations concern subjective aesthetics, artistic critique, or personal taste where bounded agreement is neither expected nor constructive.
Editorial analysisReader fit and boundary conditions for Noise: A Flaw in Human Judgment.

Chapter-by-chapter map of Noise

Comprehensive map of all 28 numbered chapters in Noise: A Flaw in Human Judgment.

  1. Chapter 1: Crime and Noisy Punishment

    Unconstrained judicial discretion turns criminal sentencing into a lottery where identical offenses receive starkly different prison terms.

    Judge Marvin Frankel exposed severe disparities in federal sentencing, prompting the Sentencing Reform Act of 1984 and mandatory sentencing guidelines. Controlled audits and field data confirmed that mandatory rules cut interjudge scatter, whereas the 2005 Supreme Court decision rendering guidelines advisory caused disparity to double again.

    Example: Two first-time offenders cashed counterfeit checks for $58.40 and $35.20; one judge imposed 15 years in prison while another imposed 30 days.

  2. Chapter 2: A Noisy System

    System noise silently drains corporate profits because executives operate under a false illusion of professional agreement.

    A formal noise audit inside a large financial and insurance enterprise tested underwriters and claims adjusters on identical case files. Leaders predicted peer disagreements would stay near 10%, yet the audit uncovered a 55% median difference in underwriting quotes and a 43% difference in claims payouts, showing that independent pricing errors add up rather than cancel out.

    Example: On an identical commercial risk file, one qualified underwriter quoted a premium of $9,500 while a colleague in the same office quoted $16,700.

  3. Chapter 3: Singular Decisions

    One-of-a-kind strategic choices are subject to noise and should be analyzed as recurrent decisions that occur only once.

    Unique, non-routine decisions cannot be repeated empirically to calculate a standard deviation, yet they are drawn from an internal cloud of possibilities. Counterfactual reasoning shows that changes in meeting sequence, advisor framing, or transient mood could easily shift a leader toward a different path, requiring the same procedural discipline used for routine choices.

    Example: President Barack Obama decided in 2014 to deploy 3,000 troops and health personnel to West Africa to contain Ebola at its source rather than closing US borders.

  4. Chapter 4: Matters of Judgment

    Judgment is a measurement exercise conducted by the human mind under an expectation of bounded disagreement and guided by an internal signal of coherence.

    Matters of judgment sit between deterministic computation and subjective taste, encompassing both verifiable predictive judgments and unverifiable evaluative judgments. When evaluators weave conflicting cues into a story, the brain produces a rewarding internal signal of completion that feels identical whether the prediction is valid or entirely fabricated.

    Example: When 115 MBA students estimated the two-year survival probability of fictitious CEO candidate Michael Gambardi, their confident answers ranged from 10% to 95%.

  5. Chapter 5: Measuring Error

    In the Mean Squared Error equation, bias and noise play orthogonal, additive, and mathematically equal roles in total judgment error.

    Evaluating predictive accuracy across an ensemble of forecasts demonstrates that overall error equals the square of average directional bias plus the square of noise variance. Because the two terms sum independently, shrinking judgment scatter by a given magnitude lowers total error just as effectively as shrinking bias by the same amount.

    Example: At the hypothetical retail firm GoodSell, regional sales forecasts averaged 44% against a true market share of 34%, combining a 10% optimistic bias with a 10% standard deviation of noise.

  6. Chapter 6: The Analysis of Noise

    System noise decomposes mathematically into level noise across judges' average baselines and pattern noise from judge-by-case interactions.

    Re-examining the 1981 INSLAW federal sentencing dataset reveals that total system variance equals the sum of squared level noise and squared pattern noise. Level noise reflects consistent differences in general severity or leniency, whereas pattern noise reflects how individual judges reverse case rankings due to idiosyncratic weighting of case facts.

    Example: Across 208 federal judges evaluating 16 criminal cases, the 3.4-year system noise standard deviation split equally into 2.4 years of level noise and 2.4 years of pattern noise.

  7. Chapter 7: Occasion Noise

    The same judge evaluating the exact same case at different times produces variable judgments driven by mood, fatigue, weather, and case sequence.

    Within-person variability occurs because human responses are sampled from an internal probability distribution influenced by transient physiological and environmental states. While occasion noise is smaller than stable interpersonal differences, individuals can dampen it by averaging two separate estimates made over time to tap the crowd within.

    Example: US immigration judges became 19% less likely to grant asylum when they had approved the two immediately preceding applications on their docket.

  8. Chapter 8: How Groups Amplify Noise

    Unstructured group deliberation destroys independent judgment and increases system noise across committees through informational cascades and group polarization.

    While averaging independent individual estimates cuts noise sharply, open face-to-face discussion allows early or vocal speakers to steer the group. Social influence triggers cascades and pushes deliberating groups toward more extreme positions than their members held individually, creating wide divergence between separate committees.

    Example: In Matthew Salganik's Music Lab experiment across eight parallel social-influence worlds, the song Best Mistakes became a top hit in one world and flopped in another due to early download cascades.

  9. Chapter 9: Judgments and Models

    Simple statistical models and actuarial formulas consistently out-predict human experts primarily because mechanical rules are completely free of noise.

    Decades of research initiated by Paul Meehl show that mechanical combinations of predictors beat clinical intuition across psychology, medicine, and hiring. Lewis Goldberg proved that a linear regression model trained on a judge's own past ratings predicts real outcomes more accurately than the judge does, because the human expert adds pattern and occasion scatter.

    Example: In Martin Yu and Nathan Kuncel's analysis of 847 executive assessments, 10,000 randomly weighted linear models outperformed human HR evaluators across two validation samples.

  10. Chapter 10: Noiseless Rules

    Equal-weight linear formulas and frugal rules match or beat complex regressions on small samples and decisively surpass human clinical judgment.

    Robyn Dawes demonstrated that standardizing predictors and adding them with equal unit weights avoids overfitting sample flukes while eliminating human inconsistency. Clinicians frequently override rules by claiming rare broken-leg exceptions, yet empirical checks show that most human overrides introduce random scatter rather than valid insight.

    Example: An unweighted sum of seven candidate ratings predicted executive job performance with a correlation of .25, compared to .15 for holistic expert clinical evaluations.

  11. Chapter 11: Objective Ignorance

    The primary ceiling on predictive validity is objective ignorance, which arises from irreducible uncertainty and unknowable future events.

    Professionals often confuse their internal feeling of narrative coherence with genuine foresight, falling into a denial of ignorance. Even noiseless algorithms and sophisticated machine-learning systems achieve only modest predictive correlations in complex human domains because critical future determinants simply do not exist at the moment of prediction.

    Example: Philip Tetlock's 20-year study of roughly 300 geopolitical pundits found that long-term expert forecasts were barely as accurate as a dart-throwing chimpanzee.

  12. Chapter 12: The Valley of the Normal

    Backward-looking causal thinking makes unanticipated events feel inevitable in hindsight, concealing both objective ignorance and system noise.

    Most human events fall into the valley of the normal, where outcomes cannot be predicted reliably in advance but are easily rationalized after the fact. Because System 1 automatically searches memory for a plausible agent and cause once an outcome is known, observers attribute failures to specific mistakes and remain blind to statistical scatter.

    Example: In the Fragile Families Challenge, hundreds of data-science teams using thousands of variables on 4,242 families reached a correlation of only .22 for predicting material hardship at age 15.

  13. Chapter 13: Heuristics, Biases, and Noise

    Cognitive heuristics generate statistical bias when shared uniformly across evaluators, but produce system noise when judges vary in their personal biases or cue order.

    Fast intuitive thinking relies on substitution, conclusion bias, and excessive coherence to simplify complex problems into tidy stories. When different professionals hold different initial leanings or encounter case evidence in different sequences, these universal psychological mechanisms push their final conclusions in opposite directions.

    Example: Solomon Asch showed that describing a person as Intelligent, Persistent, Cunning, Unprincipled yields a favorable impression, whereas reversing the trait order produces harsh distrust.

  14. Chapter 14: The Matching Operation

    Intuitive forecasting relies on matching subjective intensity across scales, producing non-regressive predictions and widespread pattern noise.

    System 1 translates an impression of impressiveness or severity directly onto a target outcome scale without accounting for base rates or weak correlation validity. Furthermore, human absolute discrimination along continuous dimensions breaks down beyond roughly seven categories, making unanchored absolute ratings far noisier than comparative rankings.

    Example: Told only that Julie read fluently at age four, evaluators match her childhood precocity directly to an elite 3.7 or 3.8 college GPA, ignoring regression to the class mean.

  15. Chapter 15: Scales

    Unbounded numerical response scales turn shared moral consensus into severe level and pattern noise.

    When evaluators translate a shared qualitative feeling onto an unbounded ratio scale like dollars without an explicit anchor or modulus, each person chooses an arbitrary baseline. Although subsequent relative adjustments remain internally proportional, inter-judge dispersion explodes, proving that bad measurement scales generate unnecessary noise.

    Example: In the Joan Glover defective pill-cap study of 899 mock jurors, moral outrage correlated at .98 across case means, yet punitive damage dollar awards consisted of 94% noise.

  16. Chapter 16: Patterns

    Complex cases with conflicting cues force evaluators to construct idiosyncratic causal narratives, creating stable pattern noise.

    When a file contains contradictory evidence, simple intensity matching fails and judges must selectively weight and combine cues to achieve coherence. Drawing on Walter Mischel's personality research, the chapter shows that each evaluator's unique life history and values create stable, situation-specific judgment signatures that scramble case rankings.

    Example: In the expanded Julie 2.0 vignette featuring high ambition, a sick father, past drug use, and mixed grades, evaluators form sharply divergent stories and GPA predictions.

  17. Chapter 17: The Sources of Noise

    Stable pattern noise is the largest component of system noise, exceeding both level noise and transient occasion noise across professional domains.

    Synthesizing variance decompositions across judicial, financial, and experimental audits demonstrates that idiosyncratic judge-by-case interactions typically drive 60% to 80% of system noise. Because causal thinking focuses on individual case stories while noise is a statistical property of ensembles, this dominant error source stays hidden without formal audits.

    Example: In Jon Kleinberg's machine-learning analysis of 173 bail judges and 141,833 defendants, stable pattern noise accounted for 26% of total decision variance compared to 7% for level noise.

  18. Chapter 18: Better Judges for Better Judgments

    The best human judges combine general mental ability and verified domain training with an actively open-minded cognitive style.

    In unverifiable fields, institutions frequently confuse peer prestige and high self-confidence with genuine predictive skill, elevating respect-experts. Empirical testing shows that superior judgment tracks cognitive ability, reflective restraint, and actively open-minded thinking, which is the willingness to seek disconfirming evidence and revise beliefs.

    Example: Jonathan Baron's actively open-minded thinking scale consistently predicted forecasting accuracy by measuring resistance to premature closure and belief persistence.

  19. Chapter 19: Debiasing and Decision Hygiene

    When specific directional biases cannot be diagnosed in advance, organizations must deploy preventative decision hygiene and real-time decision observers.

    Ex post mathematical corrections and ex ante nudges work against predictable directional skews, but cannot resolve conflicting psychological biases that scatter individual judgments. Appointing a neutral decision observer with a concise diagnostic checklist and enforcing procedural decision hygiene prevents unknown errors before commitments harden.

    Example: The US Office of Management and Budget's 50-page Circular A-4 regulatory guide went largely ignored until it was distilled into an actionable 1.5-page verification checklist.

  20. Chapter 20: Sequencing Information in Forensic Science

    Controlling the order of evidence through linear sequential unmasking shields forensic experts from contextual confirmation cascades.

    Even veteran fingerprint and DNA analysts reverse their conclusions when exposed to detective hunches, suspect confessions, or peer validations early in an investigation. Linear sequential unmasking forces examiners to analyze and lock in their documentation of crime-scene traces before viewing reference exemplars or background case details.

    Example: Three FBI fingerprint examiners and a court-appointed expert falsely matched Oregon attorney Brandon Mayfield to the 2004 Madrid train bombings after contextual cues triggered a bias cascade.

  21. Chapter 21: Selection and Aggregation in Forecasting

    Selecting open-minded forecasters in perpetual beta and mechanically aggregating their independent probability estimates dramatically reduces predictive noise.

    In the Good Judgment Project, top-performing superforecasters decomposed problems, anchored on outside-view base rates, and updated probabilities incrementally as weak signals arrived. Statistical decomposition via the BIN model proves that training, structured teaming, and aggregation improve accuracy primarily by stripping out noise rather than bias.

    Example: The top 2% of participants in the Good Judgment Project outperformed professional intelligence analysts who had access to classified material.

  22. Chapter 22: Guidelines in Medicine

    Clinical guidelines curb diagnostic noise by breaking holistic medical impressions into standardized, observable sub-assessments.

    Medical fields from radiology and oncology to psychiatry suffer from wide inter-clinician and intra-clinician disagreement. Effective clinical guidelines and checklists constrain discretion by specifying which empirically validated cues to inspect, anchoring each on a bounded scale, and summing the ratings mechanically.

    Example: Dr. Virginia Apgar's 1953 Apgar score standardized newborn viability assessment across five 0-to-2 clinical ratings at one and five minutes after birth, sharply cutting infant mortality.

  23. Chapter 23: Defining the Scale in Performance Ratings

    Corporate performance appraisals are dominated by system noise, which can be reduced through shared case scales and frame-of-reference training.

    Studies of 360-degree workplace evaluations reveal that 70% to 80% of rating variance stems from evaluator level and pattern noise rather than actual employee output. While forced ranking eliminates level noise at a heavy cost to morale, frame-of-reference training and behaviorally anchored case scales calibrate managers to concrete shared examples.

    Example: A case-anchored rating scale calibrates managers by defining score 2 as performing like benchmark employee Dan P. and score 4 as performing like Jenny P.

  24. Chapter 24: Structure in Hiring

    Replacing unstructured interviews with decomposed assessments, independent interviewers, and delayed holistic intuition eliminates hiring noise.

    Traditional unstructured job interviews allow rapport and first impressions formed in the opening minutes to trigger excessive coherence and halo effects. Structured hiring decomposes the role into separate mediating traits, requires interviewers to score rubric-anchored questions and work samples independently, and leaves final synthesis to a hiring committee.

    Example: Google replaced free-form interviews with structured rubrics across general cognitive ability, leadership, cultural fit, and role knowledge, forbidding interviewers from conferring before submitting scores.

  25. Chapter 25: The Mediating Assessments Protocol

    High-stakes strategic options should be evaluated like job candidates through decomposed, independent mediating assessments and delayed intuition.

    The Mediating Assessments Protocol structures executive board decisions by defining key evaluation dimensions upfront, isolating fact-finding teams, and anchoring analyses in outside-view reference classes. In decision meetings, members score each assessment sequentially using the Estimate-Talk-Estimate routine and delay holistic judgment until all dimensions are locked.

    Example: Retail CEO Joan Miller and analyst Jeff Schneider use the Mediating Assessments Protocol to guide a divided board through the multi-billion-dollar acquisition of Mapco.

  26. Chapter 26: The Costs of Noise Reduction

    Because eliminating noise entirely can introduce administrative expense, crude rigidity, and algorithmic bias, the optimal level of noise is not zero.

    Drawing on Albert Hirschman's rhetoric of reaction, the chapter examines how poorly designed rules can backfire, provoke gaming, or jeopardize fairness. When mechanical rules or algorithms perpetuate historical discrimination, leaders should neither abandon noise reduction nor tolerate human scatter, but audit and redesign the decision tools.

    Example: Mandatory three-strikes criminal sentencing laws eliminated judicial disparity at the perverse cost of imposing life imprisonment for minor third offenses.

  27. Chapter 27: Dignity

    Individualized hearings respect human agency and dignity, but cannot justify arbitrary lotteries when fundamental rights are at stake.

    Critics of rules argue that human dignity requires treating each person as a unique individual and allowing discretion to accommodate evolving moral norms. The authors reply that relying on noise to drive social progress subjects identical citizens to unequal fates, arguing that moral evolution belongs in transparent revisions of explicit policies.

    Example: In Cleveland Board of Education v. LaFleur, the US Supreme Court struck down a rigid rule forcing pregnant teachers onto leave at five months, requiring individualized fitness reviews.

  28. Chapter 28: Rules or Standards?

    Institutions must choose between ex ante rules and ex post standards by weighing upfront drafting costs against downstream decision and error costs.

    Rules eliminate discretion in advance and suppress system noise, whereas standards delegate judgment to the point of application and invite wide variability. High-volume, recurrent decisions warrant clear rules, algorithms, or structured decision hygiene, while rare or novel dilemmas without social consensus require standards disciplined by hygiene.

    Example: As analyzed by Jerry Mashaw, the US Social Security Administration adopted a mechanical disability determination matrix to deliver bureaucratic justice and end ad hoc administrative noise.

Core claims, arguments, and original examples across Parts I through VI.

Bias vs noise: The hidden geometry of flawed human judgment

Flawed human judgment stems from two distinct, mathematically equal sources of error: statistical bias, which shifts estimates systematically in one direction, and noise, which scatters estimates randomly around a target. While cognitive psychology and organizational reforms focus almost exclusively on bias, Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein demonstrate that understanding any noise flaw human evaluators introduce is equally urgent for institutional accuracy and fairness. Whenever multiple qualified professionals evaluate the same case and reach wildly different conclusions, the system suffers from unwanted variability.

To isolate where this variability occurs, the authors define a judgment as an act of measurement in which the human mind is the measuring instrument. Matters of judgment occupy the middle ground between deterministic computation, where exact mathematical rules require complete agreement, and matters of taste, where unbounded personal preference is expected. Every professional judgment carries an expectation of bounded disagreement. Within this boundary, organizations make two types of decisions: predictive judgments, which forecast a verifiable future event such as loan default or regional sales and are evaluated by empirical accuracy, and evaluative judgments, which assign a penalty or value without an objective future truth to verify them, such as a prison term or an insurance settlement, and are evaluated by procedural fairness and consistency.

Across both predictive and evaluative tasks, evaluators mistake their subjective certainty for objective validity because of an internal signal of completion. When a decision maker reviews incomplete or conflicting cues and weaves them into a coherent story, the mind delivers an emotionally rewarding feeling of cognitive ease. That internal signal operates identically whether the judge is forecasting a verifiable weather system or evaluating a fabrication. In the book's experiment involving a fictitious executive candidate named Michael Gambardi, 115 MBA students read a short biography and estimated the probability that he would succeed as CEO after two years. Every participant experienced a coherent internal impression, yet their probability estimates scattered across the scale from 10% to 95%.

Key terms defined in Noise

Definitions and original book examples for technical concepts coined or formalized by Kahneman, Sibony, and Sunstein.

TermMeaning in the bookExample
System NoiseUnwanted variability in judgments of the same problem by interchangeable professionals within an organization who are expected to agree.Underwriters in the same insurance company quoting premiums that differ by a median of 55% on identical commercial risk files.
Level NoiseVariability in the average baseline severity, leniency, or optimism across different judges evaluating a set of cases.In the INSLAW sentencing audit of 208 federal judges, the standard deviation of judges' average prison sentences across 16 cases was 2.4 years.
Pattern NoiseVariability in judges' idiosyncratic responses to specific cases, corresponding statistically to the judge-by-case interaction term.A judge who is generally lenient on violent offenses but imposes unusually harsh sentences on white-collar fraud defendants, reversing peer case rankings.
Stable Pattern NoiseThe permanent, predictable component of pattern noise driven by an evaluator's unique values, life experiences, and cognitive weighting of cues.Machine-learning models of 173 bail judges showed that individual judge-by-defendant interaction patterns accounted for 26% of total decision variance.
Occasion NoiseTransient within-person variability that causes the exact same judge to reach different conclusions on the same case at different times.Latent fingerprint examiners reversing 10% of their match decisions when secretly re-tested on the exact same print pairs seven months later.
The Crowd WithinThe statistical reduction in occasion noise achieved when a single person makes two delayed, independent estimates on the same question and averages them.Averaging two estimates spaced three weeks apart on the percentage of world airports located in the United States captured one-third of the accuracy gain of polling a second person.
The Valley of the NormalThe psychological zone where events that could not be predicted in advance are effortlessly rationalized in hindsight as normal and self-explanatory.A social worker constructing an equally convincing causal story for why the Jones family was evicted or why they kept their apartment.
Decision HygienePreventative procedural rules that reduce system noise and unspecified cognitive biases before a judgment forms, analogous to handwashing in medicine.Forbidding job interviewers from discussing a candidate until each interviewer has independently scored their assigned trait rubric.
Linear Sequential Unmasking (LSU)An information-sequencing protocol requiring analysts to document trace evidence completely before viewing comparison exemplars or contextual case facts.Requiring forensic examiners to map ridge minutiae on a crime-scene print before seeing the suspect's reference print or police file.
Mediating Assessments Protocol (MAP)A strategic decision framework that decomposes complex options into independent, outside-view assessments and delays holistic intuition until all ratings are complete.CEO Joan Miller breaking the proposed Mapco acquisition into isolated assessments of market growth, technology, and retention before board deliberation.
Estimate-Talk-EstimateA group aggregation process in which members submit independent private ratings, view the group dispersion, discuss differences, and re-vote privately.Board members scoring a target firm's integration risk silently on tablets before debating why two directors rated the risk far higher than the median.
Case ScaleA comparative rating instrument whose scale points are anchored to concrete, widely understood reference cases rather than vague adjectives or unanchored numbers.Anchoring an employee customer-relations appraisal scale using past employees Dan P. as level 2 and Jenny P. as level 4.
Core vocabulary for analyzing judgment error and implementing decision hygiene.

Executives often defend judgment variability either by confusing system noise with wanted diversity or by assuming random errors cancel each other out. Wanted diversity is an asset in creative brainstorming, venture investing, or competitive markets where independent actors bet on contrarian hypotheses. System noise, by contrast, occurs when interchangeable agents acting on behalf of one institution treat identical cases differently. Also, random errors only cancel out when multiple independent measurements are averaged on the exact same unit. When one underwriter underprices Policy A by 30% and a colleague overprices Policy B by 30%, the average error across the two quotes is zero, yet the insurance company loses money on the underpriced risk and loses the customer to a competitor on the overpriced quote. As Kahneman, Sibony, and Sunstein state, "In noisy systems, errors do not cancel out. They add up."

Even one-of-a-kind choices suffer from this hidden scatter. In Chapter 3, the authors examine singular decisions, such as President Barack Obama's 2014 choice to send 3,000 health workers and military personnel to West Africa to fight Ebola rather than shutting US borders. Leaders typically view singular crises as intuitive calls that sit outside statistical quality control. The authors counter that "A singular decision is a recurrent decision that happens only once." Because any single decision is sampled from an internal probability distribution that shifts with fatigue, meeting sequence, and advisor framing, disciplining a singular judgment requires the same noise-reduction procedures used for thousands of routine underwriting or sentencing files.

The table below contrasts statistical bias, system noise, and wanted diversity across their definitions, mathematical effects, ranking impacts, and institutional consequences.

DimensionStatistical biasSystem noiseWanted diversity
Core definitionSystematic directional deviation of the mean judgment from the true valueUnwanted variability across interchangeable judges evaluating the same caseDesirable variation in creative ideas, subjective taste, or competitive market bets
Role in total errorContributes `Bias²` to Mean Squared ErrorContributes `Noise²` to Mean Squared ErrorDoes not represent measurement error
Effect on case rankingsShifts the entire distribution up or down while preserving relative case ranksShifts baselines via level noise and scrambles relative case ranks via pattern noiseGenerates alternative hypotheses for market selection or aesthetic choice
Ground truth needed to detectYes, requires knowing the bull's-eye to measure directional offsetNo, measurable immediately via standard deviation across peers on identical vignettesNot applicable
Organizational consequenceUniform overpricing, underpricing, or systemic discriminationArbitrary lotteries in justice, silent financial leakage, and low predictive validityInnovation, portfolio differentiation, and price discovery

Distinguishing these three concepts changes how organizations diagnose failure. Because system noise produces no single directional signature in aggregate reports, leaders routinely overlook massive scatter until they compare independent peer judgments side by side.

The target analogy and MSE decomposition: Why error equals bias squared plus noise squared

The Mean Squared Error equation proves that total predictive error equals the square of average bias plus the square of noise (MSE = Bias² + Noise²), which means reducing scatter remedies a flaw human judgment displays just as effectively as eliminating an equal amount of directional bias. To illustrate this geometry, Kahneman, Sibony, and Sunstein introduce four teams at a shooting range. Team A is accurate: its shots cluster tightly around the center bull's-eye. Team B is biased: its shots cluster tightly together, but systematically off-center in one corner. Team C is noisy: its shots center on the bull's-eye on average, so there is zero bias, yet individual shots scatter widely across the paper. Team D is both biased and noisy: its wide scatter is centered far away from the bull's-eye.

In Chapter 5, the authors translate the shooting range into quantitative business forecasting through the hypothetical retail enterprise GoodSell, where executive Amy Simkin evaluates regional sales predictions. When GoodSell's forecasters estimate the five-year market share of a new store, their average prediction is 44% against an actual outcome of 34%. That 10-percentage-point overshoot is the group's statistical bias. At the same time, the individual forecasters disagree with one another, producing a standard deviation of 10 percentage points around their 44% mean. Under a normal distribution, the Mean Absolute Difference between any two randomly chosen forecasters equals 1.128 × Standard Deviation, meaning two peers at GoodSell differ by more than 11 percentage points on average.

Evaluating accuracy across an ensemble of cases using Mean Squared Error (the average of the squared distances between each forecast and the true target) yields the fundamental error equation: MSE = Bias² + Noise². Squaring errors penalizes large misses heavily, which aligns with real organizational damage where a massive underwriting or medical error is far more than twice as costly as a minor one. At GoodSell, the 10% bias contributes 100 (10²) to MSE, and the 10% noise standard deviation also contributes 100 (10²), for a total MSE of 200. Because the two squared components combine orthogonally like the sides of a right triangle whose hypotenuse squared is MSE, cutting noise in half reduces total error by the exact same amount as cutting bias in half.

The authors provide an empirical benchmark that shows why noise usually outstrips bias in daily professional work. For bias and noise to contribute equally to Mean Squared Error, the average directional error must equal one full standard deviation of noise. In a normal distribution, a bias of one standard deviation requires 84% of all judgments to err in the exact same direction, leaving only 16% on the opposite side of the true value. An organization where 84% of project budgets run over cost or 84% of sales forecasts overshoot reality exhibits a glaring, unmistakable bias. Whenever directional skew is less extreme than an 84-to-16 split, noise is the larger driver of total error. Also, reducing noise at GoodSell tightens the cluster of forecasts around 44%, which makes the underlying 10% optimistic bias immediately obvious and easy to correct.

Mental model: The back of the target shows how organizations can quantify error even when objective truth is unknowable. When looking at the front of a shooting target, an observer sees the bull's-eye and can measure both average directional offset (bias) and shot dispersion (noise). Flipping the target around hides the bull's-eye completely, so nobody can tell whether a shooter was biased toward the top left, yet the spread of the bullet holes on the blank paper backing remains completely visible. In fields such as criminal sentencing, commercial underwriting, medical triage, or long-term capital allocation where no objective bull's-eye exists at the time of decision, leaders can inspect the back of the target by having multiple professionals evaluate identical case vignettes and calculating the dispersion of their answers.

Recognizing the back-of-the-target property strips away the common excuse that judgment quality cannot be measured in uncertain or evaluative domains. An organization does not need to know the ideal sentence or the true twenty-year return of a project to prove that its decision makers are contradicting one another.

The anatomy of noise: Level noise, pattern noise, and occasion noise

System noise in human judgment breaks down into three orthogonal components: level noise across judges' average baselines, stable pattern noise from idiosyncratic judge-by-case interactions, and transient occasion noise within the same judge over time. By decomposing total variance into these three layers (System Noise² = Level Noise² + Pattern Noise², where Pattern Noise² = Stable Pattern Noise² + Occasion Noise²), Kahneman, Sibony, and Sunstein show why superficial management fixes fail to stabilize professional decisions. Replacing the single term Noise² in the error equation reveals the complete statistical architecture of judgment failure: MSE = Bias² + Level Noise² + Stable Pattern Noise² + Occasion Noise².

Level noise is the variability in the average judgments made by different evaluators across all cases. In criminal courts, level noise separates harsh "hanging judges" from lenient judges; in corporate appraisals, it separates easy graders from tough graders; in forecasting, it separates chronic optimists from chronic pessimists. Measured as the standard deviation of the judges' individual means, level noise shifts a professional's entire distribution up or down the scale while keeping the relative ranking of cases intact. If two judges differed only in level noise, they would agree completely on which defendant deserved a longer prison term, even while disagreeing on the exact number of years.

Pattern noise is the variability in judges' idiosyncratic responses to particular cases, corresponding to the statistical interaction between the judge and the case (judge × case interaction). Unlike level noise, pattern noise scrambles the relative ranking of cases across evaluators. Judge A may be generally lenient but punish white-collar embezzlement severely, whereas Judge B is generally harsh but treats first-time financial crimes gently. The majority of pattern noise is stable pattern noise: a permanent, reproducible reflection of an evaluator's unique cognitive personality, personal history, and values. Drawing on psychologist Walter Mischel's finding that human personality manifests in stable "if-then" situational signatures, Chapter 16 explains that each judge possesses a distinct mental policy for weighting conflicting cues.

To illustrate how stable pattern noise forms, the authors contrast two versions of a student prediction problem. In the simple Julie 1.0 vignette (Chapter 14), participants learn only one cue: Julie read fluently at age four, and they must predict her graduating college GPA. Evaluators automatically apply the matching operation, mapping the intensity of her childhood reading precocity directly onto a corresponding percentile of the GPA scale (predicting 3.7 or 3.8) and ignoring regression to the class mean. In the complex Julie 2.0 vignette (Chapter 16), evaluators receive multiple contradictory cues: Julie is highly ambitious, her father is severely ill, she had a brief history of drug use, and her early grades fluctuated. Because simple intensity matching cannot resolve conflicting facts, each judge constructs an idiosyncratic causal narrative, suppressing some cues and amplifying others to achieve excessive coherence. In Solomon Asch's classic 1946 adjective experiment (Chapter 13), presenting traits in the order "Intelligent, Persistent, Cunning, Unprincipled" generated a positive evaluation, whereas reversing the order made "Intelligent" seem dangerous. Similarly, conclusion bias (illustrated by George Lucas resisting co-writer Lawrence Kasdan's push to kill off major characters in Return of the Jedi and rationalizing story arguments afterward) and heuristic substitution (such as judging that "Bill" is more likely to be a jazz-playing accountant than a jazz player because he fits the accountant stereotype) operate differently across individuals, generating massive stable pattern noise.

Across empirical studies synthesized in Chapter 17, stable pattern noise is consistently the largest component of system noise, typically accounting for 60% to 80% of total interpersonal scatter. In Jon Kleinberg and colleagues' machine-learning study of 173 bail judges evaluating 141,833 defendants, case characteristics accounted for 67% of decision variance while system noise accounted for 33%. By building predictive algorithms for each individual judge, the researchers proved that 79% of that system noise was stable pattern noise (26% of total decision variance) and only 21% was level noise (7% of total variance), a nearly four-to-one ratio.

The third component, occasion noise, is the transient variability in judgments of the exact same case by the same person at different times. Just as an elite basketball player like Stephen Curry (~90% free-throw accuracy) and a poor shooter like Shaquille O'Neal (~53%) both experience shot-to-shot physical variation on an identical court, cognitive judgments fluctuate with mood, fatigue, ambient temperature, and case sequence. In Chapter 7, the authors cite US asylum judges who became 19% less likely to grant asylum if they had approved the two preceding cases on their docket (a sequential gambler's fallacy), French judges who handed down lighter sentences on a defendant's birthday, juvenile court judges who imposed harsher sentences on Mondays following a local football team loss, and primary-care doctors whose cancer screening orders dropped steadily from morning to late afternoon.

Because occasion noise arises from random sampling inside the brain, individuals can suppress it using a technique discovered by Edward Vul and Harold Pashler (2008) called "the crowd within." As Vul and Pashler wrote, "Responses made by a subject are sampled from an internal probability distribution, rather than deterministically selected on the basis of all the knowledge a subject has." When participants estimated the percentage of the world's airports located in the United States (true value: 32%) and were immediately asked to make a second alternative guess, averaging their two answers recovered one-tenth of the error reduction of consulting a second independent person. When the second guess was delayed by three weeks so working memory cleared, averaging the individual's two estimates recovered one-third of the benefit of a second peer.

Measurement scales themselves can drastically amplify both level and pattern noise. In Chapter 14, Kahneman, Sibony, and Sunstein cite George A. Miller's 1956 discovery of the "magical number seven, plus or minus two," which sets the cognitive ceiling on how many absolute intensity categories a human can reliably distinguish along a continuous dimension. In Chapter 15, they draw on Harvard psychophysicist S. S. Stevens to show that unbounded ratio scales, such as dollar penalties, require an explicit standard reference anchor (a modulus). Without a modulus, evaluators exhibit coherent arbitrariness (documented by Dan Ariely, George Loewenstein, and Drazen Prelec): their initial dollar valuation is heavily swayed by arbitrary anchors, even though their subsequent relative valuations remain internally proportional. In the landmark Joan Glover v. General Assistance study (where 899 mock jurors evaluated 10 personal-injury scenarios involving a child poisoned by an allergy pill with a defective safety cap, conducted by David Schkade, Daniel Kahneman, and Cass R. Sunstein), jurors showed near-perfect agreement on moral outrage (r = .98 across case means). On a bounded 0-to-6 punitive intent scale, true case differences accounted for 49% of variance and noise accounted for 51% (19% level noise, 32% pattern noise). When jurors translated that exact same outrage onto an unbounded dollar scale, true case signal collapsed to 6% and system noise exploded to 94% (37% level noise, 57% pattern noise). Converting those erratic dollar awards into relative case ranks dropped noise from 94% right back down to 49%.

Despite its size, system noise remains hidden because human cognition lives in what Chapter 12 terms "the valley of the normal." In the valley of the normal, unexpected events trigger automatic, backward-looking causal thinking rather than forward-looking statistical thinking. If the Jones family is evicted, a social worker easily constructs a coherent story blaming an inflexible landlord and a job layoff; had the family kept the apartment, the worker would have constructed an equally convincing story citing resilience and community aid. Hindsight erases subjective uncertainty and makes every outcome look inevitable. As the authors summarize in Chapter 17, "Causally, noise is nowhere; statistically, it is everywhere." Because people explain individual outcomes through causal stories, they attribute errors to specific biases while ignoring statistical dispersion across the ensemble.

The following table compares the four components of judgment error across their mathematical definitions, psychological drivers, ranking impacts, and empirical magnitudes in the book.

Error componentMathematical definitionPrimary psychological or structural driverEffect on relative case rankingsEmpirical benchmark in the book
Statistical bias`Mean(Judgments) - True Value`Universally shared heuristics, institutional incentives, or systemic prejudicePreserves case rankings while shifting the entire distributionGoodSell sales forecasts overshot true 34% market share by a 10% mean bias
Level noiseVariance of individual judge means across casesDifferences in baseline severity, optimism, or scale label interpretationPreserves case rankings within each judge while shifting personal baselinesINSLAW federal sentencing study found a 2.4-year standard deviation in judge means
Stable pattern noiseVariance of the reproducible `judge × case` interactionIdiosyncratic narrative construction, personal values, and cue weightingScrambles relative case rankings across judges; represents 60% to 80% of system noiseKleinberg bail study showed stable pattern noise was nearly 4 times larger than level noise (26% vs 7% of total variance)
Occasion noiseWithin-judge variance across separate occasions on identical casesTransient mood, fatigue, weather, case sequence, and arbitrary scale anchorsCauses an evaluator to contradict their own rankings and levels over timeFingerprint examiners reversed 10% of conclusions on re-test; software re-estimates differed by 71%

Breaking noise into these constituent parts reveals why telling professionals to "be more consistent" achieves nothing. Because stable pattern noise and scale noise dominate system scatter, reducing error requires changing the measurement scale and restructuring how cues are combined.

Conducting a noise audit: Measuring hidden disagreement in law, medicine, and finance

A noise audit exposes the hidden noise flaw in institutional judgment by presenting identical, realistic case files independently to interchangeable professionals and measuring the statistical dispersion of their answers. Organizations rarely discover this scatter organically because of the illusion of agreement. Driven by naive realism (the conviction that one sees reality objectively as it is and that any reasonable peer sees it the same way) and by workplace conflict avoidance, professionals assume their colleagues would make the same call. When a disastrous decision occurs, leaders hold a causal postmortem that treats the failure as an isolated exception, leaving the illusion of daily consensus undisturbed.

In Chapter 1 and Chapter 6, Kahneman, Sibony, and Sunstein trace how noise audits transformed the debate over federal criminal justice. In his 1973 book Criminal Sentences: Law Without Order, federal district judge Marvin Frankel denounced unguided judicial discretion as "arbitrary cruelties," documenting how two men with no prior criminal records convicted of cashing counterfeit checks for $58.40 and $35.20 received 15 years in prison and 30 days in jail respectively. To test whether such disparities were systematic, Anthony Partridge and William Eldridge conducted the 1974 Second Circuit Sentence Study, giving 20 identical pre-sentence files to 50 federal judges; they found that an "absence of consensus was the norm." In 1981, John Bartolomeo and colleagues at INSLAW conducted an even larger audit of 208 federal judges evaluating 16 identical hypothetical criminal files covering fraud and bank robbery. Across all 16 cases, the average prison sentence was 7.0 years, but the system noise standard deviation was 3.4 years, meaning two randomly paired judges differed by an average of 3.8 years (1.128 × 3.4). Level noise (SD = 2.4 years) and pattern noise (SD = 2.4 years) contributed equally to total variance (2.4² + 2.4² ≈ 3.4²). For instance, Judge 208 sentenced a fraud defendant in Case O to 25.5 years and a defendant in Case P to 20.0 years, completely inverting the severity ranking assigned by the average of all other judges.

Those audit findings led Senator Edward M. Kennedy to sponsor the Sentencing Reform Act of 1984, creating the US Sentencing Commission. As original commissioner Justice Stephen Breyer noted, the commission bypassed irresolvable philosophical disputes over retribution versus rehabilitation by grounding its two-dimensional sentencing grid (crime offense level by criminal history) in the historical average of past sentences. Although legal scholars Kate Stith and Jose Cabranes (Fear of Judging, 1998) criticized the guidelines as mechanical rules that stripped judges of human wisdom, empirical studies proved the guidelines sharply cut interjudge scatter. When the US Supreme Court's 2005 Booker ruling downgraded the guidelines from mandatory to advisory, Harvard law professor Crystal Yang's 2014 analysis of 400,000 federal sentences demonstrated that interjudge disparity doubled once judicial discretion returned.

Private corporations exhibit equally severe scatter when audited. In Chapter 2, the authors detail a noise audit conducted at a large insurance company across its commercial underwriters and claims adjusters. Before the experiment, the authors polled senior executives on how much they expected two qualified underwriters in the same branch to differ when pricing the exact same risk file; the median executive estimate was 10% or less. When multiple underwriters and adjusters independently evaluated realistic sample cases, the median difference between pairs of underwriters was 55% (five and a half times larger than executives believed), and the median difference among claims adjusters was 43%. On one routine financial risk, one underwriter quoted $9,500 while an equally experienced peer down the hall quoted $16,700.

Similar audits across specialized domains reveal pervasive variability wherever unstructured human judgment governs:

  • In immigration courts (Introduction), asylum grant rates for applicants assigned randomly to judges within the exact same Miami courthouse ranged from 5% to 88%.
  • In software engineering (Chapter 7), developers asked to estimate completion times for identical coding tasks on two separate days differed from their own prior estimates by 71% on average.
  • In forensic science (Chapter 20), cognitive neuroscientist Itiel Dror showed that experienced latent fingerprint examiners secretly re-presented with identical print pairs seven months later changed their conclusions in 10% of cases due to occasion noise, and reversed up to 17% of their verdicts when exposed to biasing contextual clues, a mechanism that caused three senior FBI examiners and an independent court expert to falsely match Oregon attorney Brandon Mayfield to the 2004 Madrid train bombings.
  • In clinical medicine (Chapter 22), studies document wide inter-clinician and intra-clinician disagreement in diagnosing breast and skin cancer biopsy slides, interpreting tuberculosis X-rays and cardiac angiograms, grading endometriosis, and assigning psychiatric DSM classifications.
  • In human resources (Chapter 23), large-scale analyses of corporate 360-degree performance reviews show that 70% to 80% of rating variance reflects evaluator level and pattern noise, leaving only 20% to 30% as valid performance signal.

When organizations try to curb individual scatter by handing decisions to deliberative committees, boards, or juries, they frequently make system noise worse. Chapter 8 explains that while averaging independent votes reduces noise by 1 / sqrt(N), face-to-face group discussion destroys statistical independence through informational cascades and group polarization. When confident or senior members speak first, colleagues withhold private doubts and fall in line. In Matthew Salganik, Peter Dodds, and Duncan Watts's 2006 Music Lab study, participants downloaded 72 unknown songs across eight separate online market worlds where previous download tallies were visible. Because early random downloads triggered social cascades, identical songs experienced wildly different fates across groups: "Best Mistakes" became a runaway smash in one world and flopped in another, whereas "I Am Error" did the exact opposite. Similarly, Lev Muchnik and colleagues (2013) demonstrated on a social news website that a single artificial initial up-vote on a comment raised the probability of subsequent up-votes by 32% and inflated mean ratings by 25% over five months. In mock civil jury studies, deliberating juries polarized so strongly that 27% awarded punitive dollar damages higher than their single harshest member had proposed privately before discussion.

Mental model: The two lotteries of system noise capture how organizational scatter subjects citizens, patients, and clients to compounding chance. Every time a case file enters an institution, the individual faces an interpersonal lottery determined by which judge, underwriter, doctor, or interviewer is randomly assigned to the file, locking in that professional's baseline severity (level error) and unique cognitive sensitivities (stable pattern error). Immediately afterward, the applicant faces an occasion lottery determined by whether the file is reviewed on a Monday morning after a sports loss, late on a Friday afternoon, or immediately after two consecutive approvals. Institutions claim to apply a single policy, yet unguided human judgment turns outcomes into a double draw of the dice.

Exposing these two lotteries through a formal noise audit transforms organizational awareness. Once executives see that their own experts diverge by 40% to 55% on identical data, they can no longer attribute errors to a few bad apples and must address the architecture of the decision process.

Decision hygiene in practice: The six principles and mediating assessments protocol

Decision hygiene remedies the flaw human judgment introduces by establishing preventative procedural rules that strip out scatter and prevent cognitive contamination before an intuitive conclusion hardens. Unlike targeted debiasing, which requires knowing the exact directional bias in advance (such as applying an ex post mathematical penalty to chronically optimistic project timelines or an ex ante nudge), decision hygiene targets an invisible, unpredictable enemy. Kahneman, Sibony, and Sunstein compare it to Ignaz Semmelweis and Joseph Lister introducing surgical handwashing: "Decision hygiene is like handwashing: you don't know which germ you are preventing, but you prevent infections nonetheless." Because noise prevention averts errors that never leave a causal narrative behind, the authors note that "Noise is an invisible enemy, and preventing the assault of an invisible enemy can yield only an invisible victory."

The benchmark for noiseless judgment is the mechanical rule or statistical model. In 1954, University of Minnesota psychologist Paul Meehl published Clinical Versus Statistical Prediction, reviewing 20 studies comparing complete human evaluations against simple actuarial formulas and showing that formulas consistently won. By 2000, William Grove and colleagues meta-analyzed 136 studies across medicine, psychology, forensics, and education, finding a median predictive correlation of .56 for mechanical formulas versus .32 for human clinicians. In 1968 and 1970, Lewis Goldberg uncovered why formulas triumph by constructing "the model of the judge." Goldberg modeled 98 clinical psychologists who predicted graduate student GPA from 10 profile cues by fitting a linear regression to each clinician's own past decisions. When tested on new students, the mathematical model of the judge beat the actual human judge in all 98 instances. The clinician's model stripped away the judge's occasion noise and erratic pattern overrides while preserving the valid linear cue weights. As the authors write in Chapter 9, "You may believe that you are subtler, more insightful, and more nuanced than the linear caricature of your thinking. But in fact, you are mostly noisier."

Even when historical outcome data is too sparse to calculate regression weights, simple equal-weight rules out-predict human experts. In 1974 and 1979, Robyn Dawes (working with Bernard Corrigan) proved the power of "improper linear models," which standardize predictors into z-scores and sum them using equal unit weights (+1 or -1). Because equal weights cannot overfit random sample flukes, improper linear models matched or beat cross-validated regression models and decisively beat clinical experts. In an executive selection study cited in Chapter 10, an unweighted sum of seven candidate trait ratings achieved a correlation of .25 with later managerial performance, whereas complete expert evaluations achieved only .15. Translating correlations into the book's Percent Concordant (PC) metric (the probability that a higher-scored candidate actually outperforms a lower-scored candidate, where r = 0 is 50% coin-flip accuracy, r = .15 is 55% PC, r = .30 is 60% PC, r = .50 is 67% PC, r = .71 is 75% PC, and r = .80 is 79% PC), the equal-weight rule delivered a clear operational edge at zero cost. As Dawes and Corrigan observed, "The whole trick is to decide what variables to look at and then to know how to add," a finding statistician Howard Wainer titled "It Don't Make No Nevermind." Taking this further in 2020, Martin Yu and Nathan Kuncel generated 10,000 linear models with random positive weights across 847 executive candidate assessments; across two samples, 100% of the randomly weighted formulas beat the human experts. While clinicians frequently override models by claiming a "broken-leg" exception (a rare, decisive external fact, like a habitual Tuesday moviegoer breaking a leg), empirical studies prove that professionals invent false broken legs far too often, degrading accuracy by reintroducing pattern noise.

At the same time, Chapters 11 and 12 emphasize that even noiseless algorithms hit a low ceiling imposed by objective ignorance. In Philip Tetlock's 20-year study Expert Political Judgment (2005) tracking roughly 300 geopolitical specialists, "The average expert was roughly as accurate as a dart-throwing chimpanzee." Likewise, in Matthew Salganik and colleagues' 2020 Fragile Families Challenge, hundreds of research teams applying advanced machine learning to thousands of background variables across 4,242 families achieved a maximum correlation of only .22 for predicting material hardship and .44 for predicting GPA at age 15. Noiseless rules eliminate human scatter, but they cannot predict unknowable future turns.

Where full automation is impractical or unacceptable, organizations can capture most of the gains of noiseless models by deploying the book's six principles of decision hygiene across their people and processes:

  1. Acknowledge that the goal of judgment is accuracy, not individual expression: As the authors emphasize, "Judgment is not the place to express your individuality." In clinical medicine (Chapter 22), Dr. Virginia Apgar applied this principle in 1953 by replacing complete newborn assessments with the Apgar score, forcing delivery-room staff to rate five observable cues (heart rate, respiration, muscle tone, reflex, and color) from 0 to 2 at one and five minutes after birth. Standardizing cue evaluation and summation eliminated diagnostic scatter and reduced infant mortality worldwide.
  2. Use statistical thinking and take the outside view: Replace inside-view causal stories with historical base rates and reference classes, and appoint a trained Decision Observer (Chapter 19 and Appendix B) to monitor meetings in real time using a short diagnostic checklist (modeled on condensing the US Office of Management and Budget's 50-page Circular A-4 guide into a 1.5-page checklist) to flag base-rate neglect, substitution, and premature consensus.
  3. Structure judgments into several independent tasks: Prevent excessive coherence and halo effects by isolating evidence streams. In forensic science (Chapter 20), Itiel Dror's Linear Sequential Unmasking (LSU) forces examiners to analyze and lock in their documentation of a crime-scene trace before viewing suspect comparison prints or police files. In corporate hiring (Chapter 24, drawing on Frank L. Schmidt and John E. Hunter's 85-year meta-analyses, Daniel Kahneman's 1956 overhaul of Israeli Defense Forces recruit interviews, and Google's hiring architecture), organizations break the role into separate mediating traits (general cognitive ability, leadership, role-related knowledge, and cultural fit), use rubric-anchored behavioral questions and work-sample tests, and strictly prohibit interviewers from conferring before submitting scores.
  4. Resist premature complete intuitions: Rather than banning intuition, follow the golden rule: "Do not exclude intuition, but delay it." Keep analytical dimensions disaggregated until all factual evidence has been scored independently.
  5. Obtain independent judgments from multiple judges, then consider aggregating those judgments: Select evaluators with high General Mental Ability (GMA), verified domain knowledge, and Jonathan Baron's actively open-minded thinking (AOT) rather than overconfident "respect-experts" (Chapter 18). In the Good Judgment Project (Chapter 21, led by Philip Tetlock, Barbara Mellers, and Don Moore), the top 2% of participants ("superforecasters") outperformed intelligence analysts with classified access by practicing "perpetual beta," defined by Tetlock as "the degree to which one is committed to belief updating and self-improvement." Using Ville Satopää and Marat Salikhov's BIN model (Bias, Information, Noise), the researchers proved that probabilistic training, teaming, and statistical aggregation (Aggregated Noise = Individual Noise / sqrt(N), where averaging 4 estimates cuts noise by 50% and 100 cuts noise by 90%) improved accuracy primarily by suppressing noise. In live committees, replace open discussion with the Estimate-Talk-Estimate protocol.

6. Favor relative judgments and relative scales: Because absolute rating scales invite level and pattern scatter, anchor evaluations to concrete comparative benchmarks. In corporate performance reviews (Chapter 23), forced ranking curves (such as Jack Welch's GE vitality curve) eliminate level noise mechanically but distort small or unevenly matched teams. Frame-of-Reference (FoR) training, Behaviorally Anchored Rating Scales (BARS), and shared case scales (anchoring score levels to familiar reference peers like "Dan P." and "Jenny P.") calibrate evaluators to an identical comparative modulus.

To integrate all six principles into high-stakes strategic choices such as mergers, capital allocations, or policy launches, Chapter 25 introduces the Mediating Assessments Protocol (MAP), built on the maxim that "Options are like candidates." Illustrated through retail CEO Joan Miller and lead analyst Jeff Schneider evaluating the multi-billion-dollar acquisition of Mapco, MAP structures strategic choices by defining independent mediating assessments upfront (market potential, technology depth, regulatory approval, talent retention), assigning isolated teams to gather outside-view facts without smoothing over contradictions, reviewing each assessment sequentially in board meetings using Estimate-Talk-Estimate, and delaying complete executive judgment until every mediating dimension is scored.

The table below maps the six principles of decision hygiene to the cognitive flaws they block, their operational routines, and their tested domain implementations in the book.

Hygiene rule and protocolPrimary error mechanism blockedCore operational routineTested domain implementation
1. Goal is accuracy, not individual expressionIdiosyncratic cue weighting and stable pattern noiseDecompose complex impressions into fixed, empirically validated sub-scoresDr. Virginia Apgar's 1953 five-cue newborn viability score in obstetrics
2. Use statistical thinking and the outside viewInside-view causal stories, non-regressive matching, and conclusion biasAnchor estimates in reference-class base rates and appoint a Decision ObserverUS OMB condensed 1.5-page regulatory checklist and regressive GPA calibration
3. Structure judgment into independent tasksHalo effects, confirmation cascades, and trait-order coherenceIsolate evidence streams and sequence disclosures sequentiallyItiel Dror's Linear Sequential Unmasking (LSU) in latent fingerprint forensics
4. Resist premature holistic intuitionsExcessive coherence and early impression lock-inDelay holistic synthesis until all separate dimensions are scored and frozenGoogle's structured hiring architecture and the Mediating Assessments Protocol
5. Obtain and aggregate independent judgmentsInformational cascades, group polarization, and occasion noisePool independent estimates via statistical averaging (`1 / sqrt(N)`) or Estimate-Talk-EstimateThe Good Judgment Project superforecaster teams and the BIN model
6. Favor relative judgments and relative scalesArbitrary scale modulus choice and coherent arbitrarinessAnchor ratings to concrete peer benchmarks using case scales and Frame-of-Reference trainingBehaviorally Anchored Rating Scales (BARS) using reference employees Dan P. and Jenny P.

Combining these six principles shows that organizations do not have to choose between unguided clinical intuition and rigid automation. Structured human protocols such as the Mediating Assessments Protocol eliminate the bulk of interpersonal and group noise while preserving informed executive judgment for the final decision stage.

The costs and limits of noise reduction: When rules, algorithms, and morale collide

The optimal level of noise in human judgment is rarely zero because eliminating variability completely can impose steep administrative expenses, crude false-positive errors, algorithmic unfairness, and damage to professional morale and human dignity. In Part VI (Chapters 26 through 28), Kahneman, Sibony, and Sunstein address the practical objections raised against noise-reduction reforms by organizing them through economist Albert O. Hirschman's three categories of reactionary rhetoric: perversity, futility, and jeopardy.

The perversity objection argues that strict efforts to eradicate noise backfire and worsen decision outcomes. If an organization replaces human discretion with a crude, rigid rule, it eliminates interpersonal scatter at the cost of severe systematic bias. For example, mandatory "three strikes and you're out" criminal sentencing laws eliminated judicial disparity for third-time felons, but forced courts to hand down disproportionate life sentences for minor third offenses. Similarly, imposing rigid scripts or mandatory forced-ranking quotas on small, high-performing teams can demoralize professionals, erode workplace autonomy, and drive talented staff to quit.

The futility objection claims that noise reduction will fail because participants will game the rules or shift discretion to an unmonitored stage of the process. When mandatory federal sentencing guidelines restricted judges, critics noted that charging discretion simply shifted to prosecutors, who could manipulate indictment counts before a file ever reached the courtroom. The jeopardy objection warns that mechanical rules and predictive algorithms threaten competing moral values, especially equal protection and fairness. If a predictive bail or hiring algorithm is trained on historical data tainted by human prejudice, the model can codify and perpetuate racial or gender disparities (algorithmic bias). Kahneman, Sibony, and Sunstein reply that flawed rules and biased algorithms are arguments for building better, carefully audited decision architectures rather than surrendering to noisy human lotteries. Unlike closed human minds, algorithms can be inspected mathematically and constrained to satisfy explicit fairness criteria alongside predictive validity.

In Chapter 27, the authors confront the moral argument that individualized human judgment is essential to respect human dignity and enable moral evolution. Legal scholars Kate Stith and Jose Cabranes (Fear of Judging) argue that defendants deserve to be judged as whole, unique individuals rather than slotted into a mechanical grid. This constitutional tension surfaced in the 1974 US Supreme Court case Cleveland Board of Education v. LaFleur. Junior high school teacher Jo Carol LaFleur challenged a school board rule that forced every pregnant teacher onto unpaid maternity leave five months before her expected delivery date. Although the five-month rule was simple and completely free of administrative noise, the Supreme Court struck it down under the Due Process Clause, ruling that sweeping irrebuttable presumptions violate individual dignity when individualized medical fitness evaluations are feasible.

Defenders of discretion also argue that noise provides breathing room for moral evolution: when social norms shift, lenient judges, merciful juries, or flexible managers can bend outdated rules until formal policies catch up. Kahneman, Sibony, and Sunstein counter that using noise as an engine of social reform creates intolerable unfairness, because one progressive judge's deviation leaves defendants assigned to traditionalist judges down the hall suffering under the old rule. Legitimate moral evolution requires transparent, system-wide updates to rules and standards rather than an ongoing administrative lottery.

In Chapter 28, the authors formalize this trade-off as a choice between rules and standards governed by decision costs (the time, money, and cognitive effort required to reach a judgment) and error costs (the harm caused by inaccurate or noisy outcomes). Rules specify outcomes in advance (ex ante), restricting the evaluator to narrow fact-finding, whereas standards delegate discretion to the point of application (ex post), instructing judges to determine what is "reasonable" or in "the best interests of the child." In his 1983 book Bureaucratic Justice, Yale legal scholar Jerry Mashaw defended the US Social Security Administration's mechanical disability determination matrix, a rule-bound grid combining claimant age, education, and work capacity that replaced ad hoc judicial discretion across millions of benefit claims. By contrast, sociologist Max Weber condemned Kadi justice (informal, personalized rulings untethered from general rules) as a direct threat to the rule of law.

Mental model: The rules-versus-standards trade-off balances upfront specification costs against downstream noise and error costs. A rule resolves judgment in advance by reducing the evaluator's job to factual verification, such as checking whether a motorist exceeded a 65-mile-per-hour speed limit, which eliminates level and pattern noise across police officers and traffic courts. A standard defers judgment to the point of application, such as requiring motorists to drive at a prudent speed for prevailing conditions, which accommodates unexpected storms or traffic hazards but invites wide interpretive scatter across officers. High-volume, recurrent organizational decisions justify the upfront expense of crafting clear rules, algorithms, or structured guidelines, whereas rare, novel dilemmas without social consensus require standards disciplined by decision hygiene.

Recognizing that zero noise is neither achievable nor optimal prevents executives from treating the imperfections of rigid rules as an excuse for inaction. Wherever recurrent decisions carry high error costs, institutions owe their stakeholders structured decision hygiene.

How to conduct an organizational noise audit and calibrate predictions

Before launching an organization-wide audit, individuals can test their own internal occasion noise using the authors' simple stopwatch routine from Chapter 7: open a smartphone stopwatch with a lap button, look away from the screen, and press the lap button at what feels like five consecutive 10-second intervals; your average departure from 10 seconds measures your personal bias, while the standard deviation across your five laps measures your occasion noise. Likewise, when making single quantitative predictions such as estimating a candidate's future GPA or project revenue, Appendix C recommends applying a regressive correction (Corrected Prediction = Baseline Mean + r × (Intuitive Estimate - Baseline Mean)). If an evaluator intuitively matches Julie's early reading precocity to a 3.8 GPA in a college where the average graduating GPA is 3.2, and recognizes that childhood reading correlates with college grades at roughly r = .20, the calibrated forecast is 3.2 + 0.20 × (3.8 - 3.2) = 3.32 (or 3.3), pulling the extreme intuition back toward the outside-view base rate. To measure interpersonal level and pattern noise across an entire professional unit, Appendix A provides a six-step noise audit routine:

  1. Secure upfront commitment from senior executive leadership by framing the executive team as the audit client, polling their expectations of current peer agreement, and obtaining a pledge to implement process reforms if significant scatter appears.
  2. Assemble four to six realistic, self-contained written case vignettes drawn from actual organizational files that span typical daily complexity without requiring external research.
  3. Administer the case files simultaneously and independently to all interchangeable professionals in the unit under masked conditions so no participant can consult colleagues or identify how peers rated the cases.
  4. Collect a structured process questionnaire immediately after the evaluations are locked, asking participants to explain their reasoning, rate the weight they gave to each case cue, and place each case on a comparative outside-view percentile scale.
  5. Compute the statistical dispersion across evaluators for every case by calculating the mean, standard deviation, and mean absolute difference, then decompose total system variance into level noise across judge means and pattern noise from judge-by-case interactions.

6. Present the empirical scatter alongside leadership's initial expectations to shatter the illusion of agreement, identify whether unbounded scales or conflicting cue weights drove the variance, and replace unstructured discretion with decision hygiene protocols.

Executing this six-step audit converts an abstract statistical problem into an undeniable institutional benchmark. Once leaders see the gap between their assumed 10% disagreement and an actual 40% to 55% peer divergence, resistance to structured guidelines and shared case scales rapidly dissolves.

Orthogonal error decomposition and decision hygiene: How the book reframes institutional accuracy

The central contribution of Noise: A Flaw in Human Judgment is shifting behavioral decision science from directional cognitive biases to the statistical geometry of system noise. Kahneman, Sibony, and Sunstein expand behavioral decision science by shifting focus from directional cognitive biases to the statistical geometry of system noise. For decades, researchers following the heuristics-and-biases tradition treated judgment flaws almost exclusively as systematic skews where most people err in the same direction, such as overconfidence or anchoring. By formalizing the Mean Squared Error decomposition (MSE = Bias² + Level Noise² + Stable Pattern Noise² + Occasion Noise²), the authors prove that random scatter across interchangeable professionals carries the exact same mathematical weight in total error as bias, and exceeds bias whenever fewer than 84% of judgments err in one direction. Also, their empirical synthesis establishes that stable pattern noise (the reproducible interaction between an individual judge's cognitive personality and specific case cues) accounts for 60% to 80% of system noise, overshadowing both baseline leniency differences and transient mood effects.

Equally consequential is the book's shift from post-hoc debiasing to preventative decision hygiene. Because stable pattern noise stems from idiosyncratic narrative construction and conflicting personal biases whose net direction cannot be predicted on any single file, traditional nudges and directional corrections fail to stabilize complex evaluations. By codifying practical architectures such as Linear Sequential Unmasking in forensics, Frame-of-Reference case scales in performance management, the BIN aggregation model in forecasting, and the Mediating Assessments Protocol in corporate governance, the book gives institutions a concrete blueprint to suppress noise while keeping human judgment informed and delayed.

Objective ignorance, the valley of the normal, and bureaucratic justice: Where noise reduction reaches its boundary

The authors explicitly concede that eliminating system noise cannot overcome objective ignorance, which sets a hard ceiling on predictive validity in complex human domains. As demonstrated by Philip Tetlock's twenty-year political forecasting trials and Matthew Salganik's Fragile Families Challenge (where advanced machine-learning algorithms trained on thousands of variables reached correlations of only .22 for material hardship and .44 for teenage GPA), critical determinants of future life and market outcomes are fundamentally unknowable in advance. Noiseless rules and equal-weight formulas consistently outperform human clinicians by removing judgment scatter, yet their absolute predictive power remains modest whenever the underlying environment is stochastic.

A second boundary arises from organizational psychology and institutional design. Because human minds inhabit the valley of the normal, backward-looking causal thinking continually masks statistical scatter and denies decision hygiene any visible causal triumph. Leaders must enforce demanding routines such as isolated fact-finding, Estimate-Talk-Estimate voting, and Decision Observer checklists without ever seeing the specific errors they prevented. Finally, as illustrated by the LaFleur maternity-leave ruling, three-strikes sentencing failures, and the tension between Jerry Mashaw's bureaucratic justice and individualized due process, pushing noise reduction toward rigid mechanical rules can undermine employee morale, individual dignity, and adaptive flexibility when decision costs and error costs are poorly balanced.

Check your understanding of Noise

Five multiple-choice questions testing the mathematical decompositions, empirical studies, and decision hygiene protocols in Noise.

  1. Under a normal distribution of errors, what condition must hold for statistical bias to contribute equally to Mean Squared Error alongside system noise?

    1. Average directional error must equal one standard deviation of noise, meaning 84% of all judgments err in the exact same direction.
    2. At least 55% of judgments must fall above the sample median across two consecutive noise audits.
    3. Level noise and occasion noise must cancel each other out across repeated evaluations of the same case.
    4. The correlation between the predictor cue and the outcome must exceed .50 on the Percent Concordant scale.
    Show answer

    A. Average directional error must equal one standard deviation of noise, meaning 84% of all judgments err in the exact same direction. Because MSE = Bias² + Noise², bias and noise contribute equally when the mean bias equals one standard deviation of noise, which in a normal distribution requires 84% of errors to fall on the same side of the true value.

  2. According to the machine-learning analysis of 173 bail judges by Jon Kleinberg and colleagues, which component accounted for the largest share of system noise?

    1. Occasion noise triggered by time of day, weather, and docket sequence.
    2. Level noise reflecting permanent differences in baseline judicial severity.
    3. Stable pattern noise arising from reproducible judge-by-case interactions, which was nearly four times larger than level noise.
    4. Statistical bias caused by uniform overestimation of defendant flight risk.
    Show answer

    C. Stable pattern noise arising from reproducible judge-by-case interactions, which was nearly four times larger than level noise. In the Kleinberg bail study, system noise represented 33% of total decision variance; within that system noise, stable pattern noise accounted for 79% (26% of total variance) while level noise accounted for only 21% (7% of total variance).

  3. In the Joan Glover v. General Assistance mock jury study, why did system noise jump from 51% on the punitive intent scale to 94% on the dollar award scale?

    1. Jurors fundamentally disagreed on the moral outrageousness of the company's conduct.
    2. Translating shared outrage onto an unbounded dollar ratio scale without a shared modulus forced each juror to pick an arbitrary monetary baseline.
    3. Face-to-face deliberation eliminated pattern noise but doubled level noise across jurors.
    4. Jurors applied regressive base-rate corrections to their dollar estimates.
    Show answer

    B. Translating shared outrage onto an unbounded dollar ratio scale without a shared modulus forced each juror to pick an arbitrary monetary baseline. Jurors agreed almost perfectly on moral outrage (r = .98 across case means), but mapping that feeling onto an unbounded dollar scale without an explicit anchor or modulus generated massive level and pattern scatter.

  4. What did Lewis Goldberg's 'model of the judge' experiment demonstrate across 98 clinical psychologists predicting graduate student GPA?

    1. Human clinicians outperformed linear regression models whenever broken-leg exceptions were present.
    2. A linear regression model trained on an expert's own past judgments predicted reality more accurately than the expert did in all 98 cases because the model is noiseless.
    3. Equal-weight models fail unless at least 20 predictor variables are included in the formula.
    4. Averaging two estimates from the same clinician spaced three weeks apart eliminated stable pattern noise.
    Show answer

    B. A linear regression model trained on an expert's own past judgments predicted reality more accurately than the expert did in all 98 cases because the model is noiseless. Goldberg showed that a linear regression capturing a judge's own average cue weights consistently beats the human judge because the formula applies those weights without occasion or pattern noise.

  5. How does the Mediating Assessments Protocol (MAP) structure executive board evaluations of a strategic option such as a corporate acquisition?

    1. It asks executives to vote holistically at the start of the meeting and assigns teams to justify the majority view.
    2. It replaces the board's vote with a mandatory actuarial formula based solely on historical accounting ratios.
    3. It decomposes the deal into independent outside-view assessments, keeps analytical teams isolated, scores dimensions sequentially via Estimate-Talk-Estimate, and delays holistic intuition until the end.
    4. It requires all board members to debate openly before recording any private ratings so that senior experts can anchor the group.
    Show answer

    C. It decomposes the deal into independent outside-view assessments, keeps analytical teams isolated, scores dimensions sequentially via Estimate-Talk-Estimate, and delays holistic intuition until the end. Treated under the maxim that options are like candidates, MAP decomposes strategic deals into independent mediating assessments, prevents cross-contamination, uses Estimate-Talk-Estimate, and delays holistic intuition.

Test your grasp of MSE, pattern noise, scale effects, and the Mediating Assessments Protocol.

Discussion and reflection questions

Open-ended questions examining the institutional trade-offs and boundary conditions of noise reduction.

  1. Where in your organization are interchangeable professionals currently operating under an illusion of agreement, and how would you design four realistic vignettes to audit their level and pattern scatter?

    Chapter 2: A Noisy System

  2. Why do deliberative committees and boards feel more confident after an open discussion even when informational cascades and group polarization have increased their system noise?

    Chapter 8: How Groups Amplify Noise

  3. How does backward-looking causal thinking in the valley of the normal prevent managers from recognizing when a bad outcome was driven by objective ignorance or system noise rather than a specific bias?

    Chapter 12: The Valley of the Normal

  4. In high-stakes strategic decisions like the Mapco acquisition, why is delaying holistic executive intuition more effective and realistic than trying to ban intuition altogether?

    Chapter 25: The Mediating Assessments Protocol

  5. When does the demand for individualized due process and human dignity justify tolerating the noise of a flexible standard rather than imposing a mechanical rule or algorithm?

    Chapter 27: Dignity

Prompts for leadership teams, policy seminars, and book discussions.

Organizational decision hygiene works best when understood alongside broader cognitive biases and strategic choice protocols. Readers seeking the foundational dual-process model of intuitive System 1 heuristics and deliberative System 2 reasoning can review our comprehensive summary of Thinking, Fast and Slow . For leaders handling complex long-term choices that resist simple pros-and-cons lists, our analysis of Farsighted details full-spectrum mapping, scenario planning, and red-team forecasting under uncertainty. Finally, decision makers examining how extreme outliers and fat-tailed distributions break standard Gaussian risk models can explore our breakdown of The Black Swan .

Savaş Ateş
Written By

Founder & Book Reviewer

Savas Ates is the founder of Good Book Summary. A passionate lifelong learner, product builder, and developer, Savas reads across business, psychology, and personal development to create the web's most comprehensive and structured book summaries.