How to Lead in a World of Distraction: Four Simple Habits for Turning Down the Noise
von Clay Scroggins
Google also adopted a decision hygiene strategy we haven't yet described in detail: structuring complex judgments. The term structure can mean many things. As we use the term here, a structured complex judgment is defined by three principles: decomposition, independence, and delayed holistic judgment. The first principle, decomposition, breaks down the decision into components, or mediating assessments. This step serves the same purpose as the identification of the subjudgments in a guideline: it focuses the judges on the important cues. Decomposition acts as a road map to specify what data is needed. And it filters out irrelevant information. In Google's case, there are four mediating assessments in the decomposition: general cognitive ability, leadership, cultural 6 (called "googleyness"), and role-related knowledge. (Some of thea assessments are then broken down into smaller components.) Note that a candidate's good looks, smooth talk, exciting hobbies, and any other aspects, positive or negative, that a recruiter might notice in an unstructured interview are not on the list. Studies often find that true variance, that Is. variance attributable to the person's performance, accounts for no more than 20 to 30% of the total variance. The rest, 70 to 80% of the variance in the ratings, is system noise. Where does this noise come from? Thanks to multiple studies of variance in job performance ratings, we know that all the compoare nents of system noise are present. If Lynn is lenient and Mary tough, in the sense that Lynn gives higher ratings than Mary does, on average, to all people being evaluated, then we have level noise. Now, if Lynn is evaluating you and happens to have a distinctly poor opinion of you and your contributions, her general leniency may be offset by her idiosyncratic (and negative) reaction to you. This is what we have called a stable pattern: a specific rater's reaction to a specific person being rated. Because the pattern is unique to Lynn (and to her judgment of you), it is a source of pattern noise.
The results were striking: the proportion of noise in the judgments dropped from 94% to 49% (figure 14). Transforming the dollar awards into rankings revealed that jurors were actually in substantial agreement about the appropriate punishment in different cases. Indeed, the rankings of dollar awards were, if anything, slightly less noisy than the original ratings of punitive intent.
The uniqueness of personality is normally a cause for celebration, but this book is concerned with professional judgments, where variation is problematic and noise is error. The point of the analogy is that pattern noise in judgment is not random-even if we have little hope of explaining it and even if the individuals who make distinctive judgments could not explain them.
Confidence is no guarantee of accuracy, however, and many confident predictions turn out to be wrong. While both bias and noise contribute to prediction errors, the largest source of such errors is not the limit on how good predictive judgments are. It is the limit on how good they could be. This limit, which we call objective ignorance, is the focus of this chapter.
But all forms of noise are undesirable simple one: The basic message that emerges from this research is a most ratings of performance have much less to do with the perfot mance of the person being rated than we would wish. As one revieu summarizes it, “the relationship between job performance and ratings of job performance is likely to be weak or at best uncertain."
Beyond a knowledge of shared norms, experience is necessary, too. You can be a young prodigy if your specialty is chess, concert piano, or throwing the javelin, because results validate your level of performance. But underwriters, fingerprint examiners, or judges usually need some years of experience for credibility. There are no young prodigies in underwriting.
"Google also adopted a decision hygiene strategy we haven't yet described in detail: structuring complex judgments. The term structure can mean many things. As we use the term here, a structured complex judgment is defined by three principles: decomposition, independence, and delayed holistic judgment. The first principle, decomposition, breaks down the decision into components, or mediating assessments. This step serves the same purpose as the identification of the subjudgments in a guideline: it focuses the judges on the important cues. Decomposition acts as a road map to specify what data is needed. And it filters out irrelevant information. In Google's case, there are four mediating assessments in the decomposition: general cognitive ability, leadership, cultural 6 (called "googleyness"), and role-related knowledge. (Some of thea assessments are then broken down into smaller components.) Note that a candidate's good looks, smooth talk, exciting hobbies, and any other aspects, positive or negative, that a recruiter might notice in an unstructured interview are not on the list."
"Studies often find that true variance, that Is. variance attributable to the person's performance, accounts for no more than 20 to 30% of the total variance. The rest, 70 to 80% of the variance in the ratings, is system noise. Where does this noise come from? Thanks to multiple studies of variance in job performance ratings, we know that all the compoare nents of system noise are present. If Lynn is lenient and Mary tough, in the sense that Lynn gives higher ratings than Mary does, on average, to all people being evaluated, then we have level noise. Now, if Lynn is evaluating you and happens to have a distinctly poor opinion of you and your contributions, her general leniency may be offset by her idiosyncratic (and negative) reaction to you. This is what we have called a stable pattern: a specific rater's reaction to a specific person being rated. Because the pattern is unique to Lynn (and to her judgment of you), it is a source of pattern noise."
"Speaking of Heuristics, Biases, and Noise "We know we have psychological biases, but we should resist the urge to blame every error on unspecified 'biases." "When we substitute an easier question for the one we should be answering, errors are bound to occur. For instance, we will ignore the base rate when we judge probability by similarity." “Prejudgments and other conclusion biases lead people to distort evidence in favor of their initial position." We form impressions quickly and hold on to them even when contradictory information comes in. This tendency is called excessive coherence." "Psychological biases cause statistical bias if many people share the same biases. In many cases, however, people differ in their biases. In those cases, psychological biases create system noise.""
"Speaking of Guidelines in Medicine „Among doctors, the level of noise is far higher than we might have suspected. In diagnosing cancer and heart disease-even in reading X-rays- specialists sometimes disagree. That means that the treatment a patient gets might be a product of a lottery." “Doctors like to think that they make the same decision whether it's Monday or Friday or early in the morning or late in the afternoon. But it turns out that what doctors say and do might well depend on how tired they are.' "Medical guidelines can make doctors less likely to blunder at a patient's expense. Such guidelines can also help the medical profession as a whole, because they reduce variability.""
"that 360-degree feedback is a useful tool in that it predicts objectively measurable performance. Unfortunately, the use of this feedback system has created its own problems. As computerization made it effortless to add more questions to feedback systems, and as the proliferation of multiple corporate objectives and constraints added dimensions to job descriptions, many feedback questionnaires became absurdly complex. As we have seen, the halo effect implies that supposedly separate dimensions will in fact not be treated separately. A strong positive or negative rating on one of the first questions will tend to pull answers to subsequent questions in the same direction."
"your chances of picking the better candidate with a structured interview are between 65 and 69%, a marked improvement over the 56 to 61% chance an unstructured interview would give you. Google uses other data as inputs on some of the dimensions it cares about. To test job-related knowledge, it relies in part on work sample tests, such as asking a candidate for a programming job to write some code. Research has shown that work sample tests are among the best predictors of on-the-job performance. Google also uses “backdoor references," supplied not by someone the candidate has nominated but by Google employees with whom the candidate has crossed paths."
"This experiment illustrates excessive coherence: we form coherent impressions quickly and are slow to change them. In this example, we immediately developed a positive attitude toward the candidate, in light of little evidence. Confirmation bias–the same tendency that leads us, when we have a prejudgment, to disregard conflicting evidence altogether-made us assign less importance than we should to subsequent data. (Another term to describe this phenomenon is the halo effect, because the candidate was evaluated in the positive “halo" of the first impression. We will see in chapter 24 that the halo effect is a serious problem in hiring decisions.)"
- →More importantly, rankings mechanically eliminate level noise. If Lynn and Mary are evaluating the same group of twenty employees, and Lynn is more lenient than Mary, their average ratings will be different, but their average rankings will not. A lenient ranker and a tough ranker use the same ranks.
- →A skilled and careful forecaster using the best possible tools and techniques will often miss the correct number in making a quarterly inflation forecast. Meanwhile, in a single quarter, a dart-throwing chimpanzee will sometimes be right. focus on the process, not on the outeome of a single case,
- →The nature of singular decisions raises an important question for the study of noise. We have defined noise as undesirable variability in judgments of the same problem. Since singular problems are never exactly repeated, this definition does not apply to them.
- →Whenever you make a judgement, you carry your own baggage, too. The second lottery is the one that picks the moment when you your judgment, the mood you are in, and other extraneous circumstances that should not affect your judgment but do.
Ähnliche Bücher gesucht?
Beschreibe deine Situation. Die Suche findet das passende Buch aus der gesamten Bibliothek.