Decoding Decisions: Inside the Brain - The Undoing Project: A Friendship That Changed Our Minds

Okay, so, like, I want to talk about something kind of interesting today. It's about how we make decisions. You know, how our brains, like, *actually* work when we're judging stuff, making choices, that kinda thing.

So, there was this professor, Paul Hoffman, back in the... well, a while ago. He was really into how people decide things. And he, like, managed to get this grant. Like, a pretty decent amount of money, actually. And he basically used it to start this research center. The whole idea was just to, you know, dig deep into how we humans make decisions.

Apparently, he wasn't exactly thrilled with his teaching job before all this. And getting promoted was, like, a real struggle for him. So, he just quit and bought this building in Oregon, a former church, and renamed it "The Oregon Research Institute." It became this, like, super unique place because it was the only private research place that was entirely dedicated to studying human behavior. I mean, how cool is that? It sounds kinda mysterious, right?

And it was! Nobody really knew what those psychologists were up to in there. It was all a bit hush-hush. One of the researchers there, Paul Slovic, he even said when his kids asked him what he did for work, he just pointed to a poster of a human brain and said, "I study the mysteries inside there." I mean, that's a cool job description.

Now, psychology, it kind of ends up being this, like, catch-all for anything other fields don't want to deal with, right? Any problem that's a bit too messy or complicated ends up there. And the Oregon Research Institute was, basically, a bigger, better trash can! Heh. Anyway, their first real project came from this construction company who were helping build the World Trade Center, the Twin Towers. The architects wanted to make sure people on the top floors didn't feel the building swaying too much in the wind, see? And, like, it's not just an engineering problem. It's also a psychological one. How much movement would people *perceive*?

So, Hoffman built this special room, like, on top of these hydraulic wheels that would make it shake ever so slightly, you know, like you're on the top floor of a skyscraper. And, uh, nobody was supposed to know what was really going on, right? The port authority didn't want to freak out the tenants. And Hoffman was worried that if people *knew* they were in a shaking room, they'd be too sensitive to the movement, and that would mess up the results.

So, they put up a sign outside that said "Oregon Research Institute Visual Studies Center" and offered free eye exams to get people in the door. It was all very sneaky. He even had a student working there who was a qualified optometrist. So, people would go in for their eye exam, and, like, Hoffman would start the shaking. And what they found was that people were *way* more sensitive to movement than the engineers thought.

Like, people would say things like "This is a weird room. I guess I need new glasses. What’s going on?" And the optometrist was, like, totally dizzy by the end of the day. So anyway, after that, the engineers and officials from the port authority flew out to Oregon to try it themselves. They couldn't believe the results. And basically, they ended up adding, like, thousands of these metal dampers to the building to make it more stable. So those extra bits of steel? They probably helped the towers stand for longer after the planes hit.

Okay, so, the "shaking room" was just the beginning. The researchers at the institute were super interested in how people make decisions. And they were also really interested in this book that basically said that humans, like psychologists even, are not as good as statistical formulas at making predictions about their patients.

And that was a big deal. Because if *humans* can't do it better than a formula, then, like, what hope is there for other fields? Lots of professions need expert judgment, right? Doctors, judges, advisors, all that. So, Hoffman and the other psychologists at the Institute wanted to understand what exactly goes on when experts make judgments. Like, how do people put all these different bits of information together and come to a conclusion?

Their first step, though, wasn't to see how bad experts were when they were up against a formula. Instead, they wanted to build a *model* of what experts were *actually* doing when they made decisions. The idea was to, you know, see where people were most likely to go wrong. And that's where it gets super interesting.

Hoffman wrote this paper explaining how experts make judgments. It's not just about asking them, 'cause people aren't always honest about their thought process. So, you look at the information they have and what their judgment is and kind of reverse-engineer it.

So, like, say you wanted to know how Yale chooses students. You could ask them what they look for, like, GPA, test scores, sports, alumni connections, whatever. Then, you look at who they actually *choose*, and you figure out how much weight they're giving to each of those factors. And if you're good at math, you can build a model that shows how all those things interact in their decision-making process.

And Hoffman *was* good at math. But the paper title? Very academic. The main thing is that it wasn’t really meant for general consumption.

But, even though the Oregon Institute was focused on clinical psychologists initially, they knew it applied to all kinds of decision-makers, like doctors, meteorologists, baseball scouts, you name it. They realized that if they could figure out how humans make decisions, they could *improve* those decisions. And they did!

By the end of the decade, Hoffman and his team had made some pretty big discoveries, which this guy Lou Goldberg talked about. He pointed out that even the most basic statistical formulas are more accurate than the judgment of clinical experts. It’s kind of unsettling to think about, isn’t it?

So, what *are* those clinical experts doing? Everyone assumed that when a doctor makes a diagnosis, there's like, some complicated stuff happening up there. And so to understand it, you need a complex model. Lou thought, nah, I'm gonna take a simpler approach.

He chose doctors diagnosing cancer using X-rays. The docs use seven key indicators: size of the ulcer, the shape of its borders, etc. And, obviously, those indicators could be combined in a bunch of different ways, right? Lou pointed out that experts often describe their thought process as super-complex. But he created a really simple computer program that gave all seven indicators the same weight, and used that to decide whether the ulcer was good or bad.

They would have doctors look at these X-ray images and decide if they were cancerous on a scale, and they were doing it twice, unbeknownst to them so they were unknowingly seeing the same ones. They mailed the data to UCLA, and UCLA would calculate the results for them. Goldberg figured that this simple model was just a starting point. He assumed the model would have to be more complex, would have to include, you know, nuanced ways to assess the clues.

But then UCLA sent back the results. Turns out, the simple model was *really* good at predicting the doctor's diagnoses. And the doctors weren’t all saying the same thing, even amongst themselves. They couldn’t even make a consensus with themselves! On top of that, doctors contradicted themselves. They would diagnose the same one differently on the second run, and that's how you end up in dangerous territory.

So, the researchers did the same thing with psychiatrists. They had them evaluate which patients were ready to leave the hospital, using a set of indicators, and again, the results were all over the place. And what's even stranger is that the least experienced doctors were just as accurate as the most experienced doctors.

Goldberg pointed out that the root of the problem may be that doctors and psychiatrists had little opportunity to evaluate or, if necessary, recalibrate their thinking accuracy, as they lacked “immediate feedback." To test that, he broke up doctors in two groups and gave one “immediate feedback” to see if they would improve in their accuracy, but it didn’t yield great results, and that they needed “more info” than just feedback, and that maybe some of the models that were created to mirror doctors was potentially more accurate than the doctors themselves, to which Goldberg thought, “No way, that’s preposterous, how is that possible?”

Turns out it was right. So, that meant that to determine if someone has cancer, the best method would be to use that model to measure it. Which is made by a layman! It sounds almost like a hit-piece.

Goldberg obviously wasn't as optimistic when he began writing the second article, "Man Versus Model," about both experts and the methods used by the Oregon Research Institute. Apparently, it wasn’t so useful in capturing all the nuance of human judgements, and they initially thought they would move on to more complex maths to understand human judgement, but turns out that the best representation was a linear one. It was too bad the doctors themselves weren’t adhering to it.

The importance of this is that, if this is true, then how you objectively hire would give way to a mathematical model. Why is this, exactly? It's because experts are human. They have outside influences that affect their decision making process, and it would make sense that they wouldn't consistently say the same thing at the same time.

Soon after the paper came out, Amos Tversky went to see his friend Paul Slovic on his way to Stanford, and reminisced about their days in Michigan together. At that time, Amos said that him and Daniel were trying to unlock and better understand the mechanics of the human brain. As they passed the basketball to one another, Amos said they were looking for a quiet place to separate themselves from university distractions and to focus specifically on this topic. They already had some findings that said that experts make mistakes, and it’s not because they had a bad day, but because they systematically made errors.

Later that year, Amos was at Stanford and Daniel was still in Israel, collecting data from interesting questions designed by them. One of the questions asked high school students about the ratio of boy and girl children and their order of birth, which had some strange answers. Amos put the same questions to Michigan and Stanford students.

The goal was to understand how people make decisions, or rather, how they go wrong. There was a standard answer, and those who got it wrong had their answers further analyzed. What happens in the brain when people are calculating probability?

Most people got these strange questions wrong, as Daniel and Amos had already assumed, since they did it as well. In fact, Daniel had realized that he had made a mistake, and theorized why he made that mistake. And Amos was fixated with this and did the same mistake. Their commitment allowed them to focus and become an intuition. If they were making the same mistakes in their mind, then they could be sure that most people would do the same thing. Throughout that year, it wasn’t as much of an experiment, and more like a fun discovery, like seeing how the human mind works.

When Amos was young, he realized that there were some people who liked to complicate their lives, and he had the ability to remove himself from them. Daniel also had a unique quality about him, and with it, Amos was able to lower his guard and become a different person. When asked about their relationship, Amos would say that “people aren’t complicated, it’s their relationships.” When working together, Amos would put his skepticism aside.

Apparently, they did not want to be their regular selves, but become who they were together. For Amos, work was just fun, and if he didn’t feel like it was fun, then it was worth nothing. Daniel even began to be affected by this mentality. If anything, Daniel was paralyzed with the best toys and a fear of choosing the wrong ones, until Amos said, “Come on, let’s play with all of them!” And even when Daniel was feeling depressed, Amos was there to encourage him.

When they wrote, it was said that they almost looked conjoined together. The two would sit side by side on a typewriter. However, in Daniel’s words, it was because they were “sharing thoughts.”

Their first article that they did, titled “Belief in the Law of Small Numbers,” demonstrated that people make judgements differently to statisticians, even for the statisticians themselves. Now, the next question was, if people aren’t using statistical judgement, then what are they using? In their second article, they elaborated on it. The title was a difficult one that Amos struggled with, because he wanted the title to properly reflect the theme of the writing, and so they arrived at “Subjective Probability: Judgments by Representativeness.”

Subjective Probability — People have a subjective evaluation of how likely they believe something is. If you’re walking down the street and you see your teenage son walk up to the door and you tell yourself that he probably drank something, that’s subjective probability. But what is “Judgement of Representativeness?” People form conclusions based on judgments from uncertain things, like elections. Instead of calculating correctly, what exactly do we do?

Daniel and Amos said that people replace “law of chance” with “rules of thumb,” otherwise known as “heuristics.” The first one they were going to explain was “representativeness.”

People make a comparison to what they believe to be true in their mind. Am I looking at clouds I already know symbolize an oncoming storm? As a basketball player, does Lin-Manuel resemble a future star in the NBA? People are evaluating probability to judge similarity. You have an idea of something, like a cancerous tumor, and you compare it to the real world.

How they formed this or how they judge representativeness isn’t something that Amos and Daniel go deeply into, as they focus more on where these mental structures are more prominent. The closer an object looks in your mind, the more you can appreciate its representativeness. Therefore, if Event A looks more representative than Event B, then we believe Event A is more likely than Event B.

Their intuition was that people don’t make mistakes randomly, but systematically. Those students that got the questions wrong are used to evaluate and comb through the different forms of human errors. The mental shortcut they are calling “representativeness” isn’t always wrong. Although, when one is faced with uncertainty, sometimes that thought process may yield the wrong outcome.

For example, in the family that has six children, it is more likely for them to choose the sequence of “Girl, Boy, Girl, Boy, Boy, Girl” rather than “Boy, Girl, Boy, Boy, Boy, Boy,” no matter where in the world you are. But what makes this sequence make it more likely? It’s because five boys and one girl is not indicative of the population numbers for males and females. People believe the child sex sequence is more random, even if that’s not true.

What about when do people do major miscalculations when evaluating probability? If people need to judge things that contain elements of randomness, then there will be a miscalculation. In the article, they mention that only believing in something that aligns with the totality is not enough to show the problem, and that it has to represent the processes of the uncertainty, and that there needs to be a consistent interpretation of randomness.

During World War II, people in London thought that bombs were strategically dropped, and only bombed certain areas. (However, statisticians later proved that the areas dropped are consistent with random bombing.) In any group with 23 people, they would believe a coincidence to be anything where two people would have the same birthday, when in fact, it is more than 50%. The model in our mind does not align with the randomness, because we are unable to capture what they contain.

What question is then asked is this: that the model has randomness, and we misjudge by the model, would that still yield an inaccurate result?

The average height for American men is 5’10” and women 5’4,” with a standard deviation of 2.5”. An investigator decides to randomly select a group, and then select a sample of the group. What is the likely rate that the group selected is male under the following premise? One, the person is 5’10”. Two, the average height of the 6 people sampled is 5’8”.

Most people believe that the number of likelihood of the group being male is 8:1, or 2.5:1, when the actual answers are 16:1 or 29:1. But why? Amos and Daniel deduce that it is because when they look at 5’10” they think to themselves, “That has to be a man!” And that closes them off to the possibility that it’s a woman.

A town has two hospitals. In the larger one, they tend to see 45 newborn babies, and the smaller 15. What happens is that 50% of the newborns tend to be boys, but that the exact percentage changes on any given day. So the problem is which hospital would have the record for having male newborns above 60% of the time?

Again, people would get this wrong. Most people would believe the results for both hospitals to be about the same, when in reality the smaller hospital had a greater chance of not being representative of the whole. Even though people know of the effects for sample size differentiation, they won’t necessarily do it, even if you teach them the correct way.

Amos and Daniel are sure that these results apply to a greater audience, as people tend to assume the same sort of thinking when judging about someone being a scientist or a future company closing. And the reason is that it is hard to calculate the probabilities, since they are only doing similar errors when judging people. For example, if someone were to assume that a child would grow up to be a scientist, and they look similar to the stereotype scientist, they would be more likely to assume it’s a true case.

If you look to evaluate whether a statement is true when it’s difficult to know, how can you even know if it’s true?

The first opinion for Daniel and Amos is that there is a mechanism that allows for great judgement and serious judgement failure, and that is about the other mechanism for those opinions in the next article, titled “Availability: A Heuristic for Judging Frequency and Probability.” Again, the feedback was from the students themselves, and used them as experiment subjects for long-term collaboration. People are asked to identify how often letters appear as the first or third letter in a word, not using a dictionary or any text.

And so again, people have committed systemic errors, and make the wrong assumptions. People commit errors because memory warps the cognition, and easily recall words with the letters K, N, L, R, V, which makes it more difficult to remember words with those letters as the third letter.

The more easily recall something, then the more likely that event is real. Something that just happened, is extraordinarily vivid, or something that is common is likely to be thought of very easily, and dominate one’s judgement. They have found that those are very unreliable and that their judgement has become altered by new experiences, and can be changed in a two-hour film, showing that the human thinking process may not be as accurate.

Next, they demonstrate nine other small experiments to further show how human memory plays with the judgement. These things presented to the mind cause mistakes and are excited to learn why. Instead of showing a vision, they wanted to showcase a mind illusion, and these things are abundant. In one scenario, they read 39 names to Oregon University students, with one name being read every two seconds. The names contained clear gender identification. In one list, there were 19 male names, and 20 female names, and in another 20 and 19 female and male names. The list with fewer female names had more famous females, whereas the one with more had less famous women. Without being aware of all of this, the Oregon University students were then told to identify whether male names predominated the list, or female names.

Turns out, the students are backwards in their understanding. If the list was male, they tended to assume that the list was female. The article concludes that each time there is an economic decline, a surgery, a broken marriage, people are not really calculating their judgement. However, it can be used to judge the likelihood with an example. People could search memory banks for the events that happen with that.

Rather than being foolish, people are following a principle to evaluate something, and follow strong guidance to something to be true. However, if they can't find the data to use the guidance, then they could follow into misleading thought and judgement. "Therefore," write Daniel and Amos, "available heuristics can lead to systemic bias," and that human judgement can be warped by the simple thoughts that are remembered.

The next thing they asked then, was what else they could explore.