This article was transcribed from Gary Klein’s talk for the virtual NDMA Open House on May 9, 2022. It has been edited for conciseness and clarity in the written format.
You can find the original recorded presentation here.
A story to illustrate the purpose of Naturalistic Decision Making
Investigating a dangerous pastime
This example is the work of Peter Kamstra, a Canadian who was a graduate student in Australia studying geology.
In Australia, Peter was studying these strange rock outcroppings that would extend far out into the ocean. He found out that there were people called rock fishers who would climb out to these rocks and fish. The advantage of fishing from these spots was that since the water was deeper, the fish were bigger.
To fish in that area, you didn’t have to rent a boat. If you felt like going out and doing some fishing that day, you just drove over to one of these outcroppings and started to fish. For rock fishers, this was a great idea, but not completely safe. In fact, about 10 people nationally died per year during their attempt to engage in rock fishing.
The Australian government was concerned about these deaths. And so, they did what governments usually do: they asked, “why are people dying?”
Challenging oversimplistic assumptions
The answer to the Australian government’s question is that rock fishers were drowning. So, how do you stop people from drowning? Typically, you require that they wear life preservers, and you make sure that they check the weather forecast to see how a high the waves are going to be.
Peter wasn’t entirely sure that this was good advice–he thought it might be oversimplistic.
He decided to investigate the sport of rock fishing by interviewing people–not in an office, but on the rock ledges while they were fishing. In fact, he bought his own fishing gear. He would turn up sometimes as early as four in the morning so he could meet rock fishers when they arrived, and he’d talk to them while they were all fishing together to find out why were people dying.
What he discovered is that the government solutions were not particularly valuable. As I’ll share shortly, wearing a life preserver doesn’t help in many of the many of the situations where rock fishers die. And the idea of checking the weather forecast didn’t work either because the experienced rock fishers knew that while there might be high waves on one side of a cove, the other side would be protected and very safe to fish.
What rock fishers actually wanted to know was:
- Which way was the wind blowing?
- Which way were the waves running?
That’s what mattered to them, not the overall we weather forecast for that region.
Novice decisions can have deadly consequences
Consider the following three scenarios and contrasting responses from novices and experts:
1. You cast off but don’t catch anything, so you pull your line in and it snags on a rock. What do you do?
The inexperienced fishers would run over to the edge to try to free the line, which made them very vulnerable to the waves.
The experienced rock fishers would just cut the line. It wasn’t worth the risk to save the money for the lure.
2. You feel a fish tugging your line. What do you do?
Novices would run over to the edge and start pulling it in. This made them vulnerable to waves that can sweep over and wash you out to sea.
On the other hand, when experienced rock fishers caught a fish knew there was still a hazard. They would ask somebody to help them. They would work together. And when they went over towards the edge, they would be watching as the fish was flapping back and forth to see if a wave was coming. Sometimes incoming waves were something to worry about, but other times they could use the wave to help lift the fish onto the rock ledge.
3. One minute, you’re on this nice, solid rock ledge. Next, you’re in the water. What do you do?
You swim back to the rock ledge to climb onto it right?
Wrong.
Novices frequently took this course of action, but that was a bad idea because the rock ledge, which had been a solid base of support, became a hazard and the waves could smash you against the rocks or even sweep you underneath the rock. If you’re being smashed into unconsciousness against a rock, a life preserver won’t help you.
Experienced rock fishers would swim away from the rock ledge. They had decided in advance where there was maybe a sandy beach area. They would swim towards that rather than back to the rock ledge.
These scenarios give insight into how novice instincts can lead to disastrous outcomes.
Exploring cognition through expert eyes
So, what are we seeing in the previous illustration?
We’re seeing the cognitive dimension. Rather than just concerning ourselves with the surface features of a risk, we’re asking the experts:
- What can go wrong here?
- Where are the tough decisions?
- What decisions are the new rock fishers making that are getting them killed?
- What makes this a tough situation?
- How do the new rock fishers get confused?
- Where do they go?
- What kind of mistakes do they make?
- And how do you recover from a mistake?
As we saw, there are distinct, un-obvious differences between how novice and experienced performers manage the risks and trade-offs of their craft.
These are the aspects of the cognitive dimension that have fascinated the field of Naturalistic Decision Making. And they’re typically ignored, as we see in the response by the Australian government to make suggestions that doesn’t take into account what’s really going on here.
In Naturalistic Decision Making, a lot of our focus is on tacit knowledge. In many situations, people emphasize explicit knowledge, declarative information, which is factual information that’s easy to obtain and to describe: the routines, the checklists, the procedures.
However, the real expertise comes out in people’s use of tacit knowledge: the way that we can recognize patterns; the ways we can make subtle discernments; the mental models that help us understand what’s going on in a situation; our ability to gain through experience to judge whether something is typical, which allows us to see whether something is an anomaly that we need to pay attention to; and the way we can change our mindsets.
“…there are distinct, un-obvious differences between how novice and experienced performers manage the risks and trade-offs of their craft.”
Out of the laboratory, into the real world
In studying these kinds of tacit knowledge, we examine real-world situations.
There are people who have argued with us that the only way to do good research is to bring a phenomenon into the laboratory where you can study it carefully. However, what we find is that these kinds of dynamic factors are difficult, if not impossible to replicate in the laboratory. If we really want to understand decision making, we must get out of the laboratory and watch what people are doing in real-world settings.
Many of the situations we study involve high stakes, and it’s impossible to build high-stakes decision consequences in laboratory settings.
Also, many of the situations involve multiple players and organizational constraints that are difficult to set up in a controlled environment; however, these are extremely important and central to the kind of decision-making that people experience in real-world settings.
The settings that we examine are dynamic. They keep changing as the situation unfolds.
There’s lots of uncertainty, and the goals that people pursue aren’t always clear. People deal with ill-defined goals, sometimes called “wicked problems.”
Most important of all is the fact that, in the real-world settings we examine, we are studying people with experience and how they use their experience.
“Many of the situations we study involve high stakes, and it’s impossible to build high-stakes decision consequences in laboratory settings.”
Solving a cognitive mystery with expert firefighters
Here’s an example from my own work with firefighters.
It was a study that I did in with my colleagues in 1985. We wanted to know how firefighters make life-and-death decisions under extreme time pressure, when there isn’t time to follow the prescribed steps of generating a variety of options and identifying criteria to evaluate each option. There’s no time when a fire is blazing. So, are firefighters just guessing about how to handle these tough situations in the moment?
We started interviewing expert firefighters—26 of them, each with an average of about 23 years of experience.
As a side note, I remember once being at a meeting and talking about the fact that experience was important for effective decision-making. One of the people in the audience said, “I study decision making in the laboratory and I give my subjects lots of practice.” And I said, “how much practice do you give them?” And he said, “oh, I give them 10 hours of training.” And I thought, 10 hours, as opposed to 23 years? That’s not even close.
We wanted to understand how truly expert firefighters could make decisions, and what they told us was surprising:
“We never make decisions.”
I said “really?” They went on to explain, “You just look at a situation and you know what to do.”
This presented two mysteries:
- How could the firefighters be so confident that they could understand the situation and that the first option would be effective?
- How do you evaluate an option except by comparing it to others, which the expert firefighters said they didn’t do?
As a result of our cognitive interviews, we found that their decisions were built on 20 some odd years of experience in pattern-matching process. They weren’t comparing the patterns; rather, there would be an almost immediate pattern match where they would recognize a prototype of a similar situation from the past.
That’s how the firefighters could respond so quickly and only consider one option— their experience had allowed them to build up a repertoire of patterns in order to quickly size up situations and anticipate which cues to watch, what would likely happen next, the likely goals, and a set of actions that were likely to be successful.
But how do you evaluate the actions? The firefighters would mentally simulate each action, and if the action that they thought of initially would work in their mind, then they carried it out. If it almost worked, they would improve it. And if it didn’t work, they would mentally search their repertoire until they found one that would get the job done.
What we discovered about the expert firefighters’ thought process is an example of what we call Recognition Primed Decision-making, or the RPD model. This model accounts for probably 90% of the decisions people make in tough situations and a much higher percentage in routine situations.
This is just one example of what we’ve been able to learn about human cognition by stepping out of the laboratory into real-world settings.
“…[the firefighters had built] up a repertoire of patterns in order to quickly size up situations and anticipate which cues to watch.”
The origins of the Naturalistic Decision Making community
Now, a bit about the history of NDM.
In the early 1980s, I heard this from somebody at the Army Research Institute: the army had become disenchanted with the results of traditional decision research. It wasn’t helping them. And so, in the mid-1980s, Judith Orasanu, who was working in the Basic Research Office at the Army Research Institute, consulted with Ken Hammond—one of the leading lights of decision research—and her boss, Milt Katz. Together, they established a new program of research.
I was involved in this program, along with Marvin Cone, David Noble, Raanan Lipshitz, and others. We had never met each other before, but as we would encounter each other at program reviews, we said we should set up a meeting to better understand what we have in common.
Thus, in 1989, we had the first NDM conference; but we didn’t call it the first NDM conference, as we didn’t expect there would be any others. We were just meeting. And to be honest, it was more a workshop than a conference. We had selectively invited about 30 people to work on a book, and that book was Decision Making in Action: Models and Methods, which was published in 1993. The cover of the first book on Naturalistic Decision Making illustrated a ship navigating choppy waves. That theme is now reflected in the NDM Association logo.
Because of the interest in NDM, we decided to have another meeting in 1994 and open that up with a general call for participation.
Then, in 1996 Rhona Flynn had a meeting in Scotland, and it just cascaded from there in ways that nobody ever expected or predicted. As a result, we’ve been holding these meetings for over 30 years.
“…we didn’t call it the first NDM conference, as we didn’t expect there would be any others.”
NDM’s major contributions
As a community, we’ve contributed models of cognition. Many of these models are of processes that you won’t find in textbooks on cognition, including the Recognition Prime Decision model.
We have models of sensemaking of how people manage uncertainty, a Recognition Metacognition model from Marvin Cone, a Model of Insight, and many other models of how cognitive functions are performed in natural settings.
That leads us to the topic of macrocognition.
Our interest is not just around decision-making. It includes sensemaking. It includes insight. It includes common ground, and the importance of common ground. It includes what we call “flexecution,” which is the fact that sometimes when you’re executing a plan, you’re modifying not just the plan but your goals as well..
Another contribution is a set of NDM tools. The last I counted when I surveyed our community, we had 42 tools primarily for collecting data. And we’ve expanded the types of support that we can provide—not just training, but decision, support, systems evaluation, procedures, and so forth.
Another way to look at our contributions is to imagine what people would believe without NDM. We used to believe the only way to make a good decision was generate a bunch of options and pick the best one. We no longer believe that because of work involving the Recognition Prime Decision model.
We used to believe expertise depends on learning rules and procedures—the explicit knowledge I showed you before. We no longer believe that; now, we think that expertise depends heavily on tacit knowledge.
We used to believe that projects have to start with a clear description of the goal, but we now realize that in many situations, people are dealing with wicked problems and ill-defined goals. They have to be adaptable in revising their knowledge of the goal as they proceed. They can’t wait until they get a perfect understanding because their understanding develops based on what they do.
We used to believe that to make sense of situations, people would build up from data, to information, to knowledge in sort of a waterfall model. That’s partially true, but it misses the fact that our experience allows us to build patterns that help us determine what counts as data in the first place.
We used to believe that insights arrive by overcoming mental sets, because this is the way insight problems are set up in the laboratories. We now know that there’s a variety of pathways that lead to insights.
We used to believe we could reduce uncertainty by just gathering more information. We now know that there’s a variety of factors that contribute to uncertainty, and lack of information is just one of those factors.
“Our interest is not just around decision-making.”
Contrasting the naturalistic and analytical perspectives
One way of describing the NDM approach is to compare it to traditional analytical approaches.
- We’re interested in identifying the strengths of decision-makers.
They’re interested in identifying their biases. - We’re trying to unpack expertise because we appreciate it, like the experienced rock fishers.
They’re trying to de debunk expertise and show that it doesn’t matter. - We’re interested in tacit knowledge.
They’re interested in explicit knowledge. - We want to understand how people think.
They’re looking at behavior and performance—what people do regardless of what’s behind their actions. - We’re concerned with the tough decisions people make.
They try to represent people’s work in terms of flow charts and tasks and the sequence of tasks. - We work in field settings where we can study actual incidents.
They work in laboratory-based conditions with artificial tasks. - We want to build skills rather than just worry about eliminating errors and biases, and we want to make discoveries rather than merely test hypotheses.
“We’re trying to unpack expertise because we appreciate it.”
The basis and goal of NDM
The NDM approach to research entails a combination of two factors:
- It’s a positive orientation that we take. We’re looking for strengths and expertise, as opposed to the weaknesses that people might show, or biases they may or may not have.
- It’s the way we do our research—the way we conduct our inquiry. We have qualitative methods to explore the cognitive processes that people engage in.
We need both aspects for the NDM approach to work effectively.
If you have a positive orientation but you don’t conduct the inquiry in a way that’s properly appreciative, then you’re not going to get very far.
And if you conduct a cognitive inquiry without the positive orientation, without trying to look at strengths and expertise, I don’t think you’ll make valuable contributions.
These two factors compound in the NDM approach to research, but there’s a third factor; we’re interested in more than just research. We’re also interested in development.
What do we do with our findings? We don’t say “people are biased. How can we overcome their biases?” Rather, we use our findings to boost judgment and improve decision-making. That means helping people to develop expertise, to build their pattern base for improved decision accuracy, and to become faster in the decisions that they make.
This brings me to a final key element of NDM: the decision-makers themselves.
These are experts across domains who are not simply following steps, marking checklists, and performing protocols. Instead, they’re making and using discoveries as they navigate complex, dynamic situations under pressure.
The experts we study are an important part of our community as well.
“We don’t say ‘people are biased. How can we overcome their biases?'”
The future of NDM
So where are we headed in the future? I think we’ve just entered the future. The founding of the NDM Association is a shift for us. We’ve been holding NDM meetings for over 30 years, but we’ve never had a formally structured association before. Now, thanks to Brian Moon, we do.
Part of the future is the next generation of NDM researchers. We think the future is going to include more practitioners—people who make decisions in the field. And in the future, I predict that our investigations will likely expand to include issues such as wicked problems, along with team and organizational dynamics.
An open invitation
I hope you’ve enjoyed learning about the main factors of the NDM framework from my perspective. I hope our approach excites you and encourages you to learn more.
Perhaps you’ll even join our community by becoming a member of the NDM Association.