Through its scientific and applied endeavors, the NDM community has generated many tools, including methods, models, guidance, and research supports. Some of the tools are available as Courses.
This list is not intended to be comprehensive. The NDMA is seeking ideas and resources for developing this tool box – send recommendations and/or links for other NDM tools to: info@naturalisticdecisionmaking.org.
These NDM community members contributed to the list: Cindy Dominguez, Julie Gore, Robert Hoffman, Devorah Klein, Gary Klein, Laura Militello, Brian Moon, Emilie Roth, Jan Maarten Schraagen, and Neelam Naikar. Adam Zaremsky added the short descriptors and references.
CDM is a knowledge elicitation technique. It uses a set of cognitive probes to determine the basis for situation assessment and decision making during nonroutine incidents in multiple sweeps of the incident in retrospection.
Sources:
Specific changes in cue usage and goals in an expert’s understanding of the dynamics of a particular case.
Sources:
A streamlined set of interview methods for identifying cognitive demands and skills needed to perform a task proficiently.
Sources:
A set of probes organized to capture the most important knowledge categories recognized as characteristic of expertise.
Sources:
A method to assess and identify which specifically trainable cognitive skills an instructor might want to address.
Sources:
A tool for organizing knowledge, by showing the relationships of events and objects between which a perceived regularity exists, through the use of connecting lines, circles, and boxes of some type.
Sources:
A format for practitioners to focus on the analysis of intended project goals, with headings based on the types of information needed to develop a new course or design a new system.
Sources:
A use of Naturalistic Decision-Making cognitive frameworks to more accurately describe the processes involved in real-world decision making, to help decision makers with what they are actually trying to do.
Sources:
A collection of all of the informational and perceptual cues that are pinpointed in critical decision interviews.
Sources:
A system for visual monitoring of a user’s mental state, assessing in real time human emotion, levels of mental workload and stress.
Sources:
A strategy for facilitating human-machine teaming consisting of activities to analyze function allocation trade space, analyze operational demands and work requirements, analyze interdependencies between humans and automation, evaluate alternative options with human performance modeling/simulation, and identify and evaluate alternative function allocation and crewing options.
Sources:
A representation for characterizing activity in work systems that can be decomposed into both work situations and work functions, by capturing all of the combinations of work situations, work functions and control tasks that are possible.
Sources:
A diagram with multiple features for complex sociotechnical systems including: showcasing what work demands an actor can be responsible for, depiction of the fundamental boundaries on the allocation or distribution of work demands from which various possibilities may be derived, with these possibilities regarded as emergent.
Sources:
The HMT Systems Engineering Guide provides this guidance based on a 2016-17 literature search and analysis of applied research. The guide provides a framework organizing HMT research, along with methodology for engaging with users of a system to elicit user stories and/or requirements that reflect applied research findings. The framework uses organizing themes of Observability, Predictability, Directing Attention, Exploring the Solution Space, Directability, Adaptability, Common Ground, Calibrated Trust, Design Process, and Information Presentation.
Sources:
11 dimensions of complexity that make mental modeling so difficult in real-world situations.
Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson, D. K. (2019, October). Cognitive flexibility theory: Advanced knowledge acquisition in ill-structured domains. Technical Report No. 441. Southern Illinois University.
25 strategies people use to defend their mental models and reasoning
Feltovich, P., Coulson, R., Spriro, R. and Adami, J. (1994). Conceptual Understanding and Stability, and Knowledge Shields for Fending Off Conceptual Change. Technical Report No. 7. Office of Naval Research.
A training approach that allows trainees to learn how experts make sense of situations, what they pay attention to, why they make their choices, and what their mental models are, all without the expert having to be present.
Sources:
Tactical Decision Games (TDGs) are simple, fun, and effective exercises to improve one’s decision making ability and tactical acumen by repeatedly playing through problems to learn to make decisions better as well as better decisions.
Sources:
A set of tools for human-machine systems designed to increase the human operators’ knowledge and understanding of the technologies explainability.
Sources:
A cognitive model of training that has one primary function, learning management, and six subordinate functions of a provider: setting/clarifying goals, providing instruction, assessing trainee proficiency and diagnosing barriers to progress, sharing expertise, setting a climate conducive to learning, and promoting ownership of the learning process and performance of the trainee.
Sources:
A new training tool designed to improve the ability of decision makers in the oil, gas, and petrochemical industries to help plant operators build richer mental models and more effective mindsets, helping them to make better decisions.
Sources:
A framework focused on utilizing CTA methods to identify the tough, key decisions of performance, and then create the design of the technology, training, and processes based upon those identified requirements.
Sources:
A replanning tool designed to support human operators to collaborate with an automated support collaborative planner, where the planner was created to operate with observable and directable functioning, with a shared frame of reference to the human operator, to allow for work to be completed iteratively and jointly.
Sources:
A set of methodologies applicable in any context when humans directly interact with computing devices and systems, that facilitates any personal, social or cultural aspects, and addresses issues such as information design, human-information interaction, human-computer interaction, human-human interaction, and the relationships between computing technology and art, social, and cultural issues.
Sources:
A tool for organizing knowledge, showing the relationships of events and objects between which a perceived regularity exists, through the use of connecting lines, circles, and boxes of some type. Sero! is a a software platform for conducting concept mapping-based assessments of mental models.
Sources:
An alternative method to standard performance reviews, where the reviewer and reviewee independently draw up and evaluate a list of decisions made in the prior year, focusing attention to the quality of the decision itself, not the decision maker.
Sources:
An evaluation framework that focuses on a support system’s usability, usefulness, and impact in supporting human performance in complex work environments.
Sources:
A process for team members to share SA information, which may include a group exercise of questioning norms, checking for conflicting information, setting up coordination and prioritization of tasks, and establishing contingency planning.
Sources:
A model that captures the nature and origin of international cognitive differences, usually arising from a group’s origin in a specific physical and social ecology and provides mechanisms for increasing comprehension and effectiveness in the face of these cognitive differences.
Sources:
A set of CTA methods for investigating teamwork.
Sources:
A managerial strategy at the outset of a project where a team engages in a hypothetical presumption that it has failed spectacularly, subsequently working backwards to identify what weaknesses or threats to the project they can avoid.
Sources:
The six main macrocognitive functions are identified as, decision making, sensemaking, problem detection, planning, adapting, and coordinating.
Sources:
A novel method of statistical data analysis for assessing the learnability of cognitive work methods.
Sources:
A means to explain a computational system to decision makers who rely on Artificial Intelligence, so that they may decide on the reasonableness of that system.
1. Trust: A series of active exploration measures aimed at maintaining an appropriate context-dependent expectation for users to know whether, when and why to trust or mistrust an XAI system.
2. Explanation Satisfaction: The degree to which users feel that they understand the AI system or process being explained to them.
3. Explanation Goodness: Utilizing factors such as clarity and precision, a checklist for researchers to either try and design goodness into the explanation that their XAI system generates, or to evaluate a priori goodness of the explanations generated.
A measure to assess the “goodness” (i.e., correctness, comprehensiveness, coherence, usefulness) of a users’ mental model in regard to an XAI system, by calculating the percentage of concepts, relations, and propositions that are in a user’s explanation that are also in an expert’s explanation.
Sources:
A decision-making model that explains how people use situation assessment to generate plausible courses of action, while simultaneously using mental simulation to evaluate generated courses of action.
Sources:
A combination of the conceptual RPD model with computer programming to simulate realistic expert decision making.
Sources:
A description of the ever-changing and interconnected relationship between data, which is available information, and cognitive frames, which are explanatory structures that define entities by explaining their relationship to other entities.
Sources:
Use of the Data Frame model of sensemaking in the creation of computational simulations for machine systems to support human decision makers, utilizing key characteristics of computational cognition, including ontology representation, network theory, and reasoning processes with recursive feedback.
Sources:
Intertwined concepts, both revolving around good managers being able to change goals as they go based on discoveries, by trying to learn more about those goals even as they pursue them.
Sources:
This model describes three paths that can lead people to having insights: contradictions, connections, and creative desperation.
Sources:
Three models of how explanations are formed when a person tries to explain the reasons for the decisions, actions, or workings of a device to another person: Local explaining, global explaining and self-explaining.
Sources:
A model of sensemaking, where in the process by which a person becomes aware of a problem, or an unexpected or undesirable direction of a situation, that person begins to reconceptualize the situation they found themselves in.
Sources:
