CTA in Effect

Exploring the Return On Investment in Cognitive Task Analysis

Made possible through the generosity of

Table of Contents

Introduction

Sponsored by Schmidt Futures, CTA in E/Affect is an initiative of the NDMA that aims to strengthen the case for investment in Cognitive Task Analysis.

The initiative has two focused objectives:

Stage 1 (In Effect) – Documenting the return on investment (ROI) of past applications of CTA.

Stage 2 (In Affect) – Identifying high-value CTAs that could be supported by governments and/or through public-private partnerships.

In Stage One of CTA in E/Affect, the NDMA solicited case studies demonstrating significant ROI from applications of CTA across the globe.

This report summarizes the key findings in the submissions received.

Key Takeaways

1

CTA is creating value in diverse industries.

2

Many CTA methods are being put to use.

Critical Decision Method (CDM)
Team CTA
Applied Concept Mapping
Knowledge Audit
Goal-Directed Task Analysis
Concept, Processes, and Procedures (CPP)
Precursor, Action, Result, Interpretation
Previous slide
Next slide

3

CTA is enabling a spectrum of applications.

Training Programs
AI System Development
Presentations & Reports
Simulation Exercises
Instructional Design Curriculum
Previous slide
Next slide

4

CTA is at work globally.

United States
United Kingdom
Germany
Taiwan
Romania
Singapore
Philippines
Indonesia
Thailand
Previous slide
Next slide

5

The returns on investment in CTA are significant.

What is CTA?

“CTA Practitioners get ‘Inside the heads’ of experts to reveal the nature of their expertise.”

Cognitive Task Analysis is a toolkit used by NDM researchers and practitioners to understand proficient performance and how people achieve expertise.

Studies by NDMA members have found that traditional training programs often cover only a fraction of the skills that workers actually need to succeed in a given role.

CTA practitioners “get inside the heads” of performers to reveal the nature of their expertise, such as how experts:

Such findings can be translated into learning experiences and tools that accelerate the achievement of expertise in others – acceleration that has been demonstrated empirically.

“Studies by NDMA members have found that traditional training programs often cover only a fraction of the skills that workers actually need to succeed in a given role.”

CTA offers many use cases for improving performance through application of findings, including:

Capturing the knowledge of workers with key knowledge and skills so that novices can accelerate their own expertise.

Ensuring that training programs are based on what workers actually need to know and be able to do.

Designing tools to support proficient performance and augment human capabilities.

With CTA, improvements come rapidly, reducing the gap in performance between experts and novices.

It can change the trajectory of entire organizations.

Submissions Overview

The NDMA received 14 submissions a diverse set of domains.

They reinforce the benefits and applicability of CTA for any working environment across the global economy.

DomainParticipantsMethods & ApplicationsValue-Add
Landmine DetectionTotal Number of Participants = 20 novice trainees engaged in assessment of prototype training programs• Naturalistic multi-modal observation • Behavioral analysis • Concurrent verbal protocol analysis • Experimentation • Cognitive modeling • Operational field-testing assessment • Cognitive process models of expert operator’s skill blueprints for designing operator training for each device • Improvised Explosive Device detection trainingSubstantial gains in detection performance without an increase in false alarms; aggregate detection performance rose to 97%
HealthcareTotal Participants = 14 (4 novices (interns), 4 senior trainees (senior residents), 6 faculty (expert) physicians)• Critical Decision Method • Simulation-based training curriculumImproved training program for residents, resulting in a curriculum with 24 different scenarios; validation study showed that experienced physicians were more likely to recognize sepsis in the simulated scenarios than residents
Defense & SecurityTotal Number of Participants = 18 Total Number of Proficient Performers = 18• Critical Decision Method • AI system developmentEnhanced ability of analysts to construct explanatory hypotheses and direct their inquiries
Financial ServicesTotal Number of Participants = 11 (7 experts and 4 journeymen from a global bank and global asset management firm)• Critical Decision Method • Card Sort exercise • Cognitive demand table • User Journeys • UI/UX designsEstablishment of a new Research & Ideation function working across behavioral, data and computer science with the product teams; methods continued to be used in a number of high challenge projects to deliver new insights and pathways for product development and are regarded as part of their R&D toolkit
Biotechnology Manufacturing and ShippingTotal Number of Participants = 120 front-line workers Total Number of Proficient Performers = 20• Team CTA • Critical Decision Method • Applied Concept Mapping • Workplace ethnographic study • Simulation exerciseThe company had no significant backorder problems within 3 months of doing the simulation exercise and implementing the solution discovered in the rehearsals
Petrochemical EngineeringTotal Number of Participants = 32 engineering professionals• Applied CTA • In-house training presentations • Cognitive demands table • Tips guide for ACTADeveloped a common language to ‘think like an expert;’ enhanced knowledge transfer and communication with early career and novice employees
Elite Performance Sport, Elite CoachingTotal Number of Participants = 8 (3 Olympic Coaches, 3 Paralympic Coaches, 2 Inter-disciplinary Specialist Coaches)• Applied CTA • Shared Mental Model development • Coach education resource developmentContributed to one Olympic medal and Olympic Record; seven Paralympic medals and four World Records at the Tokyo Games; 52 Individual World Championships
Crime Scene ExaminationTotal Number of Participants = 12 Scene Examination Supervisors• Applied CTA • Cognitive demands table • written reports and presentations • Guidance documents • Scenario-based training toolIncreased understanding of the cognitive demands in crime scene examination; cultural and policy changes around the training of scene examiners; increased level of preparedness of 120 crime scene examiners
Visual Automotive Damage AssessmentTotal Number of Participants = 10 employees (7 proficient motor engineers and 3 not proficient employees)• Applied CTA • Training course (PowerPoint format) supplemented by a decision aidImproved performance on damage labeling tasks; successfully scaled the pool of available annotators by 4x while simultaneously reducing average annotation cost by 30%
Employment SupportTotal Number of Participants = 4 Employment Support professionals• Applied CTA • Critical Decision Method • Cognitive Audit method • Task diagrams • Cognitive demands table • Insight stories and extracts from the Knowledge audit interviews were produced and shared via a 60-page reportTransformative re-think of roles and responsibilities of ES professionals; provided helpful insights into Employment Support expertise within the organization
Prospect VettingTotal Number of Participants = 5 (2 business development representatives and 3 accounting executives)• Goal-Directed Task Analysis • Knowledge Audit • Critical Decision Method • Heuristic process of the standard procedure of prospect-vetting used for training business development representativesImproved training; enhanced disqualification effectiveness; sped up onboarding of new employees
Clinical Health CareGroup A Total Number = 18 Total Number of Proficient Performers = 18 Group B Total Number = 23 Total Number of Proficient Performers = 23• Problem-solving decision framework • Instructional design curriculumMean scores significantly increased from pretest to the initial post-test and from the initial post-test to the final post-test on the SimulationBased Mastery Learning method for assessment among learners
Social ServicesTotal Number of Participants = 23 employees hired within the previous six months Total Number of Proficient Performers = 23• Concepts, Processes, & Procedures (CPP) • Precursor, Action, Result, Interpretation (PARI) • Decision tree • Editable electronic versions of the agency’s standardized trainingDecreased training time; improved post-training assessment scores; fewer cumulative errors during the audit period
Child WelfareTotal Number of Participants = 15 child welfare professionalsText-based training scenariosSignificantly improved scores from the pretest to the posttest among child welfare professionals

Key Findings

Knowledge Management

Capturing the knowledge of workers with key knowledge and skills so that novices can accelerate their own expertise.

Clinical Heath Care
Multiple translational science outcomes were reported, i.e., setting-up specialized clinical programs in their centers for improving clinical care of patients with HRCR and FCRC, transferring knowledge to novices and peers, and conducting collaborative research studies.
Read More ⟶
High Performance Sport and Elite Coaching
The need for knowledge exchange through the contribution of experts and informal sources of learning has developed leadership and management skills to empower others in transformational, autonomous, interconnected environments.
Read More ⟶
Market Abuse Surveillance in Capital Markets, Financial Services
A cognitive demands table was drawn up illustrating the types of insights experts have in this domain, including baseline trading activities, individual trading, and e-communications data, and scanning the wider context for intelligence gathering.
Read More ⟶
Employment Support
The findings were also commended for providing one of the most helpful insights into Employment Support expertise that existed within the organization at that time.
Read More ⟶
Petrochemical Engineering
The organization developed a common language to ‘think like an expert’ and enhanced knowledge transfer and communication with early career and novice employees.
Read More ⟶
Visual Automotive Damage Assessment
Our team also identified expert strategies (such as mentally simulating how an accident may have occurred) that could be helpful to novice peers, who have historically been tasked with annotating image sets with a series of tags that represent various types of damage.
Read More ⟶
Previous slide
Next slide

Instructional System Design

Ensuring that training programs are based on what workers actually need to know and be able to do.

Clinical Health Care
At baseline testing, none of the learners in group A and group B achieved the Minimum Passing Standard (MPS). At the end of the course, all learners met or surpassed the MPS. The curriculum produced a very large pretest to final posttest learning effect size in both groups.
Read More ⟶
Biotechnology Manufacturing and Shipping
We designed a FutureViewTM Rehearsal exercise for 120 front line lab workers in manufacturing based on the results of the Cognitive Task Analysis.
Read More ⟶
Healthcare: Sepsis recognition in children
A validation study showed compelling evidence that experienced physicians were more likely to recognize sepsis in the simulated scenarios than residents – suggesting that the scenarios provided practice at thinking like an expert in realistic, challenging situations. Eighteen physicians participated in a 1-year validation study.
Read More ⟶
High Performance Sport and Elite Coaching
The coaching and support team have defined and aligned the Shared Mental Model with clarity of role and responsibility around its evolution. Consequently, tailored athlete performance plans have been created for each athlete to develop individualized processes to support performance outcomes. Task Diagrams, Knowledge Audit, and Cognitive Demands Tables (are) used to inform coach education resource development and will contribute to informing the design of training.
Read More ⟶
Landmine Detection
Three field tests employing army combat engineers showed that the prototype PSS-12 training program produced substantial gains in detection performance INSTRUCTIONAL SYSTEM DESIGN without an increase in false alarms. The most striking index of the benefits of the expertise- based approach to training development was that the greatest improvements occurred against the most difficult-to-detect low-metallic anti-personnel mines. Here, detection rates increased from a pretest level of roughly 15-17% to 87% after 15 hours of training per soldier.
Read More ⟶
Petrochemical Engineering
By training a diverse range of employees to complete ACTA, the organization went on to build its own inhouse training presentations—examining expertise in, for example, plant management, to support knowledge transfer more effectively. We supported the production of cognitive demands tables and developed a Tips guide for ACTA.
Read More ⟶
Crime Scene Examination
(CTA research) developed a scenario-based training tool for major incidents which allowed the forensic services unit to identify differences in professional judgment and decision-making application and staff proficiency in different geographic regions of Scotland, and focus training needs in these areas.
Read More ⟶
Prospect Vetting for SaaS-Developer/Provider
Knowledge Audit and Goal-Directed Task Analysis resulted in a heuristic process of the standard procedure of prospect-vetting in the organization. The heuristic was used to train business development representatives to disqualify prospects as earliest as possible in the vetting process.
Read More ⟶
Supplemental Nutrition Assistance Program (SNAP) Case Workers in Social Services
1. CTA-based training required six fewer hours of training than the control version. 2. CTA-based training cohort performed significantly better on the post-training assessment (F=2.6, p=.044). 3. Survival and hazard analyses were used to compare QA audit results as cumulative odds of errors over 3 months. Survival analysis shows the odds of being errorfree over time; hazard analysis shows the log cumulative hazard risk of making an error over time. The CTA-trained group (green line) was significantly less likely than the control group (blue line) to make errors over time, resulting in fewer cumulative errors during the audit period (χ2 = 21.34, p < .001).
Read More ⟶
Previous slide
Next slide

Cognitive Systems Engineering

Designing tools to support proficient performance and augment human capabilities.

Defense and Security​
(CTA) provided the foundations for a system that learned to predict and explore semantically relevant lines of inquiry. A crucial feature of these (AI) systems was that they provided transparency through information granules that were aligned to the decision making of experts.
Read More ⟶
Market Abuse Surveillance in Capital Markets, Financial Services
UI/UX designs were created and a working prototype containing more than eight cognitive assistants to support more efficient investigations of potential market abuse was developed for further validation with a wider group of clients.
Read More ⟶
Previous slide
Next slide

Client Testimonials

"The results have been game changing, making us a prime acquisition target compared to our competition."
“This (prototype) could support the non-linear route most of our investigations take [...] supports real complexity [...] but with an end result that is easy for lay people and lawyers in the courtroom to understand.”
"ACTA is a great tool for understanding the mental leaps made by experts when they engage in complex tasks. It requires us to think about what we mean by experts and expertise, not just in terms of what they know, but how they go about applying knowledge."

Conclusion & Next Steps

As organizations navigate challenging workforce trends—such as losing expertise to retirement and widespread education system decline—CTA methods offer an effective and efficient approach for closing workforce performance gaps.

The case studies from Stage One of the CTA in E/Affect Initiative demonstrate the diversity of domains, methods, applications, and outcomes already realized by CTA practitioners. By investing in CTA, these sponsoring organizations uncovered insights that have directly strengthened their workforce.

The potential for CTA remains mostly untapped. Many sectors have yet to realize its benefits.

Building on Stage One of this initiative, in Stage Two the NDMA will work to identify diverse and promising industry-level targets where CTA applications offer the potential of high-value returns.

Do you think CTA can add value to your organization?

We have much more work to do to capitalize on the possibilities of CTA. We hope you’ll join us.

Want to learn more about CTA?

The CTA Institute offers training in Cognitive Task Analysis for professionals of all disciplines.

NDMA members receive a discount on enrollment.