Primary Submitter:
James J. Staszewski, , jjs@cmu.edu
Domain:
Landmine detection using handheld equipment
Generic description of sponsoring organization or customer:
A military research office
CTA Methods:
Naturalistic multi-modal observation; behavioral analysis; concurrent verbal protocol analysis; experimentation; cognitive modeling, and operational field testing for training assessment were used for cognitive engineering based on expert skills (CEBES).
Number of Participants:
Two expert operators, one each for the legacy handheld detection equipment (AN19-PSS-12) and the dual-sensor technology under development (AN PSS-14).
Expertise determined by field test performance, qualitative behavioral analysis, validation by CTA. 20 novice trainees engaged for assessment of CEBES prototype training programs.
Duration:
Overall, two years for each project. Field data collection and CTA: two months.
Applications:
Landmines are an insidious threat to both military and civilian personnel. Combat and peacekeeping operations in the 1990’s as well as controlled testing showed that U.S. soldiers’ performance with the standard issue PSS-12 equipment was substandard; detection rates for low metal mines were alarmingly low (Figure 1). A technologically innovative dualsensor detector, the PSS-14, was developed to address this problem.
After nine-years and $38M had been invested, however, initial testing of the PSS-14 prototype’s performance disappointed. As the striped bars in Figure 2 show, detection rates for low-metal anti-personnel mines were worse than the PSS-12’s, the system the PSS-14 was intended to replace.
Cognitive task analyses were performed to develop cognitive process models of each expert operator’s skill (summarized in Staszewski, 2007, 2008). The models then served as blueprints for designing operator training for each device. The respective training programs were operationally tested to assess the efficacy of these applications of the CEBES approach. This approach applies the knowledge, understanding, and methods produced by basic research on human expertise and puts training design on a scientific foundation. Core concepts discovered by this work have been generalized and applied to Improvised Explosive Device detection training.
Demonstration of Value:
Empirical evidence of effect:
Three field tests employing army combat engineers showed that the prototype PSS-12 training program produced substantial gains in detection performance (Figure 1) without an increase in false alarms. The most striking index of the benefits of the expertise-based approach to training development was that the greatest improvements occurred against the most difficult-to-detect low-metallic anti-personnel mines. Here, detection rates increased from a pretest level of roughly 15-17% to 87% after 15 hours of training per soldier. Subsequent testing of the same participants on two more tests gave them further practice with the new techniques.
Overall aggregate detection performance ultimately rose to 97% and trainees successfully found 100% of the most-challenging mines in the final test.
Two operational tests of the prototype PSS-14 expertise-based training were conducted. Figure 2 shows results from both CEBES training tests (solid bars) along with the results from the initial operational test. Overall, CEBES-trained operators in both tests produced substantial and reliable detection gains relative to initial test results and achieved manifold improvement against the greatest threat, small, low-metal AP mines.
Customer Perspectives:
The performance improvements achieved by the PSS-14 training were examined by sponsors and deemed sufficient to justify continuation of funding for PSS-14 development.
Subsequent refinements raised overall detection rates to the 97-100% range (Santiago, Locke, & Reidy, 2004). Test results of the CEBES training programs for both detectors were evaluated by military and government leaders and the programs were adopted by US Army and US Marine Corps; orders were issued to cease all use of the PSS-12 until soldiers received the new CEBES PSS-12 training; upon deployment, distribution of new PSS-14 to units was made contingent upon operators’ successful completion of expert-based training, a significant recognition of the critical role of operator training for effective technology deployment; the PSS-14 is now the US military’s standard mine detector and it is also used for humanitarian demining internationally; the U.S. Army presented the Commander’s Award for Public Service for the training development reported here; the 200th Engineer Battalion awarded a Certificate of Appreciation for PSS-14 training the PI delivered prior its overseas deployment.
Return on Investment:
The training studies’ outcomes exemplify the practical utility of CEBES, whose foundation is CTA, for designing instruction that develops significant human skills. However, because cognitive task analyses are reputed to require what some consider exorbitant time and resources (Schraagen, et al, 2000), a key question is, ‘Is the CEBES approach cost-effective?’ For the PSS-14 effort (for which cost information was available), the total cost of analyzing and modeling its expert’s skill, developing the training, and testing it was 0.3% of the roughly $38M already invested in the prototype system’s hardware/software development.
Disciplinary Evaluations:
Cooke & Durso (2008) described the impact of these efforts as a leading “cognitive engineering success story.” Crandall, Klein, & Hoffman (2006) called the outcomes “a forceful demonstration of how CTA can produce effective training.” Advocating the utility and value of cognitive engineering based on expert skills—using CTA to understand how expert knowledge and strategies produce outstanding performance—Klein (2016) cited the contributions of the projects outlined here, stating “That’s what experts can buy you.”
Citations
Cooke, N. J. & Durso, F. (2008). Stories of Modern Technology Failures and Cognitive
Engineering Successes. Taylor and Francis.
Crandall, B., Klein, G. and Hoffman, R. R. (2006). Working Minds: A practitioner’s guide to
cognitive task analysis. Cambridge: MIT Press.
Klein, G. (2016), September 10) From chimps to champs; What’s behind the effort to discredit
experts? Psychology Today. https://www.psychologytoday.com/us/blog/seeing-what- othersdont/201609/chimps-champs
Schraagen, J.M., Chipman, S., & Shalin, V. L. (2000). Introduction to cognitive task analysis. In
S. Chipman, J. M. Schraagen, and V. L. Shalin (Eds.), Cognitive task analysis (pp. 3-23).
Mahwah, NJ: Erlbaum.
Staszewski, J. (2007). Spatial thinking and the design of landmine detection training. In G. A.
Allen, (Ed.), Applied spatial cognition: From research to cognitive technology (pp. 231-265).
Mahwah, NJ: Erlbaum Associates.
Staszewski, J. (2008). Cognitive engineering based on expert skill: Notes on success and
surprises. In Naturalistic decision making and macrocognition. In J.M. Schraagen, R. Hoffman
(Eds.) (pp. 318-48). London, UK: Ashgate Publications.
