Primary Submitter:
Emily Newsome, enewsome@shadowboxtraining.com
Domain:
Visual Automotive Damage Assessment
Generic description of sponsoring organization or customer:
The sponsor for this work was a large company with a focus on applied artificial intelligence. They are using automotive damage as a test bed for refining the speed and accuracy of their AI models.
Cognitive Task Analysis Method(s):
We used the simulation interview technique (Applied Cognitive Task Analysis; Militello & Hutton, 1998) during which we presented pictures of damaged vehicles with varying levels of damage. The objective of the interviews was to identify how experts recognize and categorize different kinds of vehicle damage. Each interview involved reviewing 3-4 cases, in which the interviewee described the cues and strategies they used to classify damage.
Number of Participants:
We conducted 10 simulation interviews with employees of the sponsoring organization. Seven interviewees were proficient performers; Three interviewees were not proficient. Proficiency was determined based on the interviewee’s field experience and education as a motor engineer: ‘proficient’ interviewees had the credential and experience as a motor engineer while the non-proficient interviewees did not.
Duration:
The project had two phases. Phase 1 lasted 3 months and involved performing and analyzing the simulation interviews. Phase 2, which involved developing and evaluating training materials based on findings from the simulation interviews, lasted 7 months. The entire effort (Phase 1 and 2) took place over a period of approximately 10 months.
Application(s)
Findings share
Analysis of the simulation interviews identified cues experts use to help recognize and categorize all automotive damage. Our team also identified expert strategies (such as mentally simulating how an accident may have occurred) that could be helpful to novice peers, who have historically been tasked with annotating image sets with a series of tags that represent various types of damage. For example, when reviewing a case, experts attempt to mentally simulate how the accident likely occurred. This strategy allows them to build an organized model of the visible damage and direct their attention to areas of the vehicle that may contain less obvious damage.
Novices used an approach that was more disorganized and involved scanning for visible damage in the expected impact areas. We recorded these observed expert/novice differences and developed a report, which was delivered to the client in a digital format. After reviewing a substantial list of strategies that experts use to identify damage, the client selected a few high-priority strategies that they thought would have the biggest impact on their work system.
Instructional and/or training experience
Building on the findings from Phase 1, researchers built a training course designed to accelerate
expertise in novices, who typically do not have prior experience in assessing vehicle damage.
The training course focused on cues and strategies used by experts to assess and determine the
point of impact and direction of impact of a specific accident based on photos of the resulting
damage. The course is in a PowerPoint format and is supplemented by a decision aid that can be
used to help guide decision-making about a single, specific example.
Demonstration of value
Evidence of value
The sponsoring organization conducted an internal training validation study to assess the
effectiveness of the training materials. A comparative analysis of novice performance data before
and after receiving the training indicated that performance improved drastically on damage
labeling tasks. The observed improvement can be partially attributed to completion of the
training course in addition to other changes recommended by our work in Phase 1 (e.g., reorganization of damage label categories). Specifically, novices improved in labeling different instances of the same damage type and decreased how often they missed critical cues compared with their performance before the described effort. Finally, novice abilities to identify point and direction of impact was more comparable with experienced motor engineers after participating in training.
Customer-provided perspective
The changes implemented based on the training course and recommendations made in Phase 1
enabled a transition of the damage type identification task from industry experts with decades of
damage assessment experience to novices with up to 7 months of prior experience. Following a
training period of 2-3 weeks, we have successfully scaled our pool of available annotators by 4x
whilst simultaneously reducing average annotation cost by 30%. Unblocking scale at the annotation stage will further accelerate our model development cycles and improve the service delivered to our end users.
Citations
Militello, L. G., & Hutton, R. J. (1998). Applied cognitive task analysis (ACTA): a practitioner’s toolkit for understanding cognitive task demands. Ergonomics, 41(11), 1618-1641.
