Primary Submitter:
Wendy Jephson, wendy.jephson@letsthink.com
Domain:
Market Abuse Surveillance in Capital Markets, Financial Services
Generic description of sponsoring organization or customer:
Fortune 500 company serving $125 trillion public capital markets and $9.8 trillion private capital markets with data, technology and trading services.
Cognitive Task Analysis Method(s):
Critical Decision Method and Card Sort Exercise (Stanton, 2005, Crandall, et al., 2006; Gore and McAndrew, 2009)
Number of Participants: Total Number = 11; Number of Experts = 7, Number of Journeymen = 4;
Method for determining proficiency:
Proficiency scaling (Hambrick and Hoffman, 2016).
Total of 25 hours of data collection. Data was collected in the UK and US. Participants were
from a global bank and global asset management firm.
Duration:
Four calendar months.
Application(s)
Findings share
The Card Sort exercise was used to capture relationships between tasks conducted, information available and methods of accessing the information. The exercise demonstrated the variety of data sources that experts use when they are investigating potential market abuse alerts, the different datasets that are accessed for different alert types and the high number of systems that they have to go to in order to access the data they need.
A detailed Hierarchical Task Analysis was completed for a Compliance Task ‘Close Alerts’. This was initially completed by one of the research teams and then verified by all researchers. This illustrated the complexity and scope of the task for analysts who complete this task hundreds of times in their working day. The HTA was verified with some of the participants who also coded the tasks according to whether they were likely to be done by novices, those with intermediate experience or only by experts. An observation from a participant was: ‘you’ve captured what we do, but I’ve never seen it written down like that before’.
A cognitive demand table was drawn up illustrating the types of insights experts have in this
domain, including baselining trading activities, individual, trading and e-communications data
and scanning the wider context for intelligence gathering.
From this work User Journeys were created specifying the cognitive support requirements of compliance analysts undertaking investigations using their existing tooling to ensure new solutions could be both interoperable technically and cognitively compatible with current workflows. UI/UX designs were created and a working prototype containing more than eight cognitive assistants to support more efficient investigations of potential market abuse was developed for further validation with a wider group of clients.
Demonstration of value
Customer-provided perspective
One potential customer said: “This (prototype) could support the non-linear route most of our investigations take…supports real complexity…but with an end result that is easy for lay people and lawyers in the courtroom to understand.”
Evidence of value
The sponsoring organization resourced the research and prototype development in an interdisciplinary team as well as its public demonstration at client conferences and meetings to further validate the findings and application of the research and design for the development pipeline.
The value created via the use of this method was recognised through the establishment of a new Research & Ideation function working across behavioral data and computer science with the product teams. The methods have continued to be used in a number of high-challenge projects to deliver new insights and pathways for product development and are regarded as part of their R&D toolkit.
Citations
Used in the case study
Crandall, B., Klein, G., &; Hoffman, R. H. (2006). Working minds. Cambridge: MIT Press. Gore, J. Ward, P, Conway, G., Ormerod, T., Wong, W. &; Stanton N. (Eds) (2018) Naturalistic Decision Making: Navigating Uncertainty in Complex Sociotechnical Work Special Issue Cognition Technology &; Work. 20 (4) 521-527.
Gore, J., McDowall, A., Banks, A. (2018) Advancing ACTA: developing socio-cognitive competence/insight. Cognition Technology and Work, 20 (4) 555-563.
Gore, J. &; Ward, P. (Eds) (2018) Naturalistic Decision Making and Macrocognition Under Uncertainty: Hambrick, D. Z., & Hoffman, R. R. (2016). Expertise: A second look. IEEE Intelligent Systems, 31(4), 50-55.
Theoretical &; Methodological Developments. Journal of Applied Memory &; Cognition. 7 (1) 33-34.
Gore, J. and Ward, P. (2017) Eds. Naturalistic Decision Making and Uncertainty. Proceedings of
the 13th Bi-Annual Naturalistic Decision Making Conference. University of Bath, UK. ISBN 978-0-86197-194-7.
Gore, J., Ward, P., Conway, G. (2017). Naturalistic Decision Making under Uncertainty. ESRC Centre for Research Evidence on Security Threats Security Review, Oct Issue 6.
Gore, J. &; Conway, G.E. (2016) Modeling and aiding intuition in organizational decision making: a call for bridging academia and practice. Journal of Applied Research in Memory and Cognition. 4 (3)164-168.
Gore, J. &; McAndrew, C. (2009). Accessing expert cognition, The Psychologist, 22, 3, 218-219.
Hoffman, R.R., &; Militello, L.G. (2008). Perspectives on cognitive task analysis: Historical origins and modern communities of practice. New York, NY: Taylor &; Francis.
Hoffman, R.R.,Ward, P., Feltovich, P.J., Dibello, L., Fiore, S., &; Andrews, D.H. (2014). Accelerated Expertise. New York: Psychology Press, Taylor &; Francis.
Klein, G., &; Hoffman, R. (2008). Macrocognition, mental models, and cognitive task analysis methodology. In J. M. Schraagen, L. Militello, T. Ormerod, &; R. Lipshitz, (Eds.). Naturalistic decision making and macrocognition. (pp. 57-80). Ashgate: Hampshire, U.K.
Leaver, M., Griffiths. A., Reader, T. (2018) Near Misses in Financial Trading: Skills for Capturing and Averting Error. Human Factors Vol. 60, No. 5, August 2018, pp. 640–657.
McAndrew, C. &; Gore, J. (2012). Understanding preferences in experience-based choice: a study of cognition in the wild. Journal of Cognitive Engineering and Decision Making.
McAndrew, C. &; Gore, J. (2010). An Inter-Disciplinary Study of NDM and Portfolio Managers. In Mosier, K.L. &; Fischer, U.M. (Eds.) Informed by Knowledge: Expert Performance in Complex Situations. Psychologist Press, 353-36.
Militello, L.G. &; Hutton, R.J.B. (1998). Applied cognitive task analysis (ACTA): A practitioner’s toolkit for understanding cognitive task demands, Ergonomics, 41, 11, 1618-1641.
Militello, L.G., Wong, W., Kirsechenbaum, S.K., and Patterson, E. (2010). Systematizing discovery in cognitive task analysis. In Mosier, K.L. &; Fischer, U.M. (Eds.) Informed by knowledge: Expert performance in complex situations: 23-40. New York, NY: Taylor and Francis.
Roth, E., Ohara, J., Bisantz, A., Hoffman, R., Klein, G., Militello, L., Pfautz, J.D. (2014). How to recognise a “good” CTA ? Proceedings of the Human Facors and Ergonomics Scoiety 58th Annual Meeting: 320-324.
Stanton, N.A. (2005) Behaviour and Cognitive Methods. In: Stanton, N. A., Hedge, A., Brookhuis, K., Salas, E. &; Hendrick, H., eds. Handbook of Human Factors and Ergonomics Methods. Boca Raton, FL: CRC Press, 27.1-27.8.
Tofel-Grehl, C., &; Feldon, D.F. (2013). Cognitive task analysis-based training: A meta analysis of studies. Journal of Cognitive Engineering and Decision Making 7 : 293-304.
Ward, P., Gore, J. Hutton, R. J. B., Conway, G. &; Hoffman, R. R. (2018) Adaptive Skill as The SineQua Non of Expertise. Journal of Applied Memory and Cognition. 7 (1) 35-50.
Prior publication about this work
Jephson, W., Leslie, A., Wise, N., Gore, J. Re-writing Tomorrow: Insights in Trading Surveillance (2019). International Conference on Naturalistic Decision Making 2019, San Francisco.
