Primary Submitter:
Emily Newsome, enewsome@shadowboxtraining.com
Domain:
Child Welfare Investigations
Generic description of sponsoring organization or customer:
The original scenario development work was sponsored by a large nonprofit that sponsors innovative work in juvenile justice, child welfare, and workforce development. The resulting training was implemented by public child welfare agencies in Ohio and Colorado.
Cognitive Task Analysis Method(s):
Scenarios were developed based on Critical Decision Method interviews (Crandall, Klein, & Hoffman, 2006).
Number of Participants:
Researchers conducted 15 Critical Decision Method interviews. All interviewees were deemed proficient based on their recommendation from colleagues.
Duration:
Critical Decision Method interviews took place in four phases over a period of 5 years. Each phase of scenario development – which included interview preparation, data collection and analysis – lasted approximately 6 months and generated 6 scenarios. After the conclusion of the scenario development phase, the training has been implemented in five jurisdictions. The average implementation effort took 12 months.
Application(s)
Findings share
CDM interviews produced decision requirements and expert strategies that provided a knowledge basis for critical decisions in child welfare. Our team then extended interview findings (incidents and decisions) into text-based training scenarios (ShadowBox; Klein & Borders, 2016). Subject-matter experts informed the development of scenarios (prompts and decision options) and provided decision rationale statements once scenarios were completed.
Over the course of the project, we developed 23 child welfare scenarios. The scenarios and implementation materials were provided to the client digitally.
Instructional and/or training experience
The resulting training has been implemented in a total of five different jurisdictions – two of these jurisdictions participated in an 18-month pilot analysis period. Each implementation period involved forming a local planning committee, training local staff to become small-group facilitators, and selecting relevant scenarios before delivering the training. Training was delivered in groups of 8-12 child welfare professionals (a mix of line workers, supervisors, and managers) and featured peer-to-peer discussion about every decision point. Training sessions lasted 1.5-2 hours each, and groups met five or six times throughout each implementation period.
Demonstration of value:
Evidence of value
During the pilot implementation period (which focused on two implementation sites), two main
metrics were collected: 1) a pre- and post-test comparison designed to assess cognitive skills related to critical thinking and 2) a survey measuring participant reactions.
A pre-test was administered before the first scenario and a post-test was given immediately following the final scenario. The questions on the pre-test were designed to measure the concepts within the scenarios themselves, which were identified as sub-skills of critical thinking: prioritizing, analyzing concerns, developing interventions, assessing actions, and generating hypotheses. A paired sample t-test compared pre-test scores to post-test scores indicated that performance significantly improved from the pretest to the posttest, Sd = .10, t(37) = 3.17, p < .001; d = .51.
Participant perceptions of the experience suggest that they found the training sessions worthwhile, and particularly found value in the opportunity to gain insight into how their peers managed challenging decisions. Participant surveys with the other implementation sites have also found similar results.
Citations
Used in the case study
Crandall, B., Klein, G. A., & Hoffman, R. R. (2006).
Working minds: A practitioner’s guide to cognitive task analysis. MIT Press. Klein, G., & Borders, J. (2016).
The ShadowBox approach to cognitive skills training: An empirical evaluation. Journal of Cognitive Engineering and Decision Making, 10(3), 268-280.
Prior publication about this work
Jalaeian, M., Newsome, E., & Militello, L. G. (in press).
Applying ShadowBox training method to improve child welfare staff critical thinking skills. British Journal of Social Work. Newsome, E., & Klein, G. (2017, June).
Cognitive skills training for frontline social workers: A pilot study. In Proceedings of the 13th International Conference on Naturalistic Decision Making, Bath, UK. Newsome, E., Wright, C., Klein, G., Flory, J., & Baker, A. (2015, October).
Using the ShadowBox™ method to detect the ‘Investigator’ and ‘Proceduralist’ mindsets in frontline social workers. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59(1), 816-820.
