Effectiveness of an Artificial Intelligent Tutoring System in Simulation Training

Surgical Education Clinical Trial

Official title:

Comparing the Virtual Operative Assistant to Expert-based Instruction in Surgical Education: A Randomized Controlled Trial

Clinical Trial Details — Status: Completed

Administrative data

• NCT number: NCT04700384
• Other study ID #: 2010-270, NEU-09-042
• Status: Completed
• Phase: N/A
• Start date: January 15, 2021
• Est. completion date: May 15, 2021

Study information

• Verified date: May 2021
• Source: McGill University
• Contact: n/a
• Is FDA regulated: No
• Study type: Interventional

Clinical Trial Summary

Brief Summary:

Background:

Although surgical experience and technical skill are associated with better patient outcomes, quantitating surgical ability in the operating room is challenging. In surgical education, large datasets generated by high-fidelity virtual reality simulators can be employed by machine learning algorithms to objectively measure trainee performance and competence on expert benchmarks. This allows repetitive practice of surgical skills in safe and risk-free environments with immediate feedback.

Our group developed and has a patent pending for an intelligent tutoring system called the Virtual Operative Assistant (VOA). Utilizing an Artificial Intelligence (AI) support vector machine algorithm, the VOA assesses data derived from the NeuroVR (CAE Healthcare) simulator platform and provides individualized audiovisual feedback to improve learner performance during simulated brain tumor resections. The effectiveness of intelligent tutoring systems such as the VOA to the human surgical apprenticeship pedagogy remains to be elucidated.

The aim of this study is to compare the effectiveness and educational impact of personalized VOA feedback to expert instruction on medical student's technical skills learning of a virtual reality tumor resection procedure.

Specific Aims: 1) To assess if medical students receiving personalized VOA feedback statistically improve their surgical performance when compared to those having (a) no expert instructor feedback or (b) expert instructor-mediated feedback. 2) To outline if different emotions are elicited by the VOA intelligent tutoring system in medical students while performing this achievement task as compared to human instruction

Description:

Design: A three-arm partially blinded randomized controlled trial of VOA training versus remote-based expert instruction versus control.

Setting: Neurosurgical Simulation and Artificial Intelligence Learning Centre, Montreal Neurological Institute.

Participants: Eligible first- and second-year medical students from across the province of Quebec.

Task: Complete removal of a simulated tumour - distinguishable by colour and haptic properties - with minimal bleeding and damage to surrounding healthy brain using two surgical instruments (Cavitron Ultrasonic Aspirator and Bipolar pincers) of the NeuroVR (CAE Healthcare) surgical simulator.

Intervention: A single 75-minute training session, including six virtual subpial tumour resection attempts (five simple practice scenarios and one complex realistic scenario) with assessment and feedback from either:

1. the VOA intelligent tutoring system (Group 2) or

2. a remote-based expert instructor (Group 3)

Both compared to:

3. control group (Group 1) that receives no assessment or performance feedback.

To our knowledge this will be the first study to compare the effectiveness of an AI-powered intelligent tutoring system to expert instruction in the context of medical and surgical virtual reality simulation and assess the emotional response to such instruction. This study aims to begin to identify successful approaches to use this innovative technology in the medical educational curriculum and improve patient outcomes by augmenting safety, efficiency and competency of surgeons and other healthcare providers.

Recruitment information / eligibility

• Status: Completed
• Enrollment: 70
• Est. completion date: May 15, 2021
• Est. primary completion date: May 15, 2021
• Accepts healthy volunteers: Accepts Healthy Volunteers
• Gender: All
• Age group: N/A and older

Eligibility

Inclusion Criteria:• First- and second-year medical students from any Canadian institution who do not meet the exclusion criteria.

-

Exclusion Criteria: • Participation in any of our group's previous trials involving the NeuroVR (CAE Healthcare) simulator.

-

Related Conditions & MeSH terms

• Surgical Education

Intervention

Behavioral:
• Virtual Operative Assistant Training
Individuals receive the same basic information, have the same amount of time and perform the same scenarios as the control group. In the 5-minutes between attempts, participant receive the Virtual Operative Assistant assessment of their performance and audiovisual feedback.
• Remote-Based Expert Instructor Training
Individuals receive the same information, have the same amount of time and perform the same scenarios as the control group. Meanwhile, a trained instructor observes the participant's on-screen performance, that is live-streamed, remotely. Instructors are senior neurosurgery residents with extensive experience in performing and assessing this scenario. During the 5-minute feedback session, they chat with the student, discussing the performance and help in setting goals for the next trial.

Locations

Country	Name	City	State
Canada	Neurosurgical Simulation and Artificial Intelligence Learning Centre	Montreal	Quebec

Sponsors (1)

Lead Sponsor	Collaborator
McGill University

Country where clinical trial is conducted

Canada, 

References & Publications (8)

Birkmeyer JD, Finks JF, O'Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ; Michigan Bariatric Surgery Collaborative. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013 Oct 10;369(15):1434-42. doi: 10.1056/NEJMsa1300625. — View Citation

Birkmeyer JD, Stukel TA, Siewers AE, Goodney PP, Wennberg DE, Lucas FL. Surgeon volume and operative mortality in the United States. N Engl J Med. 2003 Nov 27;349(22):2117-27. — View Citation

Duffy, M.C., et al., Emotions in medical education: Examining the validity of the Medical Emotion Scale (MES) across authentic medical learning environments. Learning and Instruction, 2020. 70: p. 101150.

Leppink J, Paas F, Van der Vleuten CP, Van Gog T, Van Merriënboer JJ. Development of an instrument for measuring different types of cognitive load. Behav Res Methods. 2013 Dec;45(4):1058-72. doi: 10.3758/s13428-013-0334-1. — View Citation

Mirchi N, Bissonnette V, Yilmaz R, Ledwos N, Winkler-Schwartz A, Del Maestro RF. The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine. PLoS One. 2020 Feb 27;15(2):e0229596. doi: 10.1371/journal.pone.0229596. eCollection 2020. — View Citation

Stulberg JJ, Huang R, Kreutzer L, Ban K, Champagne BJ, Steele SR, Johnson JK, Holl JL, Greenberg CC, Bilimoria KY. Association Between Surgeon Technical Skills and Patient Outcomes. JAMA Surg. 2020 Oct 1;155(10):960-968. doi: 10.1001/jamasurg.2020.3007. Erratum in: JAMA Surg. 2020 Oct 1;155(10):1002. — View Citation

Winkler-Schwartz A, Marwa I, Bajunaid K, Mullah M, Alotaibi FE, Bugdadi A, Sawaya R, Sabbagh AJ, Del Maestro R. A Comparison of Visual Rating Scales and Simulated Virtual Reality Metrics in Neurosurgical Training: A Generalizability Theory Study. World Neurosurg. 2019 Jul;127:e230-e235. doi: 10.1016/j.wneu.2019.03.059. Epub 2019 Mar 15. — View Citation

Winkler-Schwartz A, Yilmaz R, Mirchi N, Bissonnette V, Ledwos N, Siyar S, Azarnoush H, Karlik B, Del Maestro R. Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. JAMA Netw Open. 2019 Aug 2;2(8):e198363. doi: 10.1001/jamanetworkopen.2019.8363. — View Citation

Outcome

Type	Measure	Description	Time frame	Safety issue
Primary	Change in procedural performance .	Performance in each practice attempt is measured utilizing raw data from the simulator that is used for assessment by previously established AI algorithms on validated metrics.	Day of Study
Primary	Change in learning	Performance on the complex realistic scenario is evaluated by expert instructors using the Objective Structured Assessments of technical Skills (OSATS) Visual Rating Scale (weighted at 50%) and the AI assessment algorithms (weighted at 50%) creating a composite performance score.	Day of Study
Secondary	Difference in the strength of emotions elicited	Measured using Duffy's Medical Emotions Scale (MES), before, during and after the intervention.	Day of Study
Secondary	Difference in cognitive load	Measured using Leppink's Cognitive Load Index (CLI) after the intervention.	Day of Study