Physician Reasoning on Management Cases With Large Language Models

Clinical Decision-making Clinical Trial

Official title:

Management Reasoning With AI Chat Bots

Clinical Trial Details — Status: Recruiting

Administrative data

• NCT number: NCT06208423
• Other study ID #: 71319b
• Status: Recruiting
• Phase: N/A
• Start date: December 28, 2023
• Est. completion date: March 30, 2024

Study information

• Verified date: February 2024
• Source: Stanford University
• Contact: Robert J Gallo, MD
• Phone: 6507234000
• Email: rjgallo[@]stanford.edu
• Is FDA regulated: No
• Study type: Interventional

Clinical Trial Summary

This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional management decision support tools on performance on case-based management reasoning tasks.

Description:

Artificial intelligence (AI) technologies, specifically advanced large language models like OpenAI's ChatGPT, have the potential to improve medical decision-making. Although ChatGPT-4 was not developed for its use in medical-specific applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultation. However, little is known about how ChatGPT augments the clinical reasoning abilities of clinicians.

Clinical reasoning is a complex process involving pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could potentially help reduce clinician workload and decrease the likelihood of mismanagement. However, ChatGPT-4 was not developed for clinical reasoning nor has it been validated for this purpose. Further, it may be subject to disinformation, including convincing confabulations that may mislead clinicians. If clinicians misuse this tool, it may not improve reasoning and could even cause harm. Therefore, it is important to study how clinicians use large language models to augment clinical reasoning prior to routine incorporation into patient care.

In this study, participants will be randomized to answer clinical management cases with or without access to ChatGPT-4. Each case has multiple components, and the participants will be asked to discuss their reasoning for each component. Answers will be graded by independent reviewers blinded to treatment assignment. A grading rubric was developed for each case by a panel of 4-7 expert discussants. Discussants independently developed a rubric for each case, and then any discrepancies were resolved through multiple rounds of discussions.

Recruitment information / eligibility

• Status: Recruiting
• Enrollment: 50
• Est. completion date: March 30, 2024
• Est. primary completion date: March 30, 2024
• Accepts healthy volunteers: Accepts Healthy Volunteers
• Gender: All
• Age group: N/A and older

Eligibility

Inclusion Criteria:

• Participants must be licensed physicians and have completed at least post-graduate year 2 (PGY2) of medical training.

• Training in Internal medicine, family medicine, or emergency medicine.

Exclusion Criteria:

• Not currently practicing clinically.

Related Conditions & MeSH terms

• Clinical Decision-making

Intervention

Other:
• GPT-4
OpenAI's GPT-4 large language model with chat interface.

Locations

Country	Name	City	State
United States	Stanford University	Palo Alto	California

Sponsors (3)

Lead Sponsor	Collaborator
Stanford University	Beth Israel Deaconess Medical Center, University of Minnesota

Country where clinical trial is conducted

United States, 

Outcome

Type	Measure	Description	Time frame	Safety issue
Primary	Management Reasoning	Percent correct (range: 0 to 100) for each case.	Within one-hour study
Secondary	Time Spent on Management	Time (in minutes) participants spend per case between the two study arms.	Within one-hour study