Clinical Trials Logo

Clinical Trial Details — Status: Completed

Administrative data

NCT number NCT06457269
Other study ID # 1426887-2024-1
Secondary ID 22XQT0309CBY22-Q
Status Completed
Phase N/A
First received
Last updated
Start date October 1, 2023
Est. completion date April 4, 2024

Study information

Verified date June 2024
Source North Sichuan Medical College
Contact n/a
Is FDA regulated No
Health authority
Study type Interventional

Clinical Trial Summary

The clinical trial aimes to evaluate multiple large language models in respiratory disease consultations by comparing their performance to that of human doctors across three major medical consultation scenarios. The main question aims to answer are: - How do large language models perform in comparison to human doctors in diagnosing and consulting on respiratory diseases across various clinical scenarios? In three clinical scenarios including the online query section, the disease diagnosis section and the medical explanation section, research assistants or volunteers will be asked to cross-question all LLMs or real doctors using predefined online questions and their own issues. After each questioning session, a short washout period is implemented to eliminate potential biases.


Recruitment information / eligibility

Status Completed
Enrollment 703
Est. completion date April 4, 2024
Est. primary completion date December 12, 2023
Accepts healthy volunteers No
Gender All
Age group N/A and older
Eligibility Inclusion Criteria: 1. Self-reported symptoms of common respiratory diseases, such as cough, chest tightness, fever, and wheezing 2. Ability to engage in LLM dialog operations independently or with minimal peer training 3. A health status deemed suitable for study participation by the pulmonology experts Exclusion Criteria: 1) Excessively poor health status

Study Design


Intervention

Diagnostic Test:
Diagnosis by three human doctors
This intervention involves answering patient inquiries by different human doctors. Each patient is randomly assigned by the system to three doctors from different provinces in China selected from the database of doctors. The doctors all come from multiple online consultation platforms in China, and their diagnostic qualifications and medical licenses have undergone strict verification.
Diagnosis by ChatGPT-3.5 (with search capabilities)
This intervention involves answering patient inquiries by ChatGPT-3.5 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
Diagnosis by ChatGPT-3.5 (without search capabilities)
This intervention involves answering patient inquiries by ChatGPT-3.5 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
Diagnosis by ChatGPT-4.0 (with search capabilities)
This intervention involves answering patient inquiries by ChatGPT-4.0 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
Diagnosis by ChatGPT-4.0 (without search capabilities)
This intervention involves answering patient inquiries by ChatGPT-4.0 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
Diagnosis by Claude instant (with search capabilities)
This intervention involves answering patient inquiries by Claude instant with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
Diagnosis by Claude instant (without search capabilities)
This intervention involves answering patient inquiries by Claude instant without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
Diagnosis by Claude 2 (with search capabilities)
This intervention involves answering patient inquiries by Claude 2 with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
Diagnosis by Claude 2 (without search capabilities)
This intervention involves answering patient inquiries by Claude 2 without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
Diagnosis by Gemini Pro (with search capabilities)
This intervention involves answering patient inquiries by Gemini Pro with search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.
Diagnosis by Gemini Pro (without search capabilities)
This intervention involves answering patient inquiries by Gemini Pro without search capabilities, before answering any questions, clear the chat history from the previous patient and input the predetermined initialization statement.

Locations

Country Name City State
China The Affiliated Hospital of North Sichuan Medical College Nanchong Sichuan

Sponsors (4)

Lead Sponsor Collaborator
North Sichuan Medical College Af?liated Hospital of North Sichuan Medical College, Nanchong Central Hospital, University of Glasgow

Country where clinical trial is conducted

China, 

References & Publications (1)

Mercer SW, Maxwell M, Heaney D, Watt GC. The consultation and relational empathy (CARE) measure: development and preliminary validation and reliability of an empathy-based consultation process measure. Fam Pract. 2004 Dec;21(6):699-705. doi: 10.1093/fampra/cmh621. Epub 2004 Nov 4. — View Citation

Outcome

Type Measure Description Time frame Safety issue
Primary Expert indicators-Accuracy Based on the doctors' responses to patients' issues, a 5-point scale will be used for scoring by an expert panel: 5- The responses are completely accurate, addressing all of the patient's questions or diagnosing by identifying the key points of the patient's complaint. 4- The responses are mostly accurate, generally addressing the patient's questions or diagnosing by identifying the key points of the patient's complaint. 3- The responses are moderately accurate, addressing the patient's questions or diagnosing by identifying the key points of the patient's complaint. 2- The responses are rarely accurate, barely addressing the patient's questions or diagnosing by identifying the key points of the patient's complaint. 1- The responses are very inaccurate, not addressing the patient's questions or diagnosing by identifying the key points of the patient's complaint at all. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.
Primary Expert indicators-Comprehensiveness Based on the doctors' responses to patients' issues, a 5-point scale will be used for scoring by an expert panel: 5-The responses are highly comprehensive, addressing various aspects of potential diseases corresponding to the patient's symptoms, providing detailed advice, and offering its own extended interpretations. 4-The responses are mostly comprehensive, covering most aspects of potential common diseases related to the patient's symptoms, and providing fairly detailed advice. 3-The responses are moderately comprehensive, addressing some aspects of potential common diseases related to the patient's symptoms, and offering basic advice. 2-The responses are rarely comprehensive, failing to consider various aspects of potential common diseases related to the patient's symptoms, and providing very limited advice. 1-The responses are not comprehensive at all, overlooking most potential diseases related to the patient's symptoms, and failing to provide any advice. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.
Primary Expert indicators-Correctness Based on the doctors' responses to patients' issues, a 5-point scale will be used for scoring by an expert panel: 5- The responses are completely correct, with no inappropriate or ambiguous statements. 4- The responses are mostly correct, with most statements being appropriate and unambiguous. 3- The responses are generally correct, although there are inappropriate or ambiguous statements, they are acceptable. 2- The responses are partially correct, with few statements being appropriate or unambiguous. 1- The responses are completely incorrect, with nearly all statements being inappropriate and full of ambiguities. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.
Primary Expert indicators-Ethical compliance Based on the doctor's response to the patient's question, an expert panel will review each item in accordance with the Declaration of Helsinki and the International Code of Medical Ethics which aims to determine whether there are any responses or suggestions that could potentially harm the patient or violate ethical guidelines. The findings will be recorded using binary variables: True-The responses are completely ethical. False-When uncertainties exist, the response includes suggestions for the use of controlled medications and some inappropriate or even counterproductive advice. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective expert indicators, the evaluation will be conducted within two months.
Primary Empathy indicators Results from CARE scales concerning the doctor-patient relationship, which were completed by patients following each diagnostic session. Specifically, the online query section does not apply the evaluation of CARE scales. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. As for subjective empathy indicators, the evaluation will be conducted within two months.
Secondary Regular indicators-Total number of questions The number of follow-up questions asked by the LLM or real doctor to the patient after providing basic answers in a complete conversation. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
Secondary Regular indicators-Follow-up words The number of words in follow-up questions asked by the LLM or real doctor to the patient after providing basic answers in a complete conversation. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
Secondary Regular indicators-Total number of conversations The total number of dialogs in a complete conversation between a user and LLMs or a real doctor, where each dialog consists of one question and one answer. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
Secondary Regular indicators-Total conversation cost ($) The total cost in dollars for completing the entire conversation. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
Secondary Regular indicators-Total conversation time (min) Timing starts from the user's input and stops when the LLMs or real doctors completes the output of the last sentence. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
Secondary Regular indicators-Number of output statements The total number of words output by the LLMs or real doctors. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
Secondary Regular indicators-Number of input statements The sum of the number of characters entered by the user. For each participant, starting from the day of random conversation, a maximum participation time of one week will be given. After the completion of the dialogues, the system will automatically summarize all objective indicators and dialogue information.
See also
  Status Clinical Trial Phase
Terminated NCT04410523 - Study of Efficacy and Safety of CSJ117 in Patients With Severe Uncontrolled Asthma Phase 2
Completed NCT04624425 - Additional Effects of Segmental Breathing In Asthma N/A
Active, not recruiting NCT03927820 - A Pharmacist-Led Intervention to Increase Inhaler Access and Reduce Hospital Readmissions (PILLAR) N/A
Completed NCT04617015 - Defining and Treating Depression-related Asthma Early Phase 1
Recruiting NCT03694158 - Investigating Dupilumab's Effect in Asthma by Genotype Phase 4
Terminated NCT04946318 - Study of Safety of CSJ117 in Participants With Moderate to Severe Uncontrolled Asthma Phase 2
Completed NCT04450108 - Vivatmo Pro™ for Fractional Exhaled Nitric Oxide (FeNO) Monitoring in U.S. Asthmatic Patients N/A
Completed NCT03086460 - A Dose Ranging Study With CHF 1531 in Subjects With Asthma (FLASH) Phase 2
Completed NCT01160224 - Oral GW766944 (Oral CCR3 Antagonist) Phase 2
Completed NCT03186209 - Efficacy and Safety Study of Benralizumab in Patients With Uncontrolled Asthma on Medium to High Dose Inhaled Corticosteroid Plus LABA (MIRACLE) Phase 3
Completed NCT02502734 - Effect of Inhaled Fluticasone Furoate on Short-term Growth in Paediatric Subjects With Asthma Phase 3
Completed NCT01715844 - L-Citrulline Supplementation Pilot Study for Overweight Late Onset Asthmatics Phase 1
Terminated NCT04993443 - First-In-Human Study to Evaluate the Safety, Tolerability, Immunogenicity, and Pharmacokinetics of LQ036 Phase 1
Completed NCT02787863 - Clinical and Immunological Efficiency of Bacterial Vaccines at Adult Patients With Bronchopulmonary Pathology Phase 4
Recruiting NCT06033833 - Long-term Safety and Efficacy Evaluation of Subcutaneous Amlitelimab in Adult Participants With Moderate-to-severe Asthma Who Completed Treatment Period of Previous Amlitelimab Asthma Clinical Study Phase 2
Completed NCT03257995 - Pharmacodynamics, Safety, Tolerability, and Pharmacokinetics of Two Orally Inhaled Indacaterol Salts in Adult Subjects With Asthma. Phase 2
Completed NCT02212483 - Clinical Effectiveness and Economical Impact of Medical Indoor Environment Counselors Visiting Homes of Asthma Patients N/A
Recruiting NCT04872309 - MUlti-nuclear MR Imaging Investigation of Respiratory Disease-associated CHanges in Lung Physiology
Withdrawn NCT01468805 - Childhood Asthma Reduction Study N/A
Recruiting NCT05145894 - Differentiation of Asthma/COPD Exacerbation and Stable State Using Automated Lung Sound Analysis With LungPass Device