Clinical Trials Logo

Clinical Trial Details — Status: Recruiting

Administrative data

NCT number NCT05272189
Other study ID # 2007P000646-B
Secondary ID
Status Recruiting
Phase N/A
First received
Last updated
Start date January 1, 2020
Est. completion date January 1, 2025

Study information

Verified date July 2023
Source Brigham and Women's Hospital
Contact Jeremy M Wolfe, PhD
Phone 6178511166
Email jwolfe@bwh.harvard.edu
Is FDA regulated No
Health authority
Study type Interventional

Clinical Trial Summary

The study is one part of a "bundle" of experiments that constitute Project Three of a National Eye Institute grant. Project Three includes a series of experiments that investigate how changing the input from a simulated AI can affect the decisions made by human observers in a two-alternative forced choice task (like the decision to recall a woman for further examination in mammography). HAICT 7, the experiment described here, investigates how changing prevalence affects human performance when AI is used as a Second Reader.


Description:

This text is the text of the pre-registration for the HAICT 7 experiment as described on the Open Science Framework. https://osf.io/hngu4/ NOTE: This study is representative of studies conducted in Project 3 of this grant. There are multiple experiments in the bundle of experiments represented by Project 3 but it is not possible to register a bundle of studies on CT.gov. NOTE: Since the pronoun comment is advisory, we will leave it for now. Human-AI Collaboration Tester (HAICT) Exp. 7 (lightly edited from OSF) 1. Data collection. Have any data been collected for this study already? (Yes/No) yes 2. Hypothesis. What's the main question being asked or hypothesis being tested in this study? Background: In a variety of search experiments, both basic and clinical, the data have been consistent with a situation where the variability of the signal (or target) is greater than the variability of the noise (distractors). The classic sign of this is a zROC function with a slope < 1 - typically around 0.6. A slope of 1.0 is indicative of an equal variance 2AFC task. For the HAICT task that we have been testing, we would expect equal variance, but we think it would be worth checking so we will systematically vary prevalence which will shift criterion. That will sweep out an ROC curve that we can examine. We will also test the Second Reader faux-AI in order to determine if low prevalence makes Second Reader worse. - (H1): We expect to replicate the finding that human criteria become more conservative as prevalence declines. - (H2): We predict that the slope of the resulting zROC will be 1.0. - (H3): We hypothesize that low prevalence will make Second Reader AI less effective because the positive predictive value of its comments will be low. 1. Dependent variable. Describe the key dependent variable(s) specifying how they will be measured. The main dependent variables of interest are accuracy (and the signal detection derivatives of accuracy, d' and c), reaction time, and subjective ratings on the survey following each block. 2. Conditions. How many and which conditions will participants be assigned to? This series of experiments investigates how changing the input from a simulated AI can affect the decisions made by human observers in a two-alternative forced choice task (like the decision to recall a woman for further examination in mammography). We have developed a paradigm called the Human-AI Collaboration Tester (HAICT) that allows for efficient testing of interactions between a human and a simulated AI. The observers' task in all conditions is to give a 2AFC decision about whether a stimulus is "bad" or "not bad." To use language roughly mimicking a medical diagnosis, each stimulus is referred to as a "case." Observers are asked to make a 2AFC decision about arrays of colored shapes. The decision is made based on the predominant color of the case. The number of elements of each color are drawn from one of two normal distributions, one for positive (bad) stimuli and the other for negative (not bad) stimuli. The results from previous HAICT experiments (3 and 4) showed that human performance in the Second Reader condition drops off significantly at low prevalence. Performance in the Second Reader condition was better than Baseline when the prevalence of bad cases was 50% but was significantly worse than Baseline when prevalence was only 10%. In this experiment, we manipulate the prevalence of "bad" cases in the Second Reader and Baseline conditions. Four different prevalence rates will be tested - 10%, 33%, 67%, and 90%. Observers will complete 8 blocks (2 AI rules x 4 prevalence rates), and block order is random. AI rules to be tested: 1. Baseline - No AI input. Observer classifies each case as "bad" or "not" bad on their own. 2. Second Reader - The observer makes an initial decision about every case. The AI silently classifies stimuli using a conservative criterion (c = 0.5). The logic for the conservative criterion is that the second reader is being used to cut down on false positive responses and so it is intended to question positive human responses that might be marginal. If the observer and AI disagree, then the AI informs the human observer. The observer is then given a chance to either change their response or go with their first opinion. As in Experiments 1-5, the AI d-prime is fixed at 2.2. Feedback is known to increase the prevalence effect, so feedback will be given in both the practice and the test trials. Observers will complete 20 practice trials and 200 test trials in each block. Immediately after each block is completed, observers will be shown a summary of their performance. After the Second Reader blocks, they will also be asked to answer three subjective questions about the usefulness of the AI (see "Files" for more details). 3. Analyses. Specify exactly which analyses you will conduct to examine the main question/hypothesis. First, we summarize the number of hits, true negatives, misses, and false alarms in each block. From this, we can calculate the accuracy, the positive predictive value, sensitivity (d-prime), and the criterion for each observer under each of the different conditions. Given measures of performance at 4 levels of prevalence, we can estimate the ROC curve (pHit x pFA) and the zROC function (zHit x zFA). We will test the hypothesis that the slope of the zROC is equal to 1 (the consequence of an equal variance 2AFC task). 4. More analyses. Any secondary analyses? We will look to see if the observers' subjective opinions about the AI are correlated with variables such as the empirical d-prime, or the positive predictive value. 5. Sample size. How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined. We will test 12 observers. This is consistent with the sample sizes of previous experiments. 6. Other. Is there anything else that you would like to pre-register? (e.g., data exclusions, variables collected for exploratory purposes, unusual analyses planned?) N/A


Recruitment information / eligibility

Status Recruiting
Enrollment 15
Est. completion date January 1, 2025
Est. primary completion date August 1, 2024
Accepts healthy volunteers Accepts Healthy Volunteers
Gender All
Age group 18 Years and older
Eligibility Inclusion Criteria: - - All welcome to enroll on line Exclusion Criteria: - Must pass the Ishihara color vision screening test - 20/25 vision (with correction)

Study Design


Related Conditions & MeSH terms


Intervention

Behavioral:
Simulated Second Reader AI
In this experiment, in some conditions, the participant makes their decision in the presence of information about a simulated artificial intelligence decision.
Target Prevalence
The frequency with which targets are presented varies from 10% to 90%

Locations

Country Name City State
United States Visual Attention Lab / Brigham and Women's Hospital Boston Massachusetts

Sponsors (1)

Lead Sponsor Collaborator
Brigham and Women's Hospital

Country where clinical trial is conducted

United States, 

Outcome

Type Measure Description Time frame Safety issue
Primary D' D' (d-prime) is the signal detection theory measure of the level of performance on a task. Up to one week
Primary Criterion Criterion is the signal detection theory measure of the bias ("liberal" or "conservative") of observers' decisions Up to one week
Secondary Reaction Time This is the measure of how long it takes to make a response. Up to one week
See also
  Status Clinical Trial Phase
Completed NCT01799291 - Cognitive De-Biasing and the Assessment of Pediatric Bipolar Disorder N/A
Recruiting NCT03886025 - Combined Anodal Transcranial Direct Current Stimulation (tDCS) and Cognitive Training and Decision-making N/A
Completed NCT03185715 - Oocyte Recipient's Decision in the Number of Embryos to be Transferred N/A
Completed NCT05974943 - The Effect of Problem Solving and Decision Making Training Given to Nurses Managers N/A
Terminated NCT05024903 - A Novel E-health Platform Phase 2/Phase 3
Completed NCT04509063 - Investigating Public Enthusiasm for Mammography Screening in Denmark N/A
Completed NCT03762239 - Effect of Air Pollution on the Cognitive Function of Adolescents N/A
Not yet recruiting NCT06081842 - Strengthening Contraceptive Counseling Services: Research Protocol for a Multi-phase Complex Intervention in Pakistan and Nigeria N/A
Completed NCT01190488 - Feasibility Of An Advanced Care Decision Aid Among Patients And Physicians_Matlock N/A
Active, not recruiting NCT04466865 - A Communication Tool to Assist Older Adults Facing Dialysis Choices N/A
Active, not recruiting NCT06046625 - Needs and Preferences of Patients With Head-neck Cutaneous SCC
Completed NCT03181841 - Effects of PF-06412562 on Value-based Decision-making in Healthy Individuals Phase 1
Recruiting NCT03554694 - Gut-brain Axis, Brain Function, and Behaviour. N/A
Recruiting NCT05201534 - Interventions in Mathematics and Cognitive Skills N/A
Completed NCT04000802 - Evaluation of the Decision-making During Multidisciplinary Meetings for Digestive Cancers
Completed NCT04034979 - Evaluation of a Decision Aid About Life-sustaining Therapies N/A
Active, not recruiting NCT05173922 - "Safety in Dementia": An Online Caregiver Intervention N/A
Withdrawn NCT01098864 - Transcranial Magnetic Stimulation of the Prefrontal Cortex: Effects on Risky Decision Making and Temporal Discounting. N/A
Recruiting NCT05364801 - Impact of Anxiety on the Decision-making Processes of Anesthesiologists
Completed NCT06355999 - Levari Exp.20: 2D - Partial Feedback N/A