A Study to Assess the Impact of an Artificial Intelligence (AI) System on Chest X-ray Reporting

Artificial Intelligence Clinical Trial

Official title:

A Prospective Study to Assess the Impact of an Artificial Intelligence System on Reporting of Chest X-rays, Evaluate the Ability of AI Driven Worklists to Improve Reporting Times and Improve Same Day CT Pathway for Suspected Lung Cancer

Clinical Trial Details — Status: Not yet recruiting

Administrative data

• NCT number: NCT05489471
• Other study ID #: R2757
• Status: Not yet recruiting
• Start date: May 2023
• Est. completion date: July 2023

Study information

• Verified date: March 2023
• Source: Hull University Teaching Hospitals NHS Trust
• Contact: gerard avery, FRCR
• Phone: +441482623205
• Email: ged.avery1[@]nhs.net
• Is FDA regulated: No
• Study type: Observational

Clinical Trial Summary

The study has an initial short retrospective component but is predominately a prospective study with two main parts.

Initially during a 1 month period whilst reporters are familiarising themselves with the software two local databases will be reviewed by the AI software:

• A training set of 100 chest X-rays (CXR) some of which contain nodules and is used as a training tool with previously documented radiologist performance.

• A set of previously reported radiographs in patients referred by the reporter for CT, ground truth created from the prior CT report and review by two radiologists if required.

This will allow comparison of stand-alone radiologist and AI performance

This is followed by a 6 month period involving multiple groups of reporters and approximately 20,000 cases looking at the impact of an AI system which assesses 10 abnormalities on chest X-ray and reporting on the sensitivity for detection of lesions and its impact on reporter confidence. Specifically the investigators would look at:

• Missed finding by AI, but detected by reporter

• Correctly detected finding by AI

• Missed finding by the reporter but detected by AI

• Finding detected by AI but disputed by the reporter

■ AI's impact on

• Radiological report

• Further recommended imaging

• Altering patient management

• improvement in report confidence as perceived by reporter

A subsequent 3 month period looking at the impact of AI produced worklists on report turnaround times and the patient pathway from chest X-ray to CT. the investigators would specifically look at:

• number of nodules detected

• number of CXRs recommended for follow up CT

• time taken from CXR to CT

• number of lung cancers detected after CT[1]

• Time to report, measured as previously from PACS and reporting software data

The population to be studied will be all patients over 16 years of age referred by their General Practitioner to Hull University Hospitals NHS Trust for a chest radiograph and any chest radiograph performed in the Hull Royal Infirmary ED radiology for patients over 16 years of age during the 6 month study period. The ED department images patients from the emergency department and in-patients within the hospital.

All radiographs will be reviewed initially without review of the AI information and then using the additional images. Reporters will mark the effect of the AI on their decision. All disagreements between the reporter and the AI will be reviewed by senior reporters and a consensus decision made.

Description:

A single centre prospective study, in which data will be collected from Hull University Teaching Hospitals NHS Trust (HUTH).

GP chest X-rays take place in the two main hospital sites of the Trust but also at satellite units in the local community. All of these use the same radiology information system (RIS) and chest X-rays are automatically stored on the HUTH picture archiving and communication system (PACS).

The radiology unit in the Emergency Department at Hull Royal Infirmary undertakes chest X-rays for the emergency patients but also in-patients within the hospital.

All chest X-rays will be booked into the RIS, performed and sent to the PACS system as normal. Any chest X-ray performed on a patient of 16 years of age or older from either of the above sets will be automatically transferred to the AI server and once processed the AI report will automatically transfer into the same PACS folder as the original film.

Reporting of the chest X-ray and review of the AI information will take place in the normal reporting sessions undertaken within the radiology department by all grades of staff.

The radiology department does currently outsource some plain film reporting, however for the duration of the study none of the above examinations will be out-sourced and all reporting will be undertaken by HUTH radiology team members.

Chest X-ray reporting is undertaken by a range of radiology staff:

• Consultant Radiologists

• Registrar Radiologists in training pre-FRCR - Fellow of Royal College of Radiology examination post FRCR

• Reporting Radiographers The radiology department currently has a staff of 46 Consultants, 26 Specialist registrars and 8 reporting radiographers. Although not all Consultants or radiographers undertake chest X-ray reporting.

The study will involve all groups of reporters and the investigators will assess the responses of each group as well as the overall performance. Participants will be identified by group but not individually.

The study is split into three phases.

Phase One:

This will occur immediately after the integration of the AI system to the HUTH PACS. There will be a one month period where reporters gain experience in using the software.

During this time two local databases will be reviewed with the software:

• a training set of 100 chest X-ray cases in which some pulmonary nodules are present with corresponding CT scans.

• A set of previously reported chest X-rays requested by General Practitioners (GP's) which were referred by the reporter for urgent CT.

The results of these will be reviewed by a small group of reporters. The reason for these reviews is two-fold:

• To provide an overview of the software which may inform the instructions for phase two.

• To allow a comparison between previously documented reporter performance and AI performance prior to the commencement of the prospective study.

Phase Two:

This will last 6 months and will comprise all chest X-rays for patients over 16 years from either a GP referral or performed in the Emergency department (ED) of the acute hospital, which includes Accident and Emergency attendances and in-patient studies. These will be sent to the AI server for evaluation, the returning data will be available in the PACS folder with the original image. This will occur prior to the film being available for reporting.

It is estimated that this phase will include approximately 20,000 examinations in a roughly equal split. Calendar year 2019: GP 23,287 examinations and ED 22,042 examinations.

GP films were chosen as these studies are often the first examinations to raise concerns regarding lung cancer, the third phase includes implementing AI generated worklists for reporting of suspected lung nodules/cancer and evaluating the impact on same day CT for this group.

ED films as it was thought these would be the best data set to evaluate the ten findings detected by the software and the introduction of AI generated worklists in phase 3 could lead to faster reporting of abnormal studies.

The reporters will review the original chest x-ray and create a report. They will then review the AI data and decide how this would affect their report, this will then be recorded.[2] The AI looks for ten different abnormalities on each chest X-ray and produces a heat map and percentage confidence score if it detects an abnormality. A single examination could have no AI finding or multiple AI findings.

The reporters will enter their assessment of the AI for each film reported at the time of reporting.

• For each finding present in the chest radiograph and/or the AI output, readers will record:

• Missed finding by AI, but detected by reporter

• Correctly detected finding by AI

• Missed finding by the reporter but detected by AI

• Finding detected by AI but disputed by the reporter

• AI's impact on

• Radiological report

• helpful

• not helpful

• neutral

• Further recommended imaging

• significant

• not significant

• Altering patient management

• significant

• not significant

The investigators will also assess whether AI has increased confidence in reporting an abnormality or reporting a study as normal.

Any film where the reporter disagrees with the AI will immediately be placed into a PACS worklist and reviewed by a senior reporter/reporters and a consensus agreed. The result will be entered into the database.

If a patient is advised to have a follow-up chest X-ray or CT this will be recorded. These follow-up examinations will be reviewed by a small group of reporters with repeat software analysis of any chest X-rays performed.

The AI software only reviews the current image while the reporter has access to any earlier examinations which may be important in deciding on an abnormality, this will be recorded to assess its effects.

The reporter will have an excel spreadsheet available on the PACS reporting workstation through a link to the shared departmental drive with fields for their reporting group, referral type, accession number of the exam - which allows review of the individual examination in case of disagreements or review with later examinations, presence of earlier films, a separate field for each AI abnormality and fields for follow-up chest X-ray or CT. This will be completed at the time of reporting.

At the end of the reporting session the spreadsheet will be uploaded to a central Trust server, held in a departmental folder and integrated from there to a separately stored central database. Only members of the radiology department will have access to the departmental folder and the central database will be held separately, accessible to the data manager and research management team. Once uploaded into the master database the individual data will be deleted from the radiology folder. No data will be stored on the PACS workstations after the reporting session is completed and the file transferred.

The data will be stored and processed exclusively within HUTH A copy of the master database will be made replacing the accession number with an anonymous identifier. This database will be used for analysis.

Phase Three:

This will last 3 months and will comprise the development of AI produced worklists.

A GP worklist highlighting abnormalities which may represent malignancy will allow hot reporting and same day CT if required. The changes in time delay from the chest X-ray to the CT will be compared between the current system with radiographer triage of the x-rays and the AI triage.

An ED worklist will be produced for AI positive studies and reporting times for these will be compared to the current system where there is no filtering of chest x-rays.

All AI negative studies will be reported. The reporting time assessments will be made from routine data within the on the RIS and PACS systems with no patient identifiable data being included in the assessments.

The data will be collected through PACS and the data will be processed locally in a dedicated server.

Recruitment information / eligibility

• Status: Not yet recruiting
• Enrollment: 20000
• Est. completion date: July 2023
• Est. primary completion date: July 2023
• Gender: All
• Age group: 16 Years and older

Eligibility

Inclusion Criteria:

• patient 16 years or older

• Posterior-anterior and Anterior-posterior chest radiographs

• Requested by General Practitioners or performed in the Emergency Department radiology unit

Exclusion Criteria:

• Patients under 16 years of age

• lateral films

• Chest radiographs which are of suboptimal quality, to an extent that it is deemed uninterpretable by the reporter

Related Conditions & MeSH terms

• Artificial Intelligence

Intervention

Other:
• Artificial intelligence review
The AI looks for ten different abnormalities on each chest X-ray and produces a heat map and percentage confidence score if it detects an abnormality.

Locations

Country	Name	City	State
n/a

Sponsors (2)

Lead Sponsor	Collaborator
Hull University Teaching Hospitals NHS Trust	Lunit Inc.

References & Publications (21)

Hwang EJ, Hong JH, Lee KH, Kim JI, Nam JG, Kim DS, Choi H, Yoo SJ, Goo JM, Park CM. Deep learning algorithm for surveillance of pneumothorax after lung biopsy: a multicenter diagnostic cohort study. Eur Radiol. 2020 Jul;30(7):3660-3671. doi: 10.1007/s00330-020-06771-3. Epub 2020 Mar 11. — View Citation

Hwang EJ, Kim H, Yoon SH, Goo JM, Park CM. Implementation of a Deep Learning-Based Computer-Aided Detection System for the Interpretation of Chest Radiographs in Patients Suspected for COVID-19. Korean J Radiol. 2020 Oct;21(10):1150-1160. doi: 10.3348/kjr.2020.0536. Epub 2020 Jul 17. — View Citation

Hwang EJ, Kim KB, Kim JY, Lim JK, Nam JG, Choi H, Kim H, Yoon SH, Goo JM, Park CM. COVID-19 pneumonia on chest X-rays: Performance of a deep learning-based computer-aided detection system. PLoS One. 2021 Jun 7;16(6):e0252440. doi: 10.1371/journal.pone.0252440. eCollection 2021. — View Citation

Hwang EJ, Lee JH, Kim JH, Lim WH, Goo JM, Park CM. Deep learning computer-aided detection system for pneumonia in febrile neutropenia patients: a diagnostic cohort study. BMC Pulm Med. 2021 Dec 7;21(1):406. doi: 10.1186/s12890-021-01768-0. — View Citation

Hwang EJ, Nam JG, Lim WH, Park SJ, Jeong YS, Kang JH, Hong EK, Kim TM, Goo JM, Park S, Kim KH, Park CM. Deep Learning for Chest Radiograph Diagnosis in the Emergency Department. Radiology. 2019 Dec;293(3):573-580. doi: 10.1148/radiol.2019191225. Epub 2019 Oct 22. — View Citation

Hwang EJ, Park S, Jin KN, Kim JI, Choi SY, Lee JH, Goo JM, Aum J, Yim JJ, Cohen JG, Ferretti GR, Park CM; DLAD Development and Evaluation Group. Development and Validation of a Deep Learning-Based Automated Detection Algorithm for Major Thoracic Diseases on Chest Radiographs. JAMA Netw Open. 2019 Mar 1;2(3):e191095. doi: 10.1001/jamanetworkopen.2019.1095. Erratum In: JAMA Netw Open. 2019 Apr 5;2(4):e193260. — View Citation

Jang S, Song H, Shin YJ, Kim J, Kim J, Lee KW, Lee SS, Lee W, Lee S, Lee KH. Deep Learning-based Automatic Detection Algorithm for Reducing Overlooked Lung Cancers on Chest Radiographs. Radiology. 2020 Sep;296(3):652-661. doi: 10.1148/radiol.2020200165. Epub 2020 Jul 21. Erratum In: Radiology. 2022 Feb;302(2):E10. — View Citation

Jang SB, Lee SH, Lee DE, Park SY, Kim JK, Cho JW, Cho J, Kim KB, Park B, Park J, Lim JK. Deep-learning algorithms for the interpretation of chest radiographs to aid in the triage of COVID-19 patients: A multicenter retrospective study. PLoS One. 2020 Nov 24;15(11):e0242759. doi: 10.1371/journal.pone.0242759. eCollection 2020. — View Citation

Jin KN, Kim EY, Kim YJ, Lee GP, Kim H, Oh S, Kim YS, Han JH, Cho YJ. Diagnostic effect of artificial intelligence solution for referable thoracic abnormalities on chest radiography: a multicenter respiratory outpatient diagnostic cohort study. Eur Radiol. 2022 May;32(5):3469-3479. doi: 10.1007/s00330-021-08397-5. Epub 2022 Jan 1. — View Citation

Jones CM, Danaher L, Milne MR, Tang C, Seah J, Oakden-Rayner L, Johnson A, Buchlak QD, Esmaili N. Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study. BMJ Open. 2021 Dec 20;11(12):e052902. doi: 10.1136/bmjopen-2021-052902. — View Citation

Kim EY, Kim YJ, Choi WJ, Lee GP, Choi YR, Jin KN, Cho YJ. Performance of a deep-learning algorithm for referable thoracic abnormalities on chest radiographs: A multicenter study of a health screening cohort. PLoS One. 2021 Feb 19;16(2):e0246472. doi: 10.1371/journal.pone.0246472. eCollection 2021. Erratum In: PLoS One. 2021 Apr 28;16(4):e0251045. — View Citation

Kim JH, Han SG, Cho A, Shin HJ, Baek SE. Effect of deep learning-based assistive technology use on chest radiograph interpretation by emergency department physicians: a prospective interventional simulation-based study. BMC Med Inform Decis Mak. 2021 Nov 8;21(1):311. doi: 10.1186/s12911-021-01679-4. — View Citation

Kim JH, Kim JY, Kim GH, Kang D, Kim IJ, Seo J, Andrews JR, Park CM. Clinical Validation of a Deep Learning Algorithm for Detection of Pneumonia on Chest Radiographs in Emergency Department Patients with Acute Febrile Respiratory Illness. J Clin Med. 2020 Jun 24;9(6):1981. doi: 10.3390/jcm9061981. — View Citation

Koo YH, Shin KE, Park JS, Lee JW, Byun S, Lee H. Extravalidation and reproducibility results of a commercial deep learning-based automatic detection algorithm for pulmonary nodules on chest radiographs at tertiary hospital. J Med Imaging Radiat Oncol. 2021 Feb;65(1):15-22. doi: 10.1111/1754-9485.13105. Epub 2020 Oct 8. — View Citation

Lee JH, Sun HY, Park S, Kim H, Hwang EJ, Goo JM, Park CM. Performance of a Deep Learning Algorithm Compared with Radiologic Interpretation for Lung Cancer Detection on Chest Radiographs in a Health Screening Population. Radiology. 2020 Dec;297(3):687-696. doi: 10.1148/radiol.2020201240. Epub 2020 Sep 22. — View Citation

Nam JG, Hwang EJ, Kim DS, Yoo SJ, Choi H, Goo JM, Park CM. Undetected Lung Cancer at Posteroanterior Chest Radiography: Potential Role of a Deep Learning-based Detection Algorithm. Radiol Cardiothorac Imaging. 2020 Dec 10;2(6):e190222. doi: 10.1148/ryct.2020190222. eCollection 2020 Dec. — View Citation

Nam JG, Kim M, Park J, Hwang EJ, Lee JH, Hong JH, Goo JM, Park CM. Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs. Eur Respir J. 2021 May 20;57(5):2003061. doi: 10.1183/13993003.03061-2020. Print 2021 May. — View Citation

Nam JG, Park S, Hwang EJ, Lee JH, Jin KN, Lim KY, Vu TH, Sohn JH, Hwang S, Goo JM, Park CM. Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. Radiology. 2019 Jan;290(1):218-228. doi: 10.1148/radiol.2018180237. Epub 2018 Sep 25. — View Citation

Turkington PM, Kennan N, Greenstone MA. Misinterpretation of the chest x ray as a factor in the delayed diagnosis of lung cancer. Postgrad Med J. 2002 Mar;78(917):158-60. doi: 10.1136/pmj.78.917.158. — View Citation

Yoo H, Kim KH, Singh R, Digumarthy SR, Kalra MK. Validation of a Deep Learning Algorithm for the Detection of Malignant Pulmonary Nodules in Chest Radiographs. JAMA Netw Open. 2020 Sep 1;3(9):e2017135. doi: 10.1001/jamanetworkopen.2020.17135. — View Citation

Yoo H, Lee SH, Arru CD, Doda Khera R, Singh R, Siebert S, Kim D, Lee Y, Park JH, Eom HJ, Digumarthy SR, Kalra MK. AI-based improvement in lung cancer detection on chest radiographs: results of a multi-reader study in NLST dataset. Eur Radiol. 2021 Dec;31(12):9664-9674. doi: 10.1007/s00330-021-08074-7. Epub 2021 Jun 4. — View Citation

* Note: There are 21 references in all — Click here to view all references

Outcome

Type	Measure	Description	Time frame	Safety issue
Primary	Radiologist performance review	To demonstrate AI can help to improve the radiologist performance in terms of missed finding by radiologist detected by AI ( as a percentage error rate)	six months
Secondary	Lung cancer detection	The number of nodules and cancers detected only by AI (as a percentage of overall number of nodules/tumours detected).	six months
Secondary	Lung cancer pathway improvement	The time between the chest X-ray and CT scan for suspected cancers using AI generated worklists will be compared to historic time data predating the AI software .	three months
Secondary	Report turnaround times improvement	The time between abnormal chest X-ray being performed and reported in an AI driven worklist will be compared to historic data predating the AI system	three months