Clinical Trial Details
— Status: Recruiting
Administrative data
| NCT number |
NCT06109285 |
| Other study ID # |
25901 |
| Secondary ID |
R44EY035596 |
| Status |
Recruiting |
| Phase |
N/A
|
| First received |
|
| Last updated |
|
| Start date |
November 29, 2023 |
| Est. completion date |
August 2024 |
Study information
| Verified date |
April 2024 |
| Source |
Siloam Vision |
| Contact |
Karyn E Jonas, MSN |
| Phone |
5618182161 |
| Email |
karyn[@]siloamvision.com |
| Is FDA regulated |
No |
| Health authority |
|
| Study type |
Interventional
|
Clinical Trial Summary
In this study, the investigators aim to improve image acquisition and preprocessing pipeline
for FDA application. Our previous research has used various methodologies to identify images
to input into the algorithm. The investigators aim to evaluate the effect of image quality
and field of view on the output of the device in addition to performing a precision study to
evaluate the effect of different camera operators on image selection and device output.
Additionally the investigators aim to assist conducting pivotal clinical study for assistive
diagnosis. Based on two previous meetings with the FDA, the clinical study will test the
following hypothesis: use of the i-ROP DL VSS improves the accuracy of plus disease diagnosis
among clinicians performing telemedical review of images. The investigators also aim to
assist in conducting pivotal clinical study of autonomous ROP screening. Preliminary data
suggests that the i-ROP DL system could achieve 100% sensitivity for detection of severe ROP.
The investigators will perform a clinical study to support a revised indication for use as an
autonomous AI screening device. Output of the i-ROP DL system will be compared to masked
telemedicine grading for identification of referral-warranted and treatment-requiring ROP.
Description:
Objectives
This study involves 2 parts. The purposes of the first part involving pivotal validation of
i-ROP DL are:
- To determine the sensitivity and specificity of i-ROP DL for the detection of more than
mild retinopathy of prematurity (MTMROP) in telemedicine setting of infants with birth
weight (BW) less than 1251 g meeting current ROP screening criteria.
- To determine the sensitivity and specificity of i-ROP DL for the detection of treatment
requiring retinopathy of prematurity (TR-ROP) in telemedicine setting of infants with
birth weight (BW) less than 1251 g meeting current ROP screening criteria.
This study is a retrospective, multicenter, observational study to assess the safety and
effectiveness of i-ROP DL in screening for MTMROP and TR-ROP. The study design conforms to
Intent to Screen (ITS) paradigm. The study will select approximately 1200 subjects from the
1285 subjects of the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of
Prematurity (e-ROP) Study. As its primary endpoint, this study is designed to evaluate
sensitivity and specificity of i-ROP DL for detection of MTMROP and TR-ROP of premature
Infants to meet the pre-specified criteria.
The safety and effectiveness of i-ROP DL system are both linked to the ability of
interpreting Retcam images to accurately screen the MTMROP disease. Each study endpoint
therefore addresses both safety and effectiveness. All of the following endpoints are
measured based on the study eye of the subjects.
The co-primary endpoints are:
- Sensitivity and specificity of the i-ROP DL detecting MTMROP at each level of the 1-9
VSS (1, 2, 3, etc.).
- AUC of ROC for detection of MTMROP based on the VSS. The proposed performance targets
are defined at 85.0% sensitivity and specificity for MTMROP, reflecting anticipated
enrollment numbers and prespecified regulatory requirements. In addition to observed
sensitivity and specificity, enrichment corrected sensitivity and specificity will be
calculated using logistic regression to evaluate whether performance would have been
different if the study population had not been enriched with subjects with higher
disease prevalence.
The secondary endpoints are:
- Sensitivity and specificity for detection TR-ROP at each level of the VSS.
- AUC of ROC for detection of TR-ROP based on the VSS.
- The positive and negative predictive values for detection of MTMROP and TR-ROP in an
unenriched population (similar to the intended use population).
The purpose of the second part of this study involves a pivotal reader study and will assess
the readers' accuracy in diagnosing plus disease versus no plus or pre-plus disease with or
without the aid of the i-ROP DL. Ophthalmologists' performance metrics for the following
modalities will be evaluated:
- Standard evaluation following the standard of care process ("without i-ROP DL")
- Evaluation following the standard of care process with the aid of the i-ROP DL ("with
i-ROP DL") This retrospective multi-reader multi-case (MRMC) study will have an enriched
sample of approximately 300 eye cases (1 study eye per subject): 60 plus cases, 120
pre-plus cases and 120 no plus cases. Enrichment is with respect to proportions of plus
cases and pre-plus cases.
The primary objective of this study is to evaluate whether the area under the receiver
operating characteristic (ROC) curve (AUC) based on probability scores of plus disease
statistically significantly non-inferior or superior with the aid of the i-ROP DL versus
without the aid of the i-ROP DL. Multiple secondary endpoints are outlined in the next
section.
The safety and effectiveness of the i-ROP DL are both linked to the ability of
ophthalmologists interpreting images to accurately diagnose the plus disease. Each study
endpoint therefore addresses both safety and effectiveness.
All of the following endpoints are measured without the aid of the i-ROP DL, with the aid of
the i-ROP DL, and their difference.
The co-primary endpoints are:
- Non-inferior per-subject average area under the ROC curve (AUC) with the aid of the
i-ROP DL versus without the aid of the i-ROP DL, based on probability scores at
diagnosis of plus disease.
- Superior per-subject average area under the ROC curve (AUC) with the aid of the i-ROP DL
versus without the aid of the i-ROP DL, based on probability scores at diagnosis of plus
disease.
To maintain the study alpha level of 0.05, the two co-primary endpoints will be tested
sequentially following the gate-keep approach.
The first co-primary endpoint will be considered to have successfully demonstrated safety and
effectiveness of the i-ROP DL if the per- subject average AUC with the aid of the i-ROP DL is
statistically significantly non-inferior to the average AUC without the aid of the i-ROP DL
for diagnosis of plus disease, at the alpha = 0.05 significance level.
If the first co-primary endpoint is statistically significant, the second co-primary endpoint
will be tested. The second co-primary endpoint will be considered to have successfully
demonstrated safety and effectiveness of the i-ROP DL if the per-subject average AUC with the
aid of the i-ROP DL is statistically significantly superior to the average AUC without the
aid of the i-ROP DL for diagnosis of plus disease, at the alpha = 0.05 significance level.
The secondary endpoints are:
• Differences in per-subject average sensitivity and specificity of readers with and without
the aid of the i-ROP DL for classifying plus disease from no plus/pre-plus cases.
Sensitivity and specificity are calculated from reader's response for the three categories
(no plus, pre-plus and plus), using plus as positive.
- Difference in inter-reader agreement with and without the aid of the i-ROP DL for
classifying plus disease from no plus/pre-plus cases.
- Non-inferior per-subject average area under the ROC curve (AUC) with the aid of the
i-ROP DL versus without the aid of the i-ROP DL, based on probability scores at
diagnosis of pre-plus or worse disease.
- Superior per-subject average area under the ROC curve (AUC) with the aid of the i-ROP DL
versus without the aid of the i-ROP DL, based on probability scores at diagnosis of
pre-plus or worse disease.
Background
Retinopathy of prematurity is a disease that develops when babies are born prematurely before
the retina has fully developed [1]. It is a leading cause of childhood blindness worldwide,
but blindness is usually avoidable with accurate diagnosis, and timely treatment of babies
with treatment-requiring disease [1]. Each country has accepted criteria for which babies
require eye examinations. In the United States, all babies born less than 31 weeks
post-menstrual age (PMA) or under 1500 grams require diagnostic eye examinations starting at
31 weeks PMA or 4 weeks after birth, whichever is later [2]. Exams are repeated at 1-2 week
intervals depending on the level of disease identified on examination.
Diagnosis of ROP is made using either ophthalmoscopy (in person examination at the bedside)
or using telemedicine with remote digital fundus imaging [2] [3]. Telemedicine has become an
acceptable standard of care for ROP diagnostic examinations, and in most telemedicine
workflows, exams are repeated weekly to minimize the risk of missing ROP. Whether exams occur
in person or via telemedicine, the ophthalmologist is responsible for documenting the
presence or absence of ROP using the International Classification of ROP (ICROP) guidelines,
including documenting the presence and degree of zone, stage, and plus disease [4].
Current treatment guidelines suggest that urgent treatment (within 48 hours) should occur
when babies develop "type 1" ROP, defined as Zone I stage 3 ROP (with or without plus), or
any zone or stage with plus [5]. Because zone I stage 3 without plus is rare, these
guidelines have established plus disease as the most important clinical feature determining
the need for treatment in ROP. Plus disease is generally defined as dilation and tortuosity
of the posterior retinal vessels in Zone I relative to a standard published photograph.
However, as digital fundus imaging has become more standard, it has become clear that in
practice, the diagnosis of plus disease is not always made consistently, with significant
inter-observer variability. As a result, the ICROP 3rd edition, published in 2021,
acknowledges that pre-plus and plus run a spectrum that is related to the overall severity of
ROP in the eye [4] [5].
Because treatment is urgent when type 1 ROP is diagnosed, when borderline disease is
diagnosed in patients, the ophthalmologist responsible for the care of that baby may suggest
referral for in person ophthalmoscopic examination and possible treatment, to avoid treatment
delay in case of logistical or transportation challenges. For that reason, it is better to
diagnose severe ROP before babies progress to type 1 ROP / plus disease.
The i-ROP DL system is a screening device for use by trained nurses, photographers, or
technicians performing screening eye examinations using the Retcam 3 camera. The output of
the device is a "refer" or "repeat" that is assigned for each patient that reflects the
spectrum of vascular changes in pre-plus and plus disease [6] [7].
The output will be sent to an ophthalmologist for review if "more than mild ROP" is detected
("refer"), using the same images analyzed by the algorithm (no separate exam required). The
clinician can decide further management based on a standard telemedicine workflow. In cases
of "no referral needed", the baby will be scheduled for a follow-up examination in 1 week
(typical for telemedicine programs). This ensures that if a baby progresses to more than mild
ROP over time there are multiple opportunities to detect it and flag it for clinical review.
Preliminary work suggests that babies usually reach the threshold for MTMROP 2-4 weeks prior
to diagnosis of TR-ROP, making the risk of a missed case very low.
The i-ROP DL system is also an assistive diagnostic device for use by the ophthalmologist
performing diagnostic eye examinations using telemedicine review of digital fundus images
using the Retcam 3 camera. The output of the assistive diagnostic device is a "vascular
severity score" (from 1-9) that is assigned at the eye level that reflects the spectrum of
vascular changes in pre-plus and plus disease [6] [7]. The vascular severity score is
calculated based on the underlying probability of a classification of no plus, pre-plus or
plus, and corresponds to the relative likelihood a random expert would diagnose plus disease
(at the extreme end of the spectrum, 9, 100% of experts would diagnose plus disease) [7].
This output has two advantages that may improve the diagnosis of pre-plus and plus disease,
and reduce the risk of late treatment for babies in the context of telemedicine. First,
higher vascular severity is not only associated with a higher likelihood of plus disease, but
also with higher stage and extent of peripheral disease [6] [8], which may or may not be
visible on all the images. On telemedicine review, if the stage of disease is not visible on
peripheral images, the ophthalmologist may want to consider either repeating the images or
referring the baby for ophthalmoscopic examination. In this case, the "positive" output would
not necessarily trigger any therapeutic intervention, but it will aid the ophthalmologist in
making the diagnosis. Second, the vascular severity score will aid in the diagnosis and
accuracy of plus disease by providing an objective metric, anchored to expert diagnosis of
plus disease [6] [7].
Study Design
Part 1: This clinical study is a retrospective, multicenter, observational study to assess
the safety and effectiveness of i-ROP DL in screening for ROP in the telemedicine care
setting. The study design conforms to Intent to Screen (ITS) paradigm.
Part 2: This clinical research is a retrospective, pivotal, multi-reader, multi-case (MRMC)
study with an enriched sample of approximately 60 plus cases (subjects), 120 pre-plus cases
and 120 no plus cases.
- Reference Standard Diagnosis The reference standard diagnosis (RSD) for the 3-class
ground truth (no plus, pre-plus, or plus) will be determined at the eye level by the
majority classification of 5 ROP experts who will independently classify all eyes in the
dataset. In the unlikely case of a 2/2/1 disagreement, the 5 experts will be asked to
regrade the eye exam together until there is a majority (at least 3/5 agree).
- The RSD Process The RSD experts will review all cases selected for inclusion in the
reader study. The RSD experts will complete an eCRF for each case, based on eye images
and available clinical information from the image acquisition site.
- The Reading Process Approximately 16 ophthalmologists will independently perform two
reads on all (approximately 300) cases. Each reader will read each case both without the
aid of the i-ROP DL and then with the aid of the i-ROP DL.
The first reading will recapitulate the current standard of care with an ophthalmologist
evaluating the overall level of ROP severity without the aid of the i-ROP DL using the
International Classification of ROP (ICROP) [1] criteria for zone, stage, and plus disease.
The second reading will evaluate the effect of the i-ROP DL output on plus disease diagnosis
using the same criteria. In order to minimize recall bias, the two reading visits will be
separated by a memory washout period of approximately four weeks.
This clinical research is a retrospective, pivotal, multi-reader, multi-case (MRMC) study
with an enriched sample of approximately 300 cases which will be selected from the library of
telemedicine cases collected under Telemedicine Approaches to Evaluating Acute Phase ROP
(e-ROP) Study.
All subjects in the library previously provided written informed consent for inclusion in the
e-ROP study. As part of the consent process, subjects agreed that image data and supporting
documentation could be used for future research and investigations. Only deidentified data
and retinal images will be used in this study.
Evaluable cases are defined as subjects who met all protocol case inclusion/exclusion
criteria, and who have a complete set of adequate-quality eye examination. The quality of
each image is categorized as good, adequate, poor, or missing, based on focus, clarity, and
field definition.
Measures to ensure subject and reader confidentiality include assignment of unique study and
reader ID numbers.
- Sample Size and Case Mix Enriched sample size of approximately 300 cases (subjects) with
the following case mix: 60 plus cases, 120 pre-plus cases and 120 no plus cases.
- Case Selection and Targeted Distribution The independent statistician will select cases
for this pivotal reader study and cases will be randomly selected to provide
representative distributions of race across enrolling sites, to the extent that this can
be accomplished using cases in the case library.
Inclusion and Exclusion Criteria All cases for this pivotal validation study will meet the
following eligibility inclusion and exclusion criteria. (see also Appendix A for inclusion
and exclusion criteria for protocol e-ROP). Only subjects who were consented, met all
inclusion criteria and none of the exclusion criteria, and completed the needed work-up to
establish the reference standard diagnosis ground truth will be considered evaluable.
The inclusion criteria for all cases in the study are as follows:
- Eligible subjects under protocol e-ROP, defined as Infants with birth weight (BW) less
than 1251 g meeting current ROP screening.
- Eye cases meet the image quality requirements to establish the reference standard
diagnosis of plus, pre-plus and no plus cases.
- Subjects have known reference standard diagnosis ground truth. If the reference standard
diagnosis process cannot create an RSD label for plus disease, the eye will not be
included in the study.
- Image acquired by the following device: RetCam Ophthalmic Imaging Systems 3 (Natus
Medical Incorporated).
The exclusion criteria for all cases in the study are as follows:
- Subjects who are in violation of protocol e-ROP.
- Subjects who are in violation of inclusion criteria of this protocol. Case selection and
Targeted Distribution All subjects meeting inclusion and exclusion criteria from the
e-ROP study will be included in this study. e-ROP study represents the natural
occurrence of the plus and pre-plus disease cases in the clinical environment. There is
no data enrichment within e-ROP study.
Recruitment Methods
This study will be conducted with board-certified ophthalmologists of varying experience
levels, with at least one year of experience performing ROP examinations. These
ophthalmologists will not be associated with the acquisition or review of study cases and did
not participate in the RSD process. A survey questionnaire will be sent to prospective
readers to determine the fraction of their professional time devoted to ROP diagnosis.
Investigators will invite prospective readers through the listserv of the American Academy of
Pediatric Ophthalmology and Strabismus and the Retina Society. Approximately 16
board-certified ophthalmologists, with a range of experience based on the years of
experience, will be selected to read and score the study cases.
ROP reading experience will be categorized into three groups:
- High Experience Reader - defined as ophthalmologists who have >10 years' experience of
reading ROP images.
- Medium Experience Reader - defined as ophthalmologists who have 5-10 years' experience
of reading ROP images.
- Low Experience Reader - defined as ophthalmologists who have 1-4 years' experience of
reading ROP images.
The inclusion criteria (qualification criteria) for all readers in the study are as follows:
- U.S. board-eligible/board-certified and licensed Medical Doctor, Doctor of
Ophthalmology. (American Board of Ophthalmology or equivalent certification)
- Signed reader study agreement.
- Signed informed consent.
- Successful training on the study protocol and the use of the study software.
Consent Process
Each reader must provide written informed consent before participating in the reader study of
this protocol. Because readers are located all of the world, the investigators will provide
electronic consent through docusign.
Procedures Involved The Reading Process Purpose The purpose of the reading process is for
readers to provide the assessment categories of plus, pre-plus, and no plus, probability
scores of plus and pre-plus or worse on the electronic case report forms for their
interpretation of the study cases with and without the aid of the i-ROP DL.
Conduct of the Reading Process Following the successful completion of the reader training,
the readers will independently review and score each of the cases of the reader study case
set. Clinical information about the subjects including birth weight, gestational age, and
post-menstrual age at eye exam will be provided to the readers. Each reader will work at a
designated reading station and will independently perform two reading sections on all cases.
Each reader will read each case both as a standard reading, and a reading with the aid of the
i-ROP DL. Each reader will read all cases as the standard reading during the first visit. The
reader will then read the cases with the aid of the i-ROP DL during the second visit. In
order to minimize memory recall bias, the two reading sections will be separated by a memory
washout period of approximately four weeks.
The reading station will provide preloaded cases from the server through secured web
connection. The reading time allowed per case will not be limited. A lunch break will be
offered during each reading session, and readers will be encouraged to take a 5-10 minute
break each hour, as needed. Each reader will enter their interpretations directly into the
electronic case report form. Each reader will sign an attestation form verifying the
documented information.
Core Reading Center The reading process will be conducted remotely at reader's home or
office, as they would be in intended use case of telemedicine evaluation.
Preparation of Cases for Interpretation To maintain subject confidentiality, no identifying
subject data will be displayed. Unique image identifiers associated with the case ID number
will be visible on each image for tracking purposes. To maintain reader confidentiality, a
unique ID number will be assigned to each reader participating in the pivotal reader study
for entry on the electronic case report form.
Information collected during the study will be reported in such a way that it precluded
identification of any image acquisition centers or readers' performance.
