Clinical Trial Details
— Status: Recruiting
Administrative data
| NCT number |
NCT05914220 |
| Other study ID # |
INGN20RM641 |
| Secondary ID |
|
| Status |
Recruiting |
| Phase |
|
| First received |
|
| Last updated |
|
| Start date |
April 1, 2023 |
| Est. completion date |
August 31, 2024 |
Study information
| Verified date |
June 2023 |
| Source |
NHS Greater Glasgow and Clyde |
| Contact |
Jacqueline Anderson |
| Email |
Jacqueline.Anderson[@]ggc.scot.nhs.uk |
| Is FDA regulated |
No |
| Health authority |
|
| Study type |
Observational
|
Clinical Trial Summary
DYNAMIC AI is an MHRA-regulated medical device trial that will examine the feasibility of
using AI predictive models within COPD multi-disciplinary team meetings, to allow clinicians
to prioritise and optimise COPD management.
Description:
Background: COPD need, challenge and opportunity.
Chronic obstructive pulmonary disease (COPD) is a global healthcare challenge. Whilst
care-quality gaps exist across the continuum of patient's with COPD's presentation, COPD
exacerbations are responsible for a large proportion of the disease-burden, adverse outcomes
and healthcare costs. People with COPD prioritise the avoidance of exacerbations and
resultant hospitalisations. The Patient Charter for COPD (March 2021), called for proactive,
preventative management to reduce the risk of exacerbations and premature death1. Digital
transformation with co-designed digital tools and emerging innovations such as wearable
sensors and AI-based predictive modelling offers the opportunity to address these COPD
care-quality gaps and achieve this care re-orientation.
DYNAMIC: NHS GG&C COPD digital service transformation, implementation and evidence.
'DYNAMIC' (Digital Innovation with Remote Management and Predictive Modelling to Integrate
COPD Care) program commenced in 2018. This was based on an innovation partnership between NHS
GG&C respiratory medicine and West of Scotland innovation teams and with LenusHealth (then
StormID). Initial funding was from a digital health technology catalyst award from
InnovateUK. Subsequent Scottish Government Technology Enabled Care (TEC) Program funding has
continued the service and allowed scale-up provision.
Our vision was to initiate digital transformation of a COPD service with the development,
implementation, and evaluation of co-designed digital tools. We co-designed the "LenusCOPD'
patient and clinician web applications and support website based on cycles of user experience
testing. We aimed to establish tools which would be sustainably used by patients during
follow-up, and support COPD co-management with an anticipation of achieving reduction in
respiratory-related admissions and occupied bed days in a high-risk COPD cohort. Evidence was
gathered in the 'RECEIVER' observational cohort study and subsequently in the scale-up
'DYNAMIC-SCOT' service evaluation.
Key results from RECEIVER / DYNAMIC-SCOT2:
- Participants continue to use the LenusCOPD patient app, with an average of 3.5
interactions per person per week sustained >1-year post-onboarding.
- COPD-related hospital admissions and occupied bed days were reduced following LenusCOPD
onboarding in participants with a history of a severe exacerbation in the previous year,
with a median time to readmission of 380 days compared with 50 days in a contemporary
matched control patient cohort.
- Median time to admission or death was 43 days in control, 338 days in RECEIVER and 400
days in DYNAMIC-SCOT participants with a severe exacerbation in the preceding year.
- The 12-month risk of admission or death was 74% in control patients, 53% in RECEIVER and
47% in the DYNAMIC-SCOT sub-cohort participants.
These feasibility and utility results support scale-up adoption of these digital tools.
Scale-up provision of the core LenusCOPD service will continue in NHS GG&C 2022-2024,
supported by renewed Scottish Government TEC funding. Currently ~400 patients have been
onboarded the COPD digital service (March 2022), with user numbers anticipated to rise as
LenusCOPD is used to support recovery and re-orientation of COPD diagnostic, regular review
and pulmonary rehabilitation services. This continued adoption and our innovation partnership
provides the test-bed infrastructure to implement and evaluate additional innovations,
including the artificial intelligence-insights for MDT decision support proposed in this
clinical investigation.
Rationale: Trained and validated COPD risk prediction models
The challenges outlined in above presented an opportunity to use AI to develop patient risk
prediction models. The data collected as part of the service, together with historical
Electronic Health Record data were used to create a suite of risk prediction models. The
following steps were taken to develop each model.
- Data exploration - data exploration and visualisations to identify features, biases, and
other issues with the datasets.
- Problem formulation - how to define the target variable of interest, and what modelling
approach to use.
- Data splitting - defining the strategy for splitting the datasets to be used for
training, validation, and testing.
- Data cleaning and missing data handling - identifying erroneous data, outliers in the
dataset, and defining a strategy for dealing with missing data (imputation, exclusion,
allowing missing values).
- Feature Engineering - The process of transforming and aggregating the raw data into
features to be used for model training. Feature engineering allows for greater insight
to be drawn from the data.
- Model training - exploration of different machine learning algorithms and choosing the
best candidate for the problem at hand. The processed features are then used for model
training.
- Model validation - validation and reporting on model performance using the hold-out test
dataset.
- Model explainability - use of third-party methods to make the model predictions as
interpretable and explainable as possible.
- Model Fairness - examining model performance on sub-groups of interest and potential
model re-training to account for any inequalities.
For full details of model development, refer to the investigator brochure and Model approval
report. The output for each of the models is a number representative of the probability of
the event of interest happening for each patient.
Intervention
We have co-designed all of the components required to provide live AI-model based risk scores
for 12-month mortality and 3-month readmission to clinicians within a COPD-MDT. To ensure the
production COPD digital service is not impacted by the AI Insights study, a separate
clinician dashboard will be developed to surface these model predictions. The output of the
exacerbation model will be a 'check-in' message sent to patients, highlighting that they have
some possible changes in their data, prompting them to review their self-management plan
and/or contact clinical team for advice if required. This is summarised in table 1, with the
example check-in message at figure 1.
All the source data feeds required to run these models have been established in NHS GG&C
Azure tenancy, using Azure data science VMs (Virtual Machine). The intervention consists of:
-
Consent process
Further details are in CIP section 10.4 and in the "Standard operating procedure for
DYNAMIC-AI consent" document. The LenusCOPD digital service support website
https://support.nhscopd.scot provides the detailed patient information and links to
explanation videos which support the consent process. Study information would not be live on
this open access website until all study approvals were obtained. The web content is
available for review prior to approvals at https://support.test.nhscopd.scot/dynamic-ai
The LenusCOPD patient app contains the consent screens where invited patients can view
information about the study and accept, reject or defer consent and access further
information via the support website, or request discussion with the study team. The LenusCOPD
clinician dashboard AI insights consent page contains the user interface for activating
consent in the patient app, revoking consent when requested by patient and auditing consent
status.
AI insights data pipelines These cloud-based digital connections ingest and aggregate
relevant health record data (demographics, diagnoses, admissions, prescribing, laboratory
results) from consented participants. Data is aggregated, pseudonymised and converted into
suitable formats for use by the AI insights service. Processes for providing this data are
approved and supported by NHS GG&C SafeHaven team, with a standard operating procedure.
AI insights model serving pipeline The cloud-based digital pipeline ingests the pre-processed
data from the data pipeline for use by the machine-learning models, producing model
predictions and explainability for consenting participants for use by the AI Insights
Clinician App.
AI insights service This consists of cloud-based electronic health system back-end functions
and processes which receive instructions and perform actions to ensure the smooth running of
the intervention.
AI insights database The data store for the AI Insights digital service. Raw and summary data
can be exported as model performance reports for review by data science and clinical leads,
to clinical trials unit for independent analysis, and for long term retention (as per IRAS
document).
AI insights clinician app The web application that provides clinicians with clinical
dashboards (Figure 1 & 2) to view and export patient lists and review machine-learning model
results, explainability and fairness data from consented participants.
STUDY OBJECTIVES
Hypotheses
Patients with COPD will find it acceptable and consent to use of their routine healthcare
data to provide AI insights risk prediction scores to their clinical team, with data used for
ongoing model training and validation.
It will be technically feasible and safe to provide AI-model derived 12-month mortality COPD
risk prediction scores for review in the COPD AI-MDT within the 'AI insights' application.
Primary objectives
To determine :
1. the acceptability to patients with COPD and
2. technical feasibility and
3. safety
of presenting live artificial intelligence-based 12-month mortality risk-prediction score
from the LenusCOPD AI insights application to COPD clinicians' multi-disciplinary team
meetings in NHS GG&C.
Primary feasibility, acceptability and safety measurements
These will be reported descriptively as there is no pilot data to inform a power calculation
or a priori success metric.
1. Evaluation of acceptability will be based on proportion of patients invited who consent
to participate in the DYNAMIC-AI study.
2. Evaluation of technical feasibility will be proportion of participants with adequate
source data in LenusCOPD who have 12-month mortality model-risk scores calculated and
presented for MDT review in the AI-insights model app.
3. Evaluation of safety will be based on occurrence of device-related adverse events and
from the prospective evaluation of model risk scores - actions of clinicians based on
model risk scores and calibration of predicted events : occurred events.
Secondary objectives
Acquire detailed acceptability and technical feasibility experience with the 12-month
mortality risk prediction model.
Expand dataset for training and validation of 3-month respiratory-related admission and COPD
exacerbation risk prediction models.
Evaluate technical feasibility (proportion of consented participants with live scores in
model app) and descriptive adoption experience with 3-month respiratory-related admission and
COPD exacerbation models, if review of validation performance data indicates these are ready
for clinical adoption.
Establish a preliminary experience and utility dataset from use of live AI insights by a COPD
multidisciplinary team.
Secondary exploratory evaluations
Detailed acceptability data
Proportion of invited patients who reviewed additional supporting information (video and
written materials).
Proportion of invited patients who require individual clinician input to inform their consent
decision, with themes from these discussions noted including any divergence on
acceptability/consent for the study and consent to use data for model training.
Detailed technical feasibility and model performance data
Captured in model performance reports. Serial data across the clinical investigation
including:
- numbers of participants with missing model scores and reasons (missing data per feature
group, latency of data refresh, participant withdrawal, others).
- model performance metrics calculated using the hold-out test dataset for each model run
including AUC-ROC, precision-recall AUC, classification report, and confusion matrix.
- Model fairness metrics calculated for groups of interest.
- Global and local explainability for the model to illustrate model decision making and
bio-plausibility
- Performance of model risk prediction vs events captured in LenusCOPD (death at 12
months, respiratory-related hospital admission, exacerbation).
- drift-variability in model performance with exploratory evaluation of reasons (e.g.
missing data resolved, change in underlying source data, patient condition fluctuates)
for any identified unexpected changes in model performance or individual risk scores.
Expand dataset for training and validation of 3-month respiratory-related admission and COPD
exacerbation risk prediction models
Data will be accrued from consenting participants. Updated model performance reports will be
reviewed by study steering committee. If these and interim analyses with primary endpoint are
satisfactory these models will be added to the live service.
Preliminary experience and utility data from MDT use of live COPD AI insights Number of
patients discussed at COPD AI-MDT.
Clinical actions taken from model risk score review in MDT, captured in LenusCOPD events and
notes.
If exacerbation model adopted for live clinical use:
- Number of check-in messages triggered by high-risk score from exacerbation model - per
patient and total for cohort captured in LenusCOPD messaging.
- Proportion of participants where exacerbation model check-in messages are toggled off
temporarily or permanently, captured in LenusCOPD notes.
- Outcomes from check-in messages - patient initiates contact, other event, captured in
LenusCOPD messaging and notes.
Patient events (death, hospital admission, COPD exacerbation; captured in LenusCOPD) as they
occur prospectively following model risk score. Noting intervening MDT review, messaging,
clinician action, patient action triggered by model to establish preliminary experience data
about impact of risk-prediction triggered actions and their potential influence on the
subsequent accuracy of those predictions. Evaluation will compare the model performance vs
prospective events considering patients who had a model triggered review/action vs those who
did not. This evaluation will be supplemented by external retrospective validation data
acquired from adoption of the LenusCOPD management service in other health boards. The model
performance on the groups with and without intervention will also be shared with the Data
Science team to allow for further model refinements and re-training.
Related evaluations Continued development of AI insights models up to the point of
commencement of this investigation is being undertaken within the DYNAMIC project and
RECEIVER trial evaluations. External validation of the AI insights' model performance will be
undertaken using de-identified electronic health record and LenusCOPD service data from
organisations other than NHS GG&C. Co-design, clinical user experience and acceptability
testing of AI insights will continue in parallel. These separate evaluations are supported by
all appropriate approvals, governance and documentation.
INVESTIGATION PLAN
Study Design
This clinical investigation will be performed according to the UK Policy Framework for Health
and Social Care Research (2017).
This is an observational cohort implementation-effectiveness study. Interim effectiveness
evaluations will be undertaken at 3-monthly intervals across the trial, with reporting on the
primary endpoints reviewed by the study steering committee. Results from these interim
evaluations may trigger a recommendation to adapt the implementation or intervention - e.g.
changes in consent information, technical updates. Proposed adaptations to the implementation
strategy or intervention would be considered by the study management committee, with any
required amendments to this CIP and associated study documentation submitted for sponsor and
REC consideration. Implementation changes would only be made once all reviews were complete
and approvals obtained. The final report of the study would detail any changes made based on
the interim evaluations.
