Abstract
Background
Cancer patients may undergo lengthy and painful chemotherapy treatments, comprising several successive regimens or plans. Treatment inefficacy and other adverse events can lead to discontinuation (or failure) of these plans, or prematurely changing them, which results in a significant amount of physical, financial, and emotional toxicity to the patients and their families. In this research work, we build AI driven treatment failure models that utilize the real-world evidence gathered from patients’ profiles available in an oncology EMR/EHR system, with a goal of predicting the likelihood of a plan being discontinued at the time of its prescription. The selected AI models achieve a prediction accuracy of more than 80% and also provide reasons for their inference.
Methods
Inclusion and Exclusion Criteria: Deidentified and anonymized electronic health records of patients, with their prescribed chemotherapies, for five different primary cancer diagnoses - ICD10 codes C18, C34, C50, C61 and C90 - that have the highest plan discontinuation rates between the years 2015 and 2022 are analyzed. All patients of other cancer types are excluded. AI Models: Unique features, that influence the treatment failure, for each cancer type are engineered by using therapeutic classification of drugs, diagnoses codes, comorbidity scores, tumor and biomarker information that is extracted from the notes and lab tests. We only use features that are available at the time of selecting a treatment plan. Several machine learning classifiers are investigated, and three tree ensembles - random forests, Xgboost and boosted forests - are further evaluated on the validation set to fine tune learning parameters with an objective to reduce the complexity of decision trees for providing better interpretability without significantly compromising the accuracy.
Results
Our pilot studies reveal that boosted forests comprising of 5 random forests, each with 5 trees of depth 10 offer the best compromise between performance and interpretability. The models once trained are evaluated on unseen datasets and four performance measures of AI models are reported. On average, 15 rules are autonomously generated for a treatment failure inference for each cancer type and generally 6 of them have a significant support of 30 samples or greater.
Conclusions
Machine learning algorithms for predicting treatment efficacy of chemotherapy regimens by deriving inference from the patients’ EMR/EHR data is an emerging yet challenging research domain. Our studies demonstrate that AI models like boosted forests provide the optimal models for treatment failure use case. In future, we want to validate the system in controlled clinical trials with the help of oncologists.
| Code | Cohort size | Accuracy | Specificity | F1 score | AUROC |
|---|---|---|---|---|---|
| C18Colon | 1,034 | 0.83 | 0.90 | 0.78 | 0.90 |
| C34Bronchus and lung | 1,547 | 0.81 | 0.93 (highest) | 0.72 | 0.86 |
| C50Breast | 2,184 | 0.87 (highest) | 0.92 | 0.82 | 0.90 |
| C61Prostate | 1,074 | 0.81 | 0.86 | 0.76 | 0.85 |
| C90Multiple myeloma | 866 | 0.85 | 0.89 | 0.83 (highest) | 0.92 (highest) |
Key findings
- 1The estimate arrives at prescription: the models predict the likelihood that a chemotherapy plan will be discontinued at the time it is prescribed, and use only features available when the treatment plan is selected.
- 2Five cancers with the highest discontinuation rates: de-identified EHR records of patients with colon (C18), lung (C34), breast (C50) and prostate (C61) cancer and multiple myeloma (C90), from 2015 to 2022. Patients with other cancer types were excluded.
- 3Features built for each cancer type: therapeutic drug classes, diagnosis codes, comorbidity scores, and tumor and biomarker information extracted from clinical notes and lab tests.
- 4Boosted forests chosen for interpretability: random forests, XGBoost and boosted forests were tuned to keep decision trees small without significantly reducing accuracy. Five random forests of five trees each, at depth 10, gave the best balance of performance and interpretability.
- 5Results on unseen data: breast cancer had the highest accuracy (0.87), multiple myeloma the highest F1 score (0.83) and AUROC (0.92), and lung cancer the highest specificity (0.93) and the lowest F1 score (0.72).
- 6Each prediction comes with its reasons: on average 15 rules are generated per treatment failure inference for each cancer type, and typically 6 of them are supported by 30 or more samples.
Why this matters
When a chemotherapy plan is stopped or changed early, the patient has already carried its physical, financial and emotional cost. This abstract moves the estimate to the moment the regimen is chosen, using only data the oncology EHR already holds at that point, and reports more than 80% accuracy in each of the five cancers studied.
A prediction an oncologist cannot inspect is hard to act on. Here each estimate comes with the rules that produced it, so the oncologist can see which factors drove it and judge whether they hold for the patient in front of them. That is why the authors chose a smaller boosted forest over a more complex model.
These are pilot results, and the authors’ stated next step is validation in controlled clinical trials with oncologists. The work belongs to the same line of CureMD Research as the boosted random forests paper presented at AIME 2023, and is reported here for a clinical oncology audience with results for each cancer.
