Incorporation of a machine learning model in hepatitis C virus screening leads to a marked improvement in efficiency

1. A comparison of a traditional decision tree approach with a machine-learning (ML)-aided strategy suggests that machine-learning tools offer increased accuracy for hepatitis C virus screening

2. Prospective evaluation of the machine-learning model suggests a nearly 100-fold increase in efficiency as compared to traditional screening methods 

Evidence Rating Level: 1 (Excellent)

Study Rundown: Clinical guidelines for screening typically offer simple decision trees to help providers identify those patients at high risk for developing a disease based on a few criteria. For example, low-dose lung computerized tomography (CT) scans are recommended for screening amongst patients with a certain level of smoking history. Yet, recent machine learning models, while offering less interpretability in their inherent design, incorporate far more health record-based factors to identify patients more granularly at risk, and thus more accurately identify those that would benefit from screening.

In 2020, the United States Preventive Services Task Force (USPTF) recommended all adults aged 18 to 79 years be screened for hepatitis C virus (HCV) once in their life. This comes along the coattails of a recent recommendation from the World Health Organization (WHO) for countries to identify unknown carriers given the myriad advances in treatment for HCV with direct anti-retroviral therapies.

The authors of this study aimed to identify the value of a machine learning-based screening approach using retrospective data gathered from a large healthcare organization in Israel. They evaluated two models, one simple decision tree and a second ensemble of gradient-boosted decision trees, in predicting HCV status. Not surprisingly, the ML-based ensemble of trees showed superior performance amongst cumulative metrics such as the area under the receiver operator curve (AUROC).

The authors then, in concert with policymakers affiliated with the Israel health organization, developed this model to be used prospectively in a cohort of patients presenting from August 2021 to November 2022. The authors additionally examined data from a cohort of patients with similar risk characteristics who were screened with the more traditional USPTF guidelines. Indeed, the authors found, when comparing the number needed to screen (NNS), between these cohorts, an increased efficiency (lift) of over 100-fold with the ML-based screen.

Click here to read the study in NEJM AI

Relevant Reading:  Finding undiagnosed patients with hepatitis C infection: an application of artificial intelligence to patient claims data

In-Depth [prospective and retrospective cohorts]: Dagan and colleagues designed two studies, one retrospective and a separate prospective cohort, to train, test, and validate an ensemble of trees ML-based screening approach for HCV status. The authors first identified 492,290 study members with conclusive HCV status who were in the screening-eligible age range from January 1, 2013, to May 31, 2021. The authors split these data into training, testing, and validation sets and developed a gradient-boosted decision tree (XGBoost) algorithm that incorporated multiple sociodemographic factors, health behaviors, prior diagnoses, and a range of lab values. The authors found that the ML-based approach provided an AUROC value of 0.95, the best performance as compared to a logistic regression model (AUROC: 0.92) and a more traditional single-tree model (AUROC: 0.87).

Next, the authors implemented this model proactively in 1,570,296 patients eligible for HCV screening between August 1, 2021, and November 30, 2022. Of these individuals, 2226 were included in the outreach group which were those patients that had an ML-based predicted risk above the 99.7^th percentile of the test group in the retrospective cohort. After tallying the rates of seropositivity of HCV amongst those that were contacted and extrapolating treatment eligibility, the authors pegged a number needed to screen (NNS) across the sample as 10. In contrast, the comparator cohort that did not employ the ML risk strategy identified an NNS of 1029, suggesting a 158-fold increase in efficiency (lift).

In sum, this study provides strong retrospective evidence for the utility of ML-based approaches to screening for diseases in a large national cohort. In addition, the authors go a step further and evaluate their ML model in a prospective design suggesting a drastic increase in efficiency in screening derived from the improved discriminatory power of the algorithm.

Image: PD

©2024 2 Minute Medicine, Inc. All rights reserved. No works may be reproduced without expressed written consent from 2 Minute Medicine, Inc. Inquire about licensing here. No article should be construed as medical advice and is not intended as such by the authors or by 2 Minute Medicine, Inc.