Introduction

As one of the most common causes of gastrointestinal hospitalization, acute pancreatitis (AP) is a disease with a localized inflammatory state of the pancreas or a systemic inflammatory response mainly due to pancreatin overactivation.¹ In developed countries, previous studies have shown that the incidence of AP is gradually increasing nowadays.^2,3 The assessment of the severity of AP is usually based on the revised Atlanta classification and can be divided into three levels.^4,5 Patients with mild AP progress slowly and have a relatively short clinical duration. However, approximately 20% of patients with mild AP may spontaneously progress to severe AP, which is often accompanied by systemic inflammatory response syndrome, infection, and persistent organ failure (including respiratory, cardiovascular, and renal systems) with an overall mortality rate of 20–40%.⁶ Moreover, previous study has reported that mortality rates were significantly higher among critically ill patients with AP.⁷ The mortality in critically ill patients with AP can be reduced through early detection and tailored precision treatment. Evidence from previous research indicates that implementing precision treatment within 48 hours of admission can markedly reduce the mortality rate in critically ill patients with AP cases.⁸ Therefore, it is essential for clinicians to assess the severity of AP disease earlier and to intervene in a timely manner to reduce patient mortality.

Machine learning (ML), a field of artificial intelligence, has been introduced to the development of precision medicine and has derived several tools to accurately predict outcomes and solve classification problems.^9–11 ML can use a variety of algorithms to make predictions and incorporate a large number of previously unnoticed variables to improve predictions.^12,13 In addition, ML-based models have often been rigorously validated against traditional statistical methods.¹⁴ Several studies have evaluated the use of ML as a predictive tool in conditions such as critical illness, chronic coronary disease and inflammatory bowel disease.^15–18

Due to the complexity of ML models, many explanatory methods have been applied to help users better understand the results of ML models, including SHapley Additive exPlanation (SHAP) analysis.¹⁹ At present, the SHAP analysis is widely used in the medical field,^20,21 and has also been applied as an interpretation approach for ML models.^22,23

However, there are few studies that employ ML algorithms to predict mortality in critically ill patients with AP. Therefore, we collected relevant data on hospitalizations of patients with AP from different institutions. Then, the synthetic minority oversampling technique (SMOTE) algorithm was employed to construct the ML models. Our study aimed to develop various ML models based on relevant biological and clinical risk factors to predict 28-day-mortality in critically ill patients with AP, thus providing some help for clinical decision-making.

Methods

Selection of Participants

Patients diagnosed with AP in the ICU were included in this study. These individuals were admitted to three institutions: the First Affiliated Hospital of Wenzhou Medical University, Wenzhou Hospital of Integrated Traditional Chinese and Western Medicine, and Xiamen Second Hospital, between July 2016 and October 2023. The study was conducted in accordance with the Declaration of Helsinki and received the necessary approval from the Institutional Review Board (IRB)/Ethics Committee of each hospital. As the data used for this research were anonymized, removing all personal identifiers, the requirement for written informed consent was waived. Exclusion criteria were as follows: (1) patients aged less than 18 years at the time of first admission; (2) patients with severe diseases such as end-stage renal disease, cirrhosis, or cancer; (3) patients with ICU length of stay <24 hours. In total, 534 patients were included in this study.

Data Collection

Based on the literature search and review, 38 potential risk confounders were identified and extracted from the medical records. The extraction of potential variables could be divided into six main groups: (1) demographics, such as age, sex, BMI. (2) Admission vital signs, including heart rate, mean arterial pressure (MBP), systolic blood pressure (SBP), diastolic blood pressure (DBP), respiratory rate (RR). (3) Comorbidities, including hypertension, decompensated heart failure, obesity (BMI > 30kg/m²), diabetes, chronic obstructive pulmonary disease. (4) Laboratory indicators, including erythrocyte, erythrocyte distribution width (RDW), mean corpuscular volume, mean corpuscular hemoglobin concentration, mean corpuscular hemoglobin, hemoglobin, hematocrit, platelets, white blood cells, international normalized ratio (INR), total bilirubin, prothrombin time (PT), anion gap (AG), blood urea nitrogen (BUN), bicarbonate, serum calcium, serum potassium, serum chloride, serum sodium, serum creatinine, glucose. (5) Admission severity of illness scores, including Acute Physiology Score III (APSIII), Simplified Acute Physiology Score II (SAPS-II), Oxford Acute Severity of Illness Score (OASIS), and Sepsis-related Organ Failure Assessment Score (SOFA).^24,25

Endpoint Events

Twenty-eight-day all-cause death after admission was the primary outcome of this study. Twenty-eight-day all-cause mortality was defined as the ratio of the total number of all-cause deaths during a 28-day hospital stay to the average population during the same period.

Class Imbalance

There was a class imbalance in our study. The proportion of survivors is almost eight times that of non-survivors, resulting in a significant class imbalance. Currently, the main approaches are resampling, cost-sensitive learning and ensemble learning.²⁶ SMOTE sampling is the most widely used algorithm, proposed by Chawla in 2002.²⁷ The SMOTE method adds new non-existent samples by inserting them into a small number of similarly located samples. This approach, compared to random oversampling, can potentially mitigate the risk of overfitting. To address the issue of class imbalance, we employed the SMOTE method in the formation of our training cohorts.

ML-Based Modelling

To develop the ML models with several risk factors, the Least Absolute Shrinkage and Selection Operator (LASSO) logistic regression combined with 10-fold cross-validation was used to screen out the potential risk factors. LASSO regression, a penalized regression approach, could minimize the comparatively irrelevant risk factors’ coefficient to zero, thereby excluding those unimportant risk factors and calculating the optimal parameter λ value. Then, LASSO regression analysis results were employed to perform Cox regression analysis, and finally, variables of p < 0.05 were obtained. Selected variables were put into develop ML models. Six ML algorithms including eXtreme gradient Boosting (XGB), Naïve Bayes (NB), support vector machine (SVM), random forest (RF), Advanced Boosting (AdaBoost), and linear regression (LR) were applied to develop models using the training set.

Statistical Analysis

Continuous variables were expressed as either mean ± standard deviation (SD) or median (inter-quartile range) based on their distribution. Categorical variables, on the other hand, were represented as proportions. The Kolmogorov–Smirnov test was utilized to assess the normality of continuous variables. For the analysis of these variables, we applied either the t-test or the Wilcoxon rank test. Categorical variables were compared using the Chi-square test or Fisher’s exact test. In the case of variables with missing values, multiple interpolation was employed to generate an appropriate data set for filling in these gaps. The discriminative capacity of ML models was evaluated using the area under the receiver operating curve (AUROC), with a model deemed excellent if its AUROC value exceeded 0.80.²⁸ For the completeness and accuracy of the results, accuracy, precision, recall, and F1-score were also calculated. Hosmer-Lemeshow test was introduced to test the model’s goodness-of-fit. Optimal cut-off values were obtained by Youden’s index, representing the maximum sum of specificity plus sensitivity, and further evaluated for sensitivity, specificity, and for positive and negative predictive values. Kaplan–Meier survival curves were conducted to compare the difference of 28-d all-cause mortality between high- and low-risk groups. Decision curve analysis (DCA) was employed to compare the net benefit of ML model through the difference between the number of true-positives and false-positives and consider the odds of a specific threshold probability.²⁹ Additionally, a calibration curve was utilized to visually represent the correlation between the predicted probability and the actual probability, facilitated by the calculation of the calibration slope. To visually depict the importance of features in ML models, we implemented SHAP analysis. This method effectively demonstrated the most influential features for the ML model predictions and quantified their contribution to the overall model performance for a specific prediction.³⁰ A double-sided p < 0.05 was regarded as statistically significant. All statistical analysis was conducted by the R software (version 4.0.2) and its packages, including fastshap package (version 0.0.7), tidyverse package (version 1.3.0) and tidyquant package (version 1.0.2) and Python (version 3.6).

Results

Baseline Characteristics

A total of 534 patients were enrolled in our study. Among these, a total of 400 patients were from institution 1 consisted of the training set, and 134 patients were from the other medical centers consisted of the external test set (Figure 1). Twenty-eight-d all-cause death occurred in 72 cases (13.5%) in the whole cohort. It included 314 (58.2%) males and 220 (41.2%) females. The median age of all patients was 56.5 (44.0, 72.0) years. When compared to the survival group, non-survival patients were older and had lower SBP, DBP, MBP, platelet, and bicarbonate, while higher RR, SOFA, APS-III, SAPS-II, OASIS, RDW, WBC, INR, PT, AG, and creatinine (p < 0.001) (Table 1). The baseline data of the training set and the external test set are displayed in Table 2. No significant difference was observed between the two groups (p > 0.05).

Table 1 Baseline Characteristics in the Survivors and Non-Survivors

Table 2 Baseline Characteristics in the Training and External Test Set Cohort

Figure 1 The flowchart of patients selection.

Feature Selection

A total of 38 variables were obtained from the medical records. With the optimal alpha parameter, the LASSO was employed to decrease the dimension of the variables and filter out the best variables. After compressing variable coefficients to avoid models’ overfitting and collinearity problems by LASSO analysis, 38 variables were reduced to 17. To further control the influence of confounding factors, the 17 variables were further analyzed using the Cox proportion regression analysis (Table 3). Finally, 8 variables were obtained, including white blood cell, platelet, body temperature, age, BUN, red blood cell distribution width, SpO2, and hemoglobin.

Table 3 Multivariate COX Proportional Hazard Ratio Regression of Risk Factors for Death Within 28-d in Patients with Acute Pancreatitis

Diagnostic Performance of ML Models

These 8 variables were put into developing six ML models. As shown in Figure 2a and b, all six ML models performed well in predicting 28-d all-cause death in the training and test set, and we also further calculated the relevant scores of models, including accuracy, precision, recall, F1-score, positive and negative likelihood ratios, diagnostic odds ratio, and Matthews correlation coefficient (Table 4). The ROC results demonstrated that RF was the highest in the training set and SVM was the highest in the test set. However, the AUC indicator focused on the predictive accuracy of the models and could not discriminate whether the model was clinically applicable or which one of the two is more suitable.^31,32 Then the DCA and Hosmer-Lemeshow were further analyzed among these algorithms. The DCA curve showed that SVM had better clinical suitability across a range of reasonable high-risk thresholds (Figure 3). The Hosmer–Lemeshow goodness-of-fit test demonstrated that the SVM had good calibration (Hosmer–Lemeshow test: χ²=2.557, P=0.959). Comprehensive analysis demonstrated that SVM could be considered the optimal model.

Table 4 Diagnostic Accuracy for the Five Machine Learning Algorithms with the External Test Data Set of AP Patients

Figure 2 ROC curves of 5 machine learning models in predicting 28-d all cause death for AP patients. (a) ROC curves of the machine learning models using the 10-fold cross-validation in the AP training cohort. (b) ROC curve analysis of machine learning algorithms for prediction of 28-d all cause death AP patients in the test set. Abbreviations: RF, random forests; AdaBoost, Adaptive Boosting; SVM, support vector machines; NB, naïve Bayes; XGBoost, eXtreme Gradient Boosting; LR, linear regression; ROC, receiver operating characteristic; AUC, area under the curve; CI, confidence interval.

Figure 3 Decision curves analyses for 5 machine learning models to predict the 28-d all cause death of the patients. The gray line shows the net benefit of recommending intervention to every patient, and the black line was no-intervention taking.

The Kaplan-Meier curve and calibration curve were also plotted to visualize the prognostic performance of the SVM algorithm. With the optimal cut-off value, the included patients were divided into a high-risk group (n = 78) and low-risk group (n = 56). The Kaplan–Meier method was employed to demonstrate that the 28-d death differed significantly between the high- and low-risk group (Figure 4a). The calibration plots demonstrated good agreement between the SVM prediction and the actual observed death in the external test cohorts (Figure 4b).

Figure 4 (a) Survivor (percent) during the follow-up for the y-axis of the graph and follow-up Time for the x-axis; (b) The calibration curve of the SVM for the 28-d all cause death probability.

Model’s Explainability

Figure 5a depicts the SHAP values, combining the importance and effects of features for all individuals, based on the Support Vector Machine (SVM) model. Each point on the diagram symbolized a feature and its corresponding Shapley value, indicating the contribution of each feature to the model’s predicted output. The contributions of each individual were represented by two dot types, with red dots indicating high-risk values and blue dots indicating low-risk values. The feature values were color-coded, and their importance was ordered from top to bottom along the Y-axis (as shown in Figure 5b). The current SHAP results indicated that the WBC count was the most significant feature in predicting the Shapley value, followed by the platelet count. A lower platelet count (depicted in pink) was associated with Shapley values and served as a positive predictor of all-cause mortality at 28 days. Similarly, a lower temperature and higher age were associated with Shapley values and served as positive predictors of cognitive decline. In summary, the SHAP analysis revealed that WBC, age, BUN, Red Cell Distribution Width (RDW), and hemoglobin were positive predictors of all-cause death at 28 days. Conversely, platelet count, body temperature, and SpO2 were negative predictors of all-cause death at 28 days.

Figure 5 SHAP interprets the model. (a) Attributes of features in SHAP. Each line represents a feature, and the abscissa is the SHAP value. Pink dots represent higher eigenvalues and blue dots represent lower eigenvalues. (b) Feature importance ranking as indicated by SHAP. The matrix diagram describes the importance of each covariate in the development of the final prediction model. (c) SHAP plot to correctly predict all cause death in a specific patient. (d) SHAP plot to correctly predict survivor in a specific patient. Red box represents predictors of active status of disease activity, while blue represents predictors of remission.

Figure 5c and d displayed the individual SHAP plot for predicting specific patient 28-days all-cause death. We presented several random cases. Figure 5c demonstrated the SHAP plot to correctly predict 28-days all-cause death; the predictive model was supported by the Shapley value of, older age, higher value of BUN and RDW. Figure 5d SHAP plot to correctly predict survival; the prediction model was supported by the Shapley value of higher platelets and temperature and the lower values of WBC and BUN.

Discussion

Given its computational prowess and capacity to handle vast data volumes, ML has been harnessed in the advancement of precision medicine, serving as a conduit for in-depth analysis. In the present study, we developed and validated a precise ML model to predict 28-day mortality in AP patients in critical condition. Specifically, our analysis using LASSO and SHAP revealed that higher age, WBC count, BUN, RDW, and lower counts of platelets, body temperature, and SpO2 were factors more likely to be associated with all-cause mortality at 28 days.

Previous studies suggested that age is a potential contributing factor related to worse prognosis in patients with AP.³³ Elderly patients have a higher proportion of chronic diseases, cardiac and pulmonary dysfunction, resulting in a worse outcome. Several studies have also suggested that elderly patients with AP are likely to have atypical clinical presentation and poor prognosis, mainly due to existing comorbidities.^34–37

Moreover, abnormal vital signs including temperature and SpO2 were associated with worse prognosis.^38,39 Basic medical information is commonly employed in ICUs, which means that vital signs are accessible and could be included in many prediction models. Body temperature was another important variable in our model, which potentially leads to excessive oxygen molecules that cannot be fully utilized by the tissue.⁴⁰ According to its SHAP analysis, lower body temperature was an important predictive factor for 28-d mortality. Regarding laboratory tests, five variables were selected to develop models. Several published studies have confirmed that some variables were associated with prognosis. For example, O’Connell et al⁴¹ indicated that RDW could be used to identify patients at risk of severe disease condition on the presentation to the emergency department. Zheng et al⁴² concluded that white blood cell count can be employed to construct tool to predict the severity of AP. Tokoro et al⁴³ suggested that platelets play a pivotal role in the mechanism of disease progression in AP. The merit of these models lies in their ability to contextualize the clinical features of newly admitted patients within similar clinical experiences. This provides a reliable foundation for physicians, particularly those who may be unfamiliar with the patients’ conditions.⁴⁴ Believable models are of need to provide trustworthy, available, and efficient health management, and the models we have developed are paving the way for that process.

In our current research, we employed various ML algorithms, among which the SVM model demonstrated superior performance compared to other algorithms. This superiority was established after a thorough comparison of AUC, DCA, and goodness-of-fit test. Nonetheless, comprehending the ML prediction model in its entirety posed a challenge. Consequently, we applied the SHAP methodology to the SVM model to achieve optimal predictive power and interpretability. This allowed us to identify some key variables associated with the prognosis of critically ill patients suffering from AP.

The SVM model was developed using patient routine indicators. The patients were identified as high-risk patients would preferentially receive more positive intervention. In clinical settings, the developed model could be integrated to enhance several aspects of patient care. Initially, upon hospital admission, the model could generate a predictive score from vital signs, and laboratory results, all of which are routinely available. This use of ML to predict clinical outcomes could effectively prioritize patients at high risk, ensuring they receive prompt clinical and supportive care. Furthermore, in cases of AP patients presenting with complicated or rare conditions, particularly in regions where medical resources are limited, the application of ML model can assist clinicians in making decisions regarding clinical management.

Our study was a multi-institution research, which is a common way of efficiently evaluating a new technique and could provide a reliable foundation for the subsequent generalization of our models. However, several limitations of this study should be acknowledged. First, due to its retrospective nature, selection bias of patients was inevitable. Given that our study was multi-center study with a limited sample of patients with AP, it is imperative to conduct future prospective studies with larger-sample patients to further validate these findings. Thirdly, some variables such as the level of interleukin, albumin and CRP were not considered in our study due to their unavailability of these variables, might result in confounding bias; however, the ML models analyzed up to 38 variables potential variables including vital signs, admission laboratory tests, demographics, and comorbidities, representing that the included variables were comprehensive. For another thing, missing values were unavoidable in this database, and these missing values would influence the study’s conclusions.

Conclusions

In conclusion, this study screened out 8 important variables associated with 28-day all-cause mortality and developed an SVM model that may provide clinicians with a better tool for timely intervention in critically ill patients with AP. The predictive model outperformed conventional scoring systems and provided assistance for first-line clinicians to identify high-risk patients.