Introduction

Crohn’s disease (CD), a main form of inflammatory bowel disease (IBD), is characterized as a chronic, remitting, and relapsing inflammatory disorder of the gastrointestinal tract, requiring ongoing management and surveillance.¹ Infliximab (IFX), shows efficacy in inducing and maintaining remission, also is the first anti-tumor necrosis factor (TNF) α agent. Yet, primary loss of response (PLR) to IFX is observed in 13–40% of CD patients, while 23–46% patients experience a secondary loss of response (SLR).^2,3 The reduction of response to IFX, an anti-TNF monoclonal antibody, is believed to result from inadequate drug levels, attributed to the antibody’s degradation and clearance. Consequently, when a decrease in the IFX’s effectiveness is suspected, a comprehensive assessment involving the evaluation of active inflammation, potential IBD complications, clinical symptoms, biochemical markers, endoscopic findings, and imaging should be undertaken. Furthermore, consistent monitoring of disease activity during IFX treatment is crucial, enabling timely adjustments to therapeutic strategies to achieve and maintain long-term remission.

For decades, the Crohn’s Disease Activity Index (CDAI) has been regarded as a valuable tool for measuring the clinical activity of CD.^4–6 Nevertheless, the accurate calculation of CDAI requires over a seven-day period and the necessity for patient compliance limits its utility in daily clinical practice.^7,8 While colonoscopy offers an objective and accurate assessment of the intestinal condition, its invasiveness, high costs, and potential risks, including perforation, bleeding, and cardiovascular complications related to sedation, detract from its desirability.^9,10 Furthermore, a considerable number of CD patients are unsuitable candidates for endoscopy due to intestinal luminal strictures or a penetrating disease phenotype.¹¹ Additionally, the psychological and physical burdens caused by repeated invasive procedures require extended recovery times, further complicate their application for patients.¹² Both CT and magnetic resonance (MR) enterography serve as pivotal tools for assessing disease status and detecting extraintestinal complications in patients diagnosed with Crohn’s disease (CD).^13,14 MR enterography, known for its provision of multi-sequence imaging data and superior soft tissue contrast resolution, plays a crucial role in evaluating disease activity and treatment response. Additionally, it aids in characterizing inflammatory or fibrotic lesions and facilitates routine clinical follow-up.^15,16 Conversely, computed tomography enterography (CTE) offers advantages such as shorter scanning times, cost-effectiveness, and high inter-reader agreement in image interpretation, thereby enhancing its availability and accessibility in clinical settings.¹⁷ With ongoing equipment upgrades and advancements in CT reconstruction algorithms, there has been notable progress in obtaining high-quality images at reduced radiation doses. This development is particularly advantageous when imaging young patients and conducting repeated examinations over time. CTE can effectively reflect the characteristics of CD,^18–20 and there have been studies specifically investigate the correlation between the features of the intestinal wall and mesentery, and the mucosal condition under CTE.^21,22

Recent advancements in radiomics and deep learning have shown potential use in diagnosing diseases, classifying molecular types, and forecasting treatment outcomes.^23–25 Meng et al performed research utilizing radiomics and deep learning to analyze CT scans of patients with CD.²⁶ Previous studies have identified relationship between the radiological features of CD and IFX response.^27–29 Moreover, recent studies indicates that imaging power radiomics can be applied to diagnose CD mucosal healing.^30,31 Based on this foundation, we posited that employing artificial intelligence (AI) for image analysis of CD might serve as a viable approach for predicting IFX response.

Medical imaging modalities like CT and MRI operate within three-dimensional (3D) spaces, and the application of two-dimensional (2D) techniques to 3D tasks can lead to loss of essential structural information. Training a 3D deep convolutional neural network (DCNN) typically requires larger datasets than 2D DCNN, presenting a substantial obstacle in medical research due to the limited available data.³² Moreover, deep learning-derived features may be sensitive to the global translation, rotation, and scaling of images, a vulnerability not shared by radiomics features.³³ Recent studies indicate that models integrating both radiomics and deep learning outperform models relying on a single approach.^34,35

Our study was designed to evaluate the effectiveness of 3D deep learning, radiomics, and their integrated models in the prediction of IFX response in CD patients. We implemented two strategies for model integration: early fusion based on features and late fusion based on decisions.

Material and Methods

Study Cohort

This study retrospectively collected data from 263 Crohn’s disease patients across three hospitals (Figure 1). The training cohort comprised 166 patients treated at the Center 1 (Wenzhou Medical University First Affiliated Hospital) from January, 2017, to February, 2023. The external test cohort included 97 patients treated at Center 2 (Xiamen Second Hospital) and Center 3 (Yueqing People’s Hospital) from March 2017 to January 2023. The three respective Hospital Ethics Committee granted approval for this study, which was carried out in compliance with the Declaration of Helsinki principles. Given the study’s retrospective and observational design, the requirement for informed consent was waived. Patient confidentiality was assured by the protection of personal information in the medical record system.

Figure 1 Flowchart diagram shows the patient selection process from three medical centers. CD, Crohn’s disease; CTE, computed tomography enterography; IFX, infliximab.

The criteria for inclusion were as follows: a diagnosis of CD established according to the consensus statement of the European Crohn’s and Colitis Organization;³⁶ available CTE scan within one month prior to starting IFX treatment; undergoing IFX treatment with routine follow-up for at least one year, or the IFX treatment failure within one year. Exclusion criteria encompassed: adverse reactions to IFX therapy; coexisting hematologic, metabolic diseases, or auto-immune liver disorders; missing clinical data; poor quality CTE images, including artifacts, overlap, noise; unavailability of standard IFX therapy. The recruitment process is detailed in a flowchart in Figure 1.

The primary outcome focused on IFX treatment failure within one year, defined as composite endpoints that encompass both PLR and SLR to IFX. PLR refers to a lack of reaction to IFX during the induction phase (0–14 weeks), while SLR is identified as a decline in clinical response (15–54 weeks) after initially responding to IFX treatment.^36–38 The outcome was assessed by an experienced multidisciplinary team at our institution, which based its determinations on clinical symptoms, laboratory indices, radiological findings, and endoscopic results. This assessment included the following components: (a) The persistence of disease-related symptoms or active inflammation, as indicated by criteria such as a CDAI exceeding 150 or showing less than a 70% decrease, or a C-reactive protein (CRP) level exceeding 5 mg/L or demonstrating less than a 50% decrease compared to baseline, was assessed during induction therapy. (b) In cases where persistent clinical remission was not achieved following induction therapy or where there was evidence of disease relapse or progression within one year post-treatment, despite an initial response to IFX, further evaluation was conducted. This evaluation encompassed criteria such as the return of CDAI levels to baseline or higher, along with active intestinal inflammation indicated by endoscopic or imaging examinations. (c) Within one year, criteria such as the necessity for surgical intervention, substitution of biological agents, escalation of corticosteroid therapy, or more than two consecutive hospitalizations due to disease progression were also considered.

CT Examination Procedure

Prior to CTE, all patients adhered to a standardized bowel preparation regimen and observed a fasting period of 12 hours. Subsequently, within one hour preceding the scan, patients orally received a 2.5% isotonic mannitol solution (400–500 mL) at 15-minute intervals. CTE was performed using either a 64-section (Philips Brilliance; Philips) or a 256-section (GE Revolution; GE Healthcare) multidetector scanner. Scanning encompassed the region from the top of the diaphragm to the symphysis pubis with patients in the supine position. Following the acquisition of nonenhanced images, a nonionic contrast agent (320 mg/mL; Ioversol, Jiangsu Hengrui Medicine Corp Ltd) was administered intravenously at a dose of 1.5–2.0 mL/kg, with an average injection rate of 3.0–4.0 mL/sec. Arterial phase images were obtained either 15 seconds after reaching an attenuation value of 100 hounsfield units (HU) in the abdominal aorta using automatic bolus-tracking or with a delay of 32–35 seconds post-contrast injection. Subsequently, portal venous phase images were acquired with a delay of 35 seconds following the arterial phase or at 70 seconds post-contrast injection. For radiomics analysis in our study, CTE portal venous phase images with a slice thickness of 5 mm were utilized, consistent with methodologies employed in prior research on radiomics in CD.

Volume-of-Interest Segmentation and Radiomics Features Extraction

Volumes of interest (VOIs) were delineated on CTE in intestinal segments reflecting the most serious inflammation for medical image analysis. This approach aligns with criteria established in previous bowel radiomics research, where active inflammation was identified through endoscopic and radiological findings.^27,29,31 Characteristics include (1) evident ulcers observable via endoscopy and/or CTE, (2) thickening of the intestinal wall and segmental mural hyperenhancement, and (3) complications such as penetration (excluding abscesses), stricture, or the presence of creeping fat around the diseased bowel segment. The segmentation process culminated with a senior abdominal radiologist’s review to verify the selection of optimal VOIs. Two experienced radiologists (with the experience of 5 and 7 years, respectively), in a blinded manner (blinded to clinical characteristics and outcome), outlined each lesion’s contour to capture the lesion volume using the ITK-SNAP software (version 3.8, www.itksnap.org/), excluding the intestinal lumen, guided by the diagnostic criteria for active intestinal inflammation as discerned through imaging or endoscopic outcomes.^13,39 Based on linear interpolation, images were resampled to achieve isotropic voxel dimensions of 3.0 × 3.0 × 3.0 mm³. Subsequently, using Python (version 3.7.0) with the PyRadiomics library (version 3.0.1), which adheres to the Imaging Biomarker Standardization Initiative (IBSI) guidelines, a comprehensive extraction of 1116 radiomics features was performed from the VOIs within the enteric phase images of CTE. For evaluating the reliability and consistency of the delineated VOIs, a re-annotation of all patients’ VOIs was conducted two months following the initial delineation. This subsequent dataset facilitated an analysis using the intraclass correlation coefficient (ICC).

Feature Selection and Model Construction

Radiomics features were standardized by z-score normalization. To address the issue of strong correlations among features (Spearman correlation coefficient of 0.9 or higher), a methodical approach involving greedy recursive feature deletion was adopted for feature filtration. This approach systematically eliminates the most redundant feature within the set iteratively until no feature exhibits a correlation coefficient above 0.9. Subsequently, features exhibiting high stability with intraobserver and interobserver ICC values surpassing 0.75 were maintained.

Further refinement of the feature set was achieved through the application of multivariate least absolute shrinkage and selection operator (LASSO) regression (LASSO parameter: Alpha:1; Family: Binomial; nfolds:10). The support vector machine (SVM) algorithm, renowned for its efficacy in delineating optimal hyperplanes for classification tasks, was employed to develop predictive models⁴⁰ (SVM parameter: Type: eps-regression; Kernel: Radial; Cost: 1; Gamma:0; Epsilon:0.1). Input to the SVM classifier comprised the training dataset, with each sample containing a set of features and an associated label denoting the response or non-response to IFX. The resultant output was a classifier model trained to predict the likelihood of non-response to IFX in patients within the test set, quantifying the probability on a scale from 0 to 1.

Clinical Model Construction

The clinical characteristics in Table 1 were used to construct the clinical model. Feature selection was performed using LASSO regression, followed by SVM classifier for classification prediction. Detailed procedures were described in the above section of “Feature Selection and Model Construction”.

Table 1 Patient Demographics and Clinical Characteristics

3D Deep Learning Model Development and Feature Extraction

In the context of 3D deep learning, the term “bounding box” specifically refers to the cubic boundary encapsulating the lesion’s VOI. To standardize input dimensions, all VOI cubes were resized to 96x96x96 pixels through linear interpolation. To augment the dataset, techniques including inversion along the X, Y, and Z axes were applied. These enhanced 3D images, paired with their respective labels, served as inputs for the 3D ResNet. The training regimen spanned 300 epochs, employing an initial learning rate of 0.02, culminating in a model capable of predicting IFX response in patients. To extract the 3D deep learning features of a patient’s lesion, the 3D ResNet model’s penultimate average pooling layer yielded a total of 512 features.

Construction of the Fusion Model

In this study, two distinct fusion strategies were implemented to develop the fusion model.⁴¹

The first strategy, known as feature-level fusion or early fusion, entails the amalgamation of all features derived from various modalities into a unified feature vector. Specifically, radiomics features of the VOI were extracted utilizing PyRadiomics, and 3D deep learning features were acquired via 3D ResNet, as previously outlined. These features, in conjunction with clinical and radiological data, underwent z-score normalization. Spearman correlation, ICC analyses, and least absolute shrinkage and selection operator (LASSO) analyses were subsequently conducted to refine the feature selection. An SVM classifier was then trained to create the feature-based fusion model, designated as early fusion, with comprehensive methods detailed in the section “Feature selection and model construction”.

The second strategy, decision-level fusion or late fusion, integrates the output probabilities from various models. This approach utilized a stacking ensemble learning technique to amalgamate the output probabilities derived from the three above models.⁴¹ The trained SVM model of late fusion was evaluated in both the training and external test sets. Workflow diagram for the development of the predictive models was presented in the Figure 2.

Figure 2 Workflow diagram for the development of the predictive models. Lesion segmentation and volume of interest (VOI) delineation are performed by experienced radiologists. The Radiomics model is developed using PyRadiomics. The deep learning model is developed based on 3D ResNet. The clinical data are used to construct the clinical model. For the early fusion model, the extracted features from three basic models are integrated to train an SVM classifier. For the late fusion model, the output probabilities from three basic models are used to develop a stacking model with a SVM classifier. A, the process of image processing; B, the construction of single modality models; C, the construction of fusion models. SVM, support vector machine.

Statistical Analyses

To determine if the sample size in our study was adequate to identify an area under the receiver operating characteristic (AUC) curve value different from 0.500, we calculated the necessary sample size using the following parameters: 80% power, a two-sided significance level of 0.05, and an alternative hypothesis that the true AUC values of Early fusion model in both the training and total test cohorts differ from the null hypothesis (AUC = 0.5). The class ratios were based on actual data from the study: 91 non-responders and 75 responders cases in the training cohort, and 44 non-responders and 53 responders cases in the total test cohort.^42,43

The comparison of categorical variables was conducted using either the Chi-square test or the Fischer test, while the Mann–Whitney U-test or the independent T-test was utilized for continuous variables. The assessment of the predictive model’s performance leveraged Receiver Operating Characteristic (ROC) curves, along with metrics such as AUC, accuracy, sensitivity, specificity, and Brier’s score. The R package “glmnet” and “e1071” was used to perform LASSO analysis and SVM model construction, respectively. All statistical analyses were executed using the R programming language (version 3.4.3) and the scikit-learn library (version 0.18) in Python 3.7.

Results

Demographics and Clinical Characteristics

A total of 263 CD patients were included in this study (median age, 28 years IQR: 22–42.5 years; 201 male) treated with IFX, including 166 patients in the training set, 97 patients in the external test set. The detailed information of included patients’ detailed information was summarized in Table 1. Between the non-response and response groups, there was no significant differences in terms of sex, age, history of smoking, drinking habit, BMI, ALT, AST, serum creatinine, triglyceride, low density lipoprotein, lymphocyte count. The albumin, high density lipoprotein, neutrophil count, monocyte count, hemoglobin, platelet, prothrombin time, C-reactive protein, and erythrocyte sedimentation rate in the non-response group were significant different with those of response group.

Feature Selection and Model Development

Among the 1116 radiomic features and 512 deep learning features obtained from the CTE images, 212 radiomics features and 89 deep learning features with intraobserver and interobserver ICCs greater than 0.75 differed between the IFX treatment failure group and non-IFX treatment failure group in the training cohort (P<0.05). Following LASSO regression analysis, 6 radiomics features and 7 deep learning features were finally screened out. Then, we further conduct LASSO on clinical characteristics selection. Finally, 4 clinical characteristics were included into the next model construction.

Performance Analysis of the Clinical, Radiomics, Deep Learning Models and the Fusion Model

We created SVM machine learning models based on clinical, radiomics, deep-learning, and fusion model. The predictive efficiency of each model is summarized in Table 2. Figure 3A demonstrated receiver operating characteristics curves of the SVM machine learning models, with area under the curve (AUC) of 0.91 (95% CI: 0.85–0.95), 0.82 (95% CI:0.76–0.88), 0.78 (95% CI: 0.71–0.84), 0.91 (95% CI: 0.86–0.95), and 0.88 (95% CI: 0.82–0.92) for clinical, radiomics, deep-learning, early fusion, and late fusion model in the training set, respectively. The external test set cohort had AUC of 0.71 (95% CI: 0.61–0.80), 0.72 (95% CI: 0.62–0.81), 0.72 (95% CI: 0.62–0.81), 0.85 (95% CI: 0.77–0.92), and 0.81 (95% CI: 0.72–0.87) in clinical, radiomics, deep-learning, early fusion, and late fusion model, respectively, as shown in the Figure 3B. Moreover, the sensitivity and specificity of each model was: clinical model: 0.92, 0.80; deep-learning: 0.70, 0.81; radiomics: 0.59, 0.87; early fusion: 0.92, 0.81; late fusion: 0.70, 0.99 in the training cohort. And, the sensitivity and specificity of each model was: clinical model: 0.60, 0.80; deep-learning: 0.77, 0.61; radiomics: 0.60, 0.87; early fusion: 0.77, 0.85; late fusion: 0.81, 0.67 in the test cohort. Brier’s score, commonly used metrics for assessing the accuracy of probabilistic prediction models, indicated that early fusion model obtained the best performance among all models (Table 2). Figure 4 also demonstrated that the early fusion model has top precision-recall curve AUC. Figure 5 displayed the decision curve analysis for the model’s training and external test cohort, showing significant net gains for the early fusion based SVM model. Figure 6 presented the calibration curve of each model. Kaplan-Meier curves for IFX response stratified by early fusion are shown in Figure 7. There were significant associations (Training cohort: Log rank test, p <0.001, hazard ratio [HR]=0.120; Test cohort: Log rank test, p <0.001, HR=0.113) among predicted responders and predicted non-responders patients stratified by early fusion model. This underscored the clinical relevance of our model in prediction of non-responsive to IFX treatment.

Table 2 The Performance Evaluation of the Developed Models in the Training and External Test Cohorts

Figure 3 Prediction performance of the clinical, radiomics, deep learning, early fusion, and late fusion models with receiver operating characteristic curve analysis in the training (A) and test (B) cohorts.

Figure 4 Prediction performance of the clinical, radiomics, deep learning, early fusion, and late fusion models with precision-recall curve analysis in the training (A) and test (B) cohorts.

Figure 5 Decision curve analysis of the clinical, radiomics, deep learning, early fusion, and late fusion models in the training (A) and test (B) cohorts.

Figure 6 Calibration curve of the clinical, radiomics, deep learning, early fusion, and late fusion models in the training (A) and test (B) cohorts.

Figure 7 Kaplan-Meier survival curves for early fusion model in the training (A) and test (B) cohorts.

For the early fusion training, a sample size of 14 patients was necessary, consisting of 8 non-responders and 6 responders. To perform independent validation, a separate sample size of 19 patients was required, including 10 non-responders and 9 responders. As a result, the sample sizes used in this study—166 for the training cohort and 97 for the total test cohort—were sufficient to detect true AUC values of 0.91 and 0.85, respectively, each differing from 0.500, with 80% power.

Discussion

Currently, IFX therapy is crucial in the initial clinical management of CD patients for maintaining long-term remission. Our study developed a CTE-based multimodal fusion model to predict IFX treatment outcomes. By incorporating clinical characteristics, radiomics, deep-learning features, we crafted a comprehensive model to early identify CD patients at high risk of IFX treatment failure. This model achieved an AUC of 0.91 (95% CI: 0.86–0.95) and 0.85 (95% CI: 0.77–0.92) in the training and test cohorts, respectively. Our findings, leveraging radiomic and deep-learning features of intestinal lesions from routine CTE images alongside clinical characteristics, could enhance personalized prediction and treatment planning for CD patients.

The prediction of response to IFX in CD patients has raised much concern for a long time. Bai et al suggested that baseline gene expression was related to PLR to IFX in CD patients.⁴⁴ Previous study by Nissila et al also reported that faecal bacterial and fungal microbiota composition could offer a predictive tool to evaluate IFX response in CD patients.⁴⁵ Moreover, there were several studies focused on prediction of IFX response based on clinical characteristics or radiomics data, and exhibiting moderate accuracy in predicting IFX response with AUC<0.850.^38,46,47 The present study tried to develop an integrated model based on deep-learning, radiomics, and clinical features to predict the IFX response in CD patients. Our integrated model can comprehensively analyze the routine medical imaging by deep learning and radiomics, and non-invasively predict IFX response, therefore, not only avoiding inappropriate therapy but also screening more patients suitable for IFX treatment. The findings from our study suggest that the proposed model offers additional value in the realm of personalized diagnostic approaches and treatment decision-making.

Radiomics, through the high-throughput extraction of quantitative features from medical imaging, transforms images into analyzable data for clinical decision support. This process, when combined with other significant variables, aids in constructing predictive models for diagnosis and prognosis.^48,49 Previous studies have indicated the value of radiomic features from CTE images of inflammation diseased bowel walls in predicting mucosal healing or identifying intestinal fibrosis in CD patients.^31,50 Given the common occurrence of multiple intestinal segment involvement in CD, for consistency and efficiency, our analysis focuses on the segment exhibiting the most active inflammation. Previous studies have shown a direct correlation between the thickness of the intestinal wall and disease activity in CD, with specific thickness thresholds providing accurate assessments of inflammatory activity.^51,52 Intestinal strictures in CD, characterized by varying degrees of inflammation and fibrosis, are linked to the emergence of penetrating complications.^18,53 Hence, the presence of intestinal strictures and penetrating complications in affected bowel walls are important markers of active inflammation.

Deep Learning represents a cutting-edge trend in artificial intelligence research,⁵⁴ marking significant advancements in computer vision, machine learning, and various other fields. Li et al developed and validated a deep learning-based nomogram for CT images, aimed at predicting lymph node status in gastric cancer. This model demonstrated strong discrimination ability in the test set, with an AUC of 0.821 (95% CI: 0.722–0.920).⁵⁵ However, to the best of our knowledge, there are few literatures focusing on the deep learning features from medical images in CD patients. Thus, in our study, we also separately validated the deep learning efficacy in CD.

A recent systematic review compared the performances of radiomics models, deep learning models, and multidomain fusion models in medical research.⁵⁶ The findings showed that the fusion model excelled in 63% of the studies, fell short in 25%, and matched the performance of others in 13%. Our study indicated that the effectiveness of the fusion model largely depends on the fusion methods employed. And early fusion being the predominant approach in biomedical research.⁵⁷ In a related study, Huang et al employed a CT-based Convolutional Neural Network (CNN) and clinical data to detect pulmonary embolism, evaluating seven different fusion architectures.⁵⁸ When choosing a fusion approach for a specific medical issue, relying solely on past experience is not recommended. Instead, conducting preliminary experiments with various approaches and selecting the most optimal one is advisable.

Our study introduced a noninvasive approach using a fusion model to predict IFX response in CD patients, offering prediction support and assisting in making informed treatment choices. The model recommended alternative therapy for patients identified as non-response to IFX, therefore reducing the likelihood of complications occurrence and economic burden. Additionally, this early fusion model can aid clinicians in treatment decision by allowing for the early prediction of possible non-response, ensuring that necessary therapeutic interventions are applied without delay. Compared to previous studies focus on identifying IFX non-responders, our research not only employed potential clinical characteristics, but also tried to extract data from medical imaging based on radiomics and deep learning.^47,59 Our manuscript showed that model based on clinical characteristics, radiomics, and deep learning features using early fusion strategy could contribute to decision-making for clinicians in IFX administration management in CD patients.

It’s important to acknowledge its limitations in our study. Firstly, our analysis was retrospective and prospective, which led to unavoidable selection bias. Multicenter studies are required to validate our findings. Secondly, we currently employed a manual method to delineate the VOI. In the future, we will involve leveraging artificial intelligence for automatic segmentation and measurement of the bowel, aiming to enhance the model’s accuracy. Furthermore, MR enterography also plays a crucial role in assessing the disease in CD patients. Our results warrant further investigation in subsequent MR enterography studies.

In conclusion, the proposed early fusion model showed high accuracy in identifying Crohn’s disease patients at high risk for IFX response. Our study introduced a noninvasive tool using standard CTE, which could help predict individual treatment responses in CD patients undergoing IFX therapy. This tool aims to enhance clinical remission rates through reasonable treatment planning.