Integrating multi-cohort machine learning and clinical sample validation to explore peripheral blood mRNA diagnostic biomarkers for prostate cancer

Abstract

Background

The global incidence of prostate cancer (PCa) has been rising annually, and early diagnosis and treatment remain pivotal for improving therapeutic outcomes and patient prognosis. Concurrently, advancements in liquid biopsy technology have facilitated disease diagnosis and monitoring, with its minimally invasive nature and low heterogeneity positioning it as a promising approach for predicting disease progression. However, current liquid biopsy strategies for PCa predominantly rely on prostate-specific antigen (PSA), which lacks specificity and compromises diagnostic accuracy. Thus, there is an urgent need to identify novel liquid biopsy biomarkers to enable early and precise PCa diagnosis.

Methods

We integrated 12 machine learning algorithms to construct 113 combinatorial models, screening and validating an optimal PCa diagnostic panel across five datasets from TCGA and GEO databases. Subsequently, the biological feasibility of the selected predictive model was verified in one prostate epithelial cell line and five PCa cell lines. Robust RNA diagnostic targets were further validated for their expression in plasma samples to establish an RNA-based liquid biopsy strategy for PCa. Finally, plasma samples from PCa and benign prostatic hyperplasia (BPH) patients at Wuhan Tongji Hospital were collected to evaluate the strategy’s clinical significance.

Results

Differential analysis identified 1,071 candidate mRNAs, which were input into the integrated machine learning framework. Among the 113 combinatorial models, the 9-gene diagnostic panel selected by the Stepglm[both] and Enet[alpha = 0.4] algorithms demonstrated the highest diagnostic efficacy (mean AUC = 0.91), including JPH4, RASL12, AOX1, SLC18A2, PDZRN4, P2RY2, B3GNT8, KCNQ5, and APOBEC3C. Cell line experiments further validated AOX1 and B3GNT8 as robust RNA biomarkers, both exhibiting consistent PCa-specific expression in human plasma samples. In liquid biopsy analyses, AOX1 and B3GNT8 outperformed PSA in diagnostic accuracy, achieving a combined AUC of 0.91. Notably, these biomarkers also demonstrated diagnostic utility in patients with ISUP ≤ 2.

Conclusions

Through an integrated machine learning approach and clinical validation, we developed an RNA-based diagnostic panel for PCa. Specifically, we identified AOX1 and B3GNT8 as novel liquid biopsy biomarkers with promising clinical diagnostic value. These findings provide new targets and insights for early and precise PCa diagnosis.

Introduction

Prostate cancer (PCa), the second most prevalent malignancy in males, demonstrates a steadily increasing global incidence [1,2,3]. Compared to localized disease, advanced PCa is associated with constrained therapeutic options and diminished survival outcomes, underscoring the imperative for early and accurate detection [4, 5]. Current diagnostic modalities—including digital rectal examination (DRE), prostate-specific antigen (PSA) testing, imaging techniques (e.g., MRI), and histopathological biopsy—are hampered by critical limitations [6]. DRE exhibits substantial operator dependency and low sensitivity for early-stage lesions [7]. Advanced imaging, while improving diagnostic accuracy, requires specialized infrastructure and is too costly for large-scale screening. Although biopsy remains the gold standard for diagnosis, it carries risks of complications and potential tumor dissemination, making it unsuitable for routine use. As a result, liquid biopsy platforms, exemplified by PSA testing, have emerged as a minimally invasive and increasingly preferred tool for PCa surveillance.

Liquid biopsy refers to a disease monitoring method utilizing biological fluids such as blood, urine, and saliva as samples [8, 9]. It offers advantages including minimal invasiveness, low cost, operational simplicity, and reduced heterogeneity, and has been extensively researched and applied in recent years [10, 11]. For PCa diagnosis, serum PSA testing remains the most established liquid biopsy approach, frequently employed even in routine health screenings [12, 13]. However, the low specificity of PSA often leads to overdiagnosis and overtreatment when used in isolation [12, 14]. To address this limitation, researchers have explored various alternative biomarkers, though most remain in the validation phase. For instance, at the protein level, liquid biopsy strategies based on P2PSA and the Prostate Health Index (PHI) have demonstrated superior diagnostic performance compared to PSA [15]. At the genomic level, alterations such as DNA methylation and mutations have been linked to PCa progression and prognosis [16, 17]. Additionally, transcriptomic-level RNAs, which reflect real-time gene expression states and exhibit strong associations with disease initiation and progression, represent promising targets for sensitive and specific diagnosis [18, 19]. In recent years, significant progress has been made in the study of RNA biomarkers in PCa liquid biopsy, with different types of RNA demonstrating potential value in early diagnosis and disease monitoring. For example, prostate cancer antigen 3 (PCA3) is a long non-coding RNA (lncRNA) specifically expressed in prostate cancer cells, and its urine-based detection has been approved by the FDA as an adjunct to PSA testing to improve diagnostic accuracy [20]. Additionally, other lncRNAs, such as SChLAP1 and MALAT1, have shown promising diagnostic and prognostic value in blood and urine samples from PCa patients [21, 22]. In the field of microRNAs (miRNAs), molecules such as miR-141, miR-375 are abnormally expressed in the serum and urine of PCa patients and have been associated with tumor staging, invasiveness, and prognosis [23]. Furthermore, circular RNAs (circRNAs) have recently gained attention, with some circRNAs significantly upregulated in the blood of PCa patients, suggesting their potential as novel biomarkers [24]. Notably, with recent advancements in detection technologies, cell-free RNA (cfRNA) has garnered increasing attention, and its diagnostic value is gradually being validated [25, 26]. Studies have shown that specific expression patterns of cfRNA can distinguish PCa patients from those with benign prostate diseases and demonstrate high clinical value in predicting tumor progression [27]. Additionally, multi-gene RNA-based diagnostic panels have been proposed to further enhance diagnostic specificity and sensitivity. For example, urine-based RNA tests such as SelectMDx and ExoDx Prostate have entered the clinical validation stage [28, 29]. However, current RNA biomarkers for PCa diagnosis predominantly focus on single, mechanistically prominent RNAs directly associated with tumorigenesis, lacking rigorous screening processes, which compromises diagnostic specificity [22, 30]. Although multi-gene panels represent a potential solution, their clinical robustness is often compromised by suboptimal selection methods and insufficient validation across diverse cohorts. Furthermore, the limited application of RNA biomarkers in PCa liquid biopsy has hindered their clinical translation. Therefore, future research should focus on developing large-scale, multi-center validation studies and integrating bioinformatics approaches such as machine learning to establish more stable and reproducible RNA biomarker detection systems, thereby advancing the precision application of PCa liquid biopsy.

This study establishes a machine learning framework integrating 113 combinatorial algorithms to identify robust mRNA signatures using TCGA and four GEO cohorts, followed by clinical validation in peripheral blood specimens. Our work pioneers a novel diagnostic paradigm that may enable non-invasive and precise PCa detection through transcriptomic liquid biopsy, ultimately informing personalized therapeutic strategies.

Methods

Collection and processing of public databases

Transcriptomic data from a total of 1,096 PCa patients across five cohorts (TCGA-PRAD, GSE94767, GSE200879, GSE229904, and GSE246067) were collected from the TCGA and GEO databases to construct and validate our diagnostic signature. Among these, 502 patients from TCGA were designated as the training set for panel construction, while 594 patients from the four GEO datasets served as the validation set to evaluate diagnostic performance.

The RNA-seq read count matrix from TCGA, comprising 59,428 coding RNAs, was retrieved. Differential expression analysis was performed using the R packages “DESeq2,” “edgeR,” and “limma,” with thresholds set at |logFC| > 1.5 and p-value < 0.01. Genes identified as differentially expressed by all three methods were selected and visualized via Venn diagrams. Gene expression matrices from the four GEO datasets were intersected with the TCGA-PRAD results to identify common genes across all five datasets.

Diagnostic panel generation via integrated machine learning approaches

We employed 12 machine learning algorithms to generate 113 combinatorial configurations, integrating feature selection and predictive modeling methods, including Lasso, Ridge, Elastic Net (Enet), Stepglm, SVM, glmBoost, LDA, plsRglm, RandomForest, GBM, XGBoost, and NaiveBayes. A two-step approach was implemented, where one algorithm was used for feature selection and another for model construction. Feature selection was performed using LASSO, Ridge, and Elastic Net, with the optimal λ parameter determined through 10-fold cross-validation to minimize the mean squared error. In predictive modeling, SVM with an RBF kernel was applied, and hyperparameters (C and γ) were optimized through grid search. The RandomForest model was built using 1000 decision trees. XGBoost hyperparameters were fine-tuned using Bayesian optimization, with a learning rate of 0.1 and a maximum tree depth of 2. Model performance was evaluated by calculating the area under the receiver operating characteristic curve (AUC) on external validation datasets. The optimal model was defined as the one achieving the highest average AUC across multiple datasets.

Cell lines processing

One prostate epithelial cell line (RWPE-1) and five PCa cell lines (22RV1, C4-2, DU 145, LNCaP, PC-3) were obtained from the American Type Culture Collection (ATCC, VA, USA). All cell lines were cultured in RPMI-1640 medium (Gibco, USA) supplemented with 10% fetal bovine serum (FBS, Gibco, USA) and 1% penicillin/streptomycin (Gibco, USA) at 37 °C in a humidified incubator with 5% CO₂. To minimize genetic drift, cells were maintained between passages 5 and 10. Cell line authentication was confirmed via STR profiling (PowerPlex 21 PCR Kit, Promega, USA) and cross-checked with ATCC and DSMZ databases. All cell lines tested negative for mycoplasma (MycoAlert™, Lonza, Switzerland) and bacterial contamination before use. Total RNA was extracted using the RNAsimple Total RNA Kit (Tiangen Biotech, China) and stored at − 80 °C for further analysis.

Collection and processing of human blood samples

From December 2024 to February 2025, we prospectively collected plasma samples from patients newly diagnosed with PCa or benign prostatic hyperplasia (BPH) at Tongji Hospital (Wuhan, China). All participants underwent PSA testing and needle biopsy, with exclusion criteria including infectious diseases and other malignancies. Preoperative samples were collected and centrifuged within 1 h (1,000× g, 15 min). The isolated plasma was stored at -80 °C for no more than 40 days. Clinical data, including age, PSA levels, and pathological biopsy results, were recorded. Plasma cell-free RNA was extracted using the miRNeasy Serum/Plasma Advanced Kit (QIAGEN, Germany), and stored at − 80 °C (no more than 40 days) (Figure S1). The study was approved by the Ethics Committee of Tongji Hospital (Wuhan, China) (TJ-IRB202407023), with informed consent obtained from all participants.

Quantitative real-time PCR (qRT-PCR)

qRT-PCR was performed using 2× Hieff^® Ultra-Rapid HotStart PCR Master Mix (YEASEN, China) following the manufacturer’s protocols. Primer sequences for the nine target RNAs and GAPDH are provided in Table S1. Table S2 and S3 shows the results of qRT-PCR.

Data analysis

All statistical graphs were generated using GraphPad Prism 8.0.1. ROC and regression analyses were performed in SPSS 29. Group mean differences were assessed via Student’s t-test (two groups) or one-way ANOVA (multiple groups). p < 0.05 was considered statistically significant.

Results

Screening of differentially expressed genes in prostate cancer

The workflow of this study is illustrated in Fig. 1. To identify PCa-specific biomarkers, we first performed differential expression analysis on transcriptomic data from TCGA. After gene symbol annotation, 59,428 genes were included. Three methods—DESeq2, edgeR, and limma—were employed for differential gene screening. To balance the number of candidate genes (the inclusion of an excessive number of genes can exponentially increase the computational burden on machine learning algorithms and complicate the final diagnostic panel) and stability of differential expression, thresholds were set at |log2 fold change (logFC)| > 1.5 and p-value < 0.01. With DESeq2, edgeR, and limma, 3,060, 2,743, and 1,469 differentially expressed genes (DEGs) were identified, respectively. To ensure reliability, the intersection of 1,071 genes from three methods was selected for further analysis. The detailed visualization of the DEGs is provided in Figure S2 and Figure S3.

Fig. 1

Flowchart of integrated machine learning and clinical sample validation for screening PCa liquid biopsy biomarkers

Integrated construction of the prostate cancer diagnostic panel

Direct utilization of a large number of DEGs as a diagnostic panel is impractical. Thus, we further refined these variables to construct an optimal predictive model. Twelve machine learning algorithms—Lasso, Ridge, Elastic Net (Enet), Stepwise Generalized Linear Model (Stepglm), Support Vector Machine (SVM), Generalized Linear Model Boosting (glmBoost), Linear Discriminant Analysis (LDA), Partial Least Squares Regression with GLM (plsRglm), Random Forest, Gradient Boosting Machine (GBM), eXtreme Gradient Boosting (XGBoost), and Naive Bayes—were systematically combined to generate 113 algorithmic configurations. Using TCGA data as the training set and four large GEO datasets (more than 100 patients each) for validation, the Stepglm[both] + Enet[alpha = 0.4] combination yielded a 9-gene diagnostic panel with optimal performance, achieving a mean AUC of 0.913 across all five datasets (Fig. 2a). Notably, all nine genes in this panel—JPH4, RASL12, AOX1, SLC18A2, PDZRN4, P2RY2, B3GNT8, KCNQ5, and APOBEC3C—exhibited tissue-specific downregulation in prostate cancer, suggesting that “protective role”-associated negative markers may hold unique diagnostic value for PCa. Subsequent differential expression analysis in paired clinical samples further validated the robustness of this 9-gene signature (Fig. 2b-j).

Fig. 2

(a) Performance of 113 combinatorial machine learning algorithms in constructing PCa classification models, with AUC values calculated in training and validation sets. (b-j) Validation of the 9-gene diagnostic panel in TCGA-PRAD paired samples

Validation of the gene signature in cell lines

To evaluate the biological applicability of the diagnostic panel, we validated the expression of the nine candidate genes in one prostate epithelial cell line (RWPE-1) and five PCa cell lines (22RV1, C4-2, DU 145, LNCaP, and PC-3). The detailed results of genes expression can be found in Fig. 3. While all genes showed significant downregulation in at least two prostate cancer cell lines, only AOX1 and B3GNT8 exhibited consistent low expression across all tested PCa lines. Notably, divergent or opposing expression patterns were observed among the remaining cell lines (excluding LNCaP), highlighting transcriptional heterogeneity across PCa subtypes of distinct origins.

Fig. 3

Gene expression validation of JPH4 (a), RASL12 (b), AOX1 (c), SLC18A2 (d), PDZRN4 (e), P2RY2 (f), B3GNT8 (g), KCNQ5 (h), and APOBEC3C (i) in one prostate epithelial cell line and five PCa cell lines. ^*p < 0.05, ^**p < 0.01, ^***p < 0.001, ^****p < 0.0001

RNA biomarkers for liquid biopsy and clinical significance

Based on these findings, AOX1 and B3GNT8 were confirmed as stably downregulated in PCa, the specific operational workflow can be seen in Fig. 4a. To evaluate their clinical diagnostic utility, we further investigated their expression in plasma using blood samples from 30 BPH and 42 PCa patients (Table 1). Plasma cell-free RNA was extracted, and RT-PCR revealed significant downregulation of AOX1 and B3GNT8 in PCa plasma (Fig. 4b-c), supporting their potential as liquid biopsy biomarkers. Notably, to validate the robustness of our diagnostic approach, we incorporated 26 RNA samples (10 BPH and 16 PCa) with prolonged storage durations exceeding 30 days. These long-term preserved samples demonstrated gene expression profiles consistent with fresh specimens (Figure S4). ROC curve analysis demonstrated diagnostic AUCs of 0.88 (AOX1) and 0.79 (B3GNT8), both surpassing the performance of PSA (AUC = 0.66). Notably, combining both biomarkers elevated the AUC to 0.91, reinforcing their synergistic diagnostic power (Fig. 4d).

To further assess their clinical value, we analyzed their expression in early-stage PCa (ISUP grade ≤ 2). While neither gene alone achieved specificity for PCa detection, their combined expression showed significant differential expression between BPH and early PC, suggesting their utility in PCa’s early (low-risk stage) diagnosis (Fig. 4e and S5). We further investigated the diagnostic performance of these biomarkers in patients with PSA levels 4–10 ng/mL (commonly regarded as the “gray zone”). Among 25 PCa and 20 BPH cases, AOX1 and B3GNT8 maintained significantly lower expression levels in PCa patients (Figure S6a-b). Within this PSA range, while PSA itself showed limited diagnostic value (AUC = 0.57), both AOX1 (AUC = 0.86) and B3GNT8 (AUC = 0.81) demonstrated superior discriminatory capacity. Their combination achieved enhanced diagnostic performance (AUC = 0.89), showing complementary diagnostic significance to PSA (Figure S6c). Furthermore, in clinically significant PCa (csPCa), both AOX1 and B3GNT8 demonstrated higher diagnostic accuracy than conventional PSA testing. And AOX1 exhibited a significant correlation with ISUP grading (p < 0.01) (Fig. 5), suggesting its potential as a predictive biomarker for PCa risk stratification.

Fig. 4

(a) Workflow of liquid biopsy analysis using PCa plasma samples. (b-c) Validation of AOX1 (b) and B3GNT8 (c) expression in human plasma. (d) ROC curves comparing diagnostic performance of liquid biopsy biomarkers. (e) Combined expression levels of AOX1 and B3GNT8 in plasma from BPH and PCa patients. ^***p < 0.001, ^****p < 0.0001

Fig. 5

Logistic regression analysis and forest plot evaluating the diagnostic value of PSA, AOX1, and B3GNT8 for clinically significant PCa and ISUP grading

Table 1 Clinical characteristics of prostate cancer (PCa) and benign prostatic hyperplasia (BPH) patients

Discussion

PCa is one of the most common malignant tumors worldwide. With population aging and advancements in detection methods, its incidence continues to rise [31]. Although PSA screening is widely used in clinical practice, its specificity and sensitivity are relatively low, often leading to over-biopsy and overtreatment [32, 33]. Therefore, there is an urgent need for more precise molecular diagnostic biomarkers. The emergence of transcriptome analysis has provided important molecular insights into the development and progression of PCa. However, previous studies have largely relied on single-cohort data or traditional statistical methods, making it difficult to achieve reproducible validation in independent datasets, thereby limiting the generalizability and clinical applicability of the models.

In this study, we leveraged multi-cohort transcriptome data from TCGA and GEO and innovatively integrated 113 machine learning algorithm combinations to optimize feature selection, developing a robust and efficient PCa diagnostic gene panel. This approach overcomes the limitations of previous studies that relied heavily on single algorithms or subjective gene selection. To further validate its robustness, we performed external validation across four independent GEO datasets. The results demonstrated consistently high diagnostic performance, with AUC values exceeding 0.75 in all datasets and surpassing 0.90 in some cases, significantly outperforming single-gene markers and traditional PSA testing [34]. Further feature selection analysis identified AOX1 and B3GNT8 as key genes, both of which exhibited significant differential expression between PCa and benign tissues. Notably, both genes showed consistent downregulation in PCa across all tested cell lines (22RV1, C4-2, DU 145, LNCaP, and PC-3), highlighting their stable expression patterns and their potential as reliable biomarkers for PCa diagnosis.

Previous studies further highlight the potential role of AOX1 and B3GNT8 downregulation in PCa progression. These genes are implicated in tumor development through distinct biological mechanisms: AOX1 plays a crucial role in cell metabolism, redox homeostasis, and tumor microenvironment regulation, while B3GNT8 is involved in glycosylation modifications, cell signaling, and tumor microenvironment remodeling. Their downregulation has been linked to PCa development, although the exact mechanisms remain under investigation. Specifically, AOX1 (Aldehyde Oxidase 1) is significantly downregulated in prostate cancer, which may disrupt androgen metabolism and oxidative stress balance, thereby contributing to tumor progression [35]. Moreover, AOX1 downregulation has been confirmed to result from methylation modifications [36] and is closely associated with tumor dedifferentiation, increased invasiveness, and poor prognosis [37]. A potential mechanism involves the activation of tryptophan metabolism, which may facilitate tumor immune evasion and the acquisition of drug resistance [38]. On the other hand, B3GNT8 (β-1,3-N-acetylglucosaminyltransferase 8), a key glycosyltransferase, plays a crucial role in glycosylation modifications, cell signaling transduction, and tumor microenvironment remodeling. Aberrant glycosylation has been recognized as an important molecular feature of prostate cancer [39, 40]. However, research on B3GNT8 in prostate cancer remains limited. Existing studies have primarily focused on colorectal cancer, where B3GNT8 is found to be widely upregulated [41]. Further mechanistic investigations suggest that B3GNT8-mediated aberrant glycosylation can regulate key signaling pathways, promoting tumor cell survival, drug resistance, and invasion [42]. However, some studies have also suggested that B3GNT8 may exert protective effects against biological aging [43]. These findings indicate that the function of B3GNT8 may be tumor-specific, and its precise role and underlying mechanisms in prostate cancer require further investigation.

To further validate the generalizability of our findings, particularly their applicability to the Chinese population, we collected plasma samples from hospitalized patients at Wuhan Tongji Hospital for independent validation and assessed the clinical potential of this gene panel in non-invasive PCa detection. Plasma analysis revealed that AOX1 and B3GNT8 expression levels were significantly lower in PCa patients compared to healthy controls, consistent with their expression patterns in tissue samples. Compared to traditional PSA-based plasma biomarkers, the combined detection of AOX1, B3GNT8, and PSA achieved an AUC consistently above 0.90, significantly enhancing the precise diagnostic capability for PCa. Notably, our subgroup analysis specifically focusing on the PSA “gray zone” patients further revealed their supplementary diagnostic value to PSA, highlighting their potential for early prostate cancer detection. Overall, the AOX1 + B3GNT8 panel demonstrated superior diagnostic performance over single-gene or PSA testing (AUC range: 0.66–0.79) and enabled more precise PCa diagnosis through both tissue and plasma detection. This discovery not only deepens our understanding of the molecular mechanisms underlying PCa but also offers a novel strategy for precise detection and non-invasive screening, holding significant clinical translational potential.

Despite its multiple strengths, this study has several limitations. First, our plasma sample size was relatively small (n = 72), and future studies should expand the cohort to assess the stability of this non-invasive detection method. Second, the biological mechanisms linking AOX1 and B3GNT8 to PCa progression remain incompletely understood, necessitating further functional studies. Moreover, long-term follow-up has not been conducted in this cohort, precluding further assessment of the prognostic value of the identified biomarkers. Future research should focus on multi-omics integration, larger clinical studies, functional validation, and AI-assisted diagnostics to facilitate the clinical translation of our findings. In conclusion, this study leveraged machine learning-optimized multi-cohort transcriptomic analysis to identify an accurate PCa diagnostic gene panel, laying a critical foundation for precise diagnosis and personalized screening of PCa.

Conclusion

In this study, we constructed a 9-gene PCa classification model using integrated machine learning algorithms, which exhibited robust diagnostic performance. Furthermore, AOX1 and B3GNT8 were validated as PCa-specific RNA biomarkers in plasma samples, demonstrating predictive value for both diagnosis and stratification. The two genes showed higher diagnostic accuracy than PSA in csPCa and could serve as complementary biomarkers to enhance PSA-based screening. Additionally, AOX1 exhibited a significant correlation with ISUP grading, suggesting its potential for PCa risk stratification. Overall, this work establishes a reliable RNA-based diagnostic framework for PCa and proposes novel targets to advance liquid biopsy applications in clinical practice.