Federated AI Discovery Engine

The Federated AI Discovery Engine is a computational framework designed to support biomarker discovery, evaluation, and validation in distributed data settings, where patient-level data are housed across multiple Trusted Research Environments (TREs) and cannot be centralised. The system enables consistent analytical workflows to be executed across institutions and geographies while preserving local data governance, security, and regulatory constraints.

Background and motivation

Biomarker discovery in modern biomedicine increasingly relies on the integration of large-scale clinical data with high-dimensional molecular measurements, including genomics, proteomics, and other omics modalities. While the availability of such data has expanded substantially, access to comprehensive, centralised datasets remains limited.

In practice, the most informative datasets are fragmented across hospitals, national research infrastructures, and commercial partners, each operating within distinct governance frameworks. Analyses restricted to individual cohorts frequently suffer from limited statistical power, cohort-specific biases, and poor reproducibility when applied to external populations.

Federated analytical approaches address these challenges by allowing models, rather than data, to be deployed across sites. This enables large-scale, multi-cohort analysis while respecting the constraints imposed by data protection, consent, and institutional governance.

Overview of the federated framework

The Federated AI Discovery Engine implements a standardised analytical framework that can be deployed into multiple TREs. Within each environment, identical workflows are executed, including data preprocessing, feature construction, model training, and evaluation.

Only approved summary outputs-such as model parameters, performance metrics, and feature-level statistics-are returned for aggregation and comparison. Patient-level data remain within the originating TRE at all times.

This design supports:

• Direct comparison of model performance across cohorts and populations
• Replication of findings under consistent analytical assumptions
• Systematic assessment of model robustness and generalisability

Modelling methodology

The Discovery Engine supports AI-driven modelling spanning statistical learning, deep survival analysis, and representation learning, allowing different analytical approaches to be applied according to the structure of the data and the scientific question.

Survival and progression modelling

For longitudinal and time-to-event endpoints, the platform implements deep survival models based on Cox proportional hazards formulations, extended to capture non-linear effects and complex covariate interactions. These models are suited to analysing disease progression, onset, and clinical outcomes over time.

Predictive classification

For diagnostic and therapeutic response tasks, the system supports a range of predictive classifiers, enabling stratification of patients based on risk, likely response, or disease state. These models are evaluated using clinically relevant performance metrics and thresholds.

High-dimensional and multi-omics modelling

To address the dimensionality and complexity of molecular data, the Discovery Engine incorporates neural network architectures and modern machine-learning pipelines optimised for high-dimensional feature spaces. These approaches are designed to learn structured representations from multi-omics inputs while maintaining interpretability through downstream feature analysis.

Transfer and representation learning

Where appropriate, transfer and representation learning approaches are applied to enable reuse of learned biological structure across cohorts, TREs, and populations. This allows information learned in one dataset to inform discovery in others, improving efficiency and stability in settings with limited local sample sizes.

Multi-modal benchmarking and evaluation

Analyses are conducted across multiple data modalities, including:

• Clinical and EHR-derived variables
• Proteomic measurements
• Genomic features and polygenic risk scores (PRS)
• Combined multi-modal feature representations

Performance is benchmarked consistently across modalities and cohorts, enabling explicit evaluation of incremental predictive value and interaction effects between clinical and molecular signals.

Population-scale replication and validation

Federated analyses across partner TREs can be extended through evaluation against Hurdle’s federated datasets, which collectively comprise:

• More than 2 million patient records
• Representation across 1,200+ disease areas and 5,000+ phenotypes
• Broad geographic and demographic diversity

These datasets provide an additional layer of validation, supporting the identification of biomarkers that are robust across independent populations and reducing the risk of cohort-specific artefacts.

Case example: Type II Diabetes

The federated framework was applied to the analysis of Type II Diabetes data distributed across three independent TREs located on two continents.

Using identical analytical workflows across all sites:

• Models based on clinical data alone, omics data alone, and combined multi-modal inputs were trained and evaluated
• Multi-modal models achieved AUC values of approximately 0.82, consistently across TREs
• Federated execution increased effective sample size, improving statistical power and stability of learned features
• Predictive performance generalised across populations, indicating reduced sensitivity to population structure and local biases

This example illustrates how federated analysis can support robust biomarker evaluation without centralising sensitive patient-level data.

Application domains

The Federated AI Discovery Engine is applicable across multiple stages of biomedical research and development, including:

Clinical development

• Patient stratification and enrichment
• Prognostic and predictive biomarker development
• Time-to-event and disease progression analyses

Translational research

• Cross-cohort validation of candidate biomarkers
• Integration of clinical and molecular signals to support mechanistic insight

Diagnostics development

Companion diagnostic development

Software-based diagnostics (SaMD)

Assay and kit development

Typical analytical workflow

1. A typical engagement proceeds through the following stages:
2. Deployment of the Discovery Engine into each participating TRE
3. Harmonisation of phenotypes, endpoints, and feature definitions
4. Execution of federated discovery and benchmarking workflows
5. Cross-site comparison, replication, and robustness assessment
6. Advancement of selected biomarkers into downstream validation

The Engine provides a structured, reproducible approach to biomarker discovery and validation in federated data environments, enabling population-robust analysis while maintaining strict data governance and security requirements.

by Dr Tom Stubbs, CEO of Hurdle.bio