Federated Clinical Data Sharing with Databricks Delta Sharing
A proof-of-concept federated data architecture that lets healthcare and pharma partners query curated clinical trial reference data via Databricks Delta Sharing, without any regulated patient data ever leaving the partner's own cloud environment.
Repository link coming soon
Problem Statement
Most SaaS integrations solve data sharing the same way: the customer sends their data to the vendor's cloud, the vendor processes it, and results come back. For clinical trial matching - where the customer's data is Protected Health Information (PHI) covered by HIPAA - that model is a non-starter for a lot of hospitals and pharma sponsors. Security and compliance teams will not sign off on patient data leaving their environment, and once a vendor is storing that data, the vendor also inherits the regulatory and liability burden of being a data custodian.
The requirement I was solving for: let a reference-data provider (trial definitions, eligibility criteria, sites, therapeutic areas) and a data owner (a hospital or pharma sponsor holding patient records) collaborate on patient-trial matching, while the patient data never leaves the data owner's cloud boundary - and do it in a way that works whether the data owner is running in AWS, GCP, or Azure.
The Solution: Federated Query Architecture
Instead of the traditional "send data to the SaaS platform" model, this design flips the direction: a lightweight container runs inside the data owner's own cloud account, pulls only the non-sensitive reference data it needs from the provider over Databricks Delta Sharing, and does all matching/processing locally. The provider never receives, stores, or has any access path to the customer's patient data - it only ever serves reference tables to an authenticated recipient.
Why Delta Sharing Instead of a Direct API or a Shared S3 Bucket
The obvious alternatives - building a REST API in front of the reference data, or dropping Parquet/CSV exports into a shared S3 bucket - each break down at scale:
| Without Delta Sharing | With Delta Sharing |
|---|---|
| Provider becomes a data proxy; every customer dashboard load hits the provider's server | Customer reads Delta tables directly - provider is not in the request path |
| Needs an ETL/export pipeline; customers read stale snapshots | Reads live Delta Lake tables in real time, no export step |
| Raw files mean building schema management, versioning, and format handling by hand | Full table protocol - schema, types, versioning, partition pruning handled by Databricks |
| S3-style sharing is AWS-specific; a separate mechanism is needed for GCP/Azure customers | Same .share credential file and client library work identically across clouds |
| Revoking an IAM role doesn't guarantee cached credentials stop working | Deleting the recipient invalidates the token immediately, enforced by the protocol |
| Access logs show "someone read a file," not which customer read which table | Databricks logs record exactly which recipient read which table, and when |
The key realization driving this architecture: Delta Sharing isn't a "middle layer" bolted onto the design - it is the data access protocol. Without it, the provider would need to build and maintain its own table-serving infrastructure, ETL pipeline, schema management, per-customer scoping, and revocation system from scratch.
Proof of Concept: Clinical Trial Patient Matching
Clinical trial matching was the demo use case because it has exactly the properties that make federated sharing worth the extra design effort: highly sensitive PHI on the customer side, strict HIPAA obligations, and a clean split between reference data (trial metadata, which is not sensitive) and patient data (which is).
- Reference data provider exposes a curated catalog of trial definitions, eligibility criteria, sites/sponsors, and therapeutic areas via Delta Sharing - no patient data access required or possible on the provider side.
- Data owner (modeled as a hospital or pharma sponsor) uploads patient records to their own cloud storage, runs the matching container locally against the Delta-shared reference tables, and keeps all match results in their own environment.
- The provider can revoke a partner's access at any time by deleting the Delta Share recipient - no coordination with the customer's cloud account required, and the effect is immediate.
System Architecture
Two components, deliberately kept independent so an outage on one side does not block the other:
- Admin server (reference data provider side): license management, recipient/share provisioning against Unity Catalog, and access revocation. Runs as a single, centrally hosted service (proof-of-concept used Cloud Run).
- Data-owner container: a self-contained image with a registration wizard, patient data upload, the Delta Sharing client, and the matching engine. The same container image deploys unmodified to either AWS (ECS Fargate) or GCP (Cloud Run) - the only per-deployment difference is the cloud-native storage bucket it reads patient data from and writes results to.
Validating the Cross-Cloud Requirement
The real-world driver for this project was a cross-platform requirement: an internal application was running in GCP, while a partner's data lived in AWS S3, with the requirement that the application be able to read shared reference data without either side needing to move data across cloud boundaries. The proof of concept validated this directly - one data-owner deployment on AWS reading from an S3-backed container, and a second on GCP - both authenticating against the same Delta Sharing catalog with the same client library and credential format, and both able to have their access revoked independently from the single admin server.
Technologies Used
- Databricks Unity Catalog + Delta Sharing as the sole data access path between the reference data provider and each partner's environment - an open protocol (Linux Foundation), so partners need no Databricks license of their own to read shared data.
- AWS ECS Fargate and GCP Cloud Run to run identical container images as the customer-side matching application, proving the design is cloud-agnostic.
- Python / Flask for the registration wizard, matching dashboard, and admin APIs.
- AWS S3 and GCP Cloud Storage as the local, customer-owned storage for patient data and match results - never accessed by the reference data provider.
Lessons Learned
The biggest lesson was architectural: once Delta Sharing is treated as the actual data access protocol rather than an add-on, a whole category of problems (ETL pipelines, per-customer file scoping, cross-cloud sharing, credential revocation) disappears rather than needing to be solved. The second lesson was practical - proving the design cross-cloud (GCP application reading AWS-hosted data through the shared catalog, and vice versa) early on surfaced integration details around credential provisioning and revocation that would have been much more expensive to discover after a real customer was onboarded. Company and organization names used in this write-up are generic/anonymized; the architecture and technical approach reflect real proof-of-concept work.