Cloud proof of concept
Federated Clinical Data Sharing with Databricks Delta Sharing
A proof-of-concept federated data architecture that lets healthcare and pharma partners query curated clinical trial reference data via Databricks Delta Sharing, with patient-data processing designed to remain within the partner's own cloud environment.
Source code not published
Problem Statement
Most SaaS integrations solve data sharing the same way: the customer sends their data to the vendor's cloud, the vendor processes it, and results come back. For clinical trial matching - where the customer's data is Protected Health Information (PHI) covered by HIPAA - that model is a non-starter for a lot of hospitals and pharma sponsors. Security and compliance teams will not sign off on patient data leaving their environment, and once a vendor is storing that data, the vendor also inherits the regulatory and liability burden of being a data custodian.
The requirement I was solving for: let a reference-data provider (trial definitions, eligibility criteria, sites, therapeutic areas) and a data owner (a hospital or pharma sponsor holding patient records) collaborate on patient-trial matching, while the patient data never leaves the data owner's cloud boundary - and do it in a way that works whether the data owner is running in AWS, GCP, or Azure.
The Solution: Federated Query Architecture
Instead of the traditional "send data to the SaaS platform" model, this design flips the direction: a lightweight container runs inside the data owner's own cloud account, pulls only the non-sensitive reference data it needs from the provider over Databricks Delta Sharing, and does matching/processing locally. The intended boundary is that the provider serves reference tables to an authenticated recipient without receiving patient records. Enforcing that boundary also requires application permissions, egress controls, logging/telemetry review, and deployment validation; Delta Sharing alone does not prove it.
Why Delta Sharing Instead of a Direct API or a Shared S3 Bucket
The obvious alternatives - building a REST API in front of the reference data, or dropping Parquet/CSV exports into a shared S3 bucket - each break down at scale:
| Without Delta Sharing | With Delta Sharing |
|---|---|
| A custom API needs its own authorization and data-serving implementation | The sharing service authorizes reads and supplies metadata and temporary storage access |
| Scheduled exports may introduce freshness delays | Recipients can read published table updates without a custom export pipeline; freshness depends on publication and client behavior |
| Raw files mean building schema management, versioning, and format handling by hand | Full table protocol - schema, types, versioning, partition pruning handled by Databricks |
| Direct object-store integration requires cloud-specific permissions and client setup | The sharing client provides a common interface; networking and identity still require cross-cloud validation |
| Direct object access requires a revocation and credential-expiry plan | Revoking a share or recipient controls future authorized access; it cannot erase downloaded copies, and temporary storage credentials need their own expiry considerations |
| Object-store and application logs need correlation for attribution | Sharing audit events can identify recipient/table requests; configure logging and retention rather than assuming every subsequent use of downloaded data is observable |
The key realization driving this architecture: Delta Sharing isn't a "middle layer" bolted onto the design - it is the data access protocol. Without it, the provider would need to build and maintain its own table-serving infrastructure, ETL pipeline, schema management, per-customer scoping, and revocation system from scratch.
Proof of Concept: Clinical Trial Patient Matching
Clinical trial matching was the demo use case because it has exactly the properties that make federated sharing worth the extra design effort: highly sensitive PHI on the customer side, strict HIPAA obligations, and a clean split between reference data (trial metadata, which is not sensitive) and patient data (which is).
- Reference data provider exposes a curated catalog of trial definitions, eligibility criteria, sites/sponsors, and therapeutic areas via Delta Sharing - no patient data access required by this design. Provider-side access prevention still depends on the deployed application and cloud permissions.
- Data owner (modeled as a hospital or pharma sponsor) uploads patient records to their own cloud storage, runs the matching container locally against the Delta-shared reference tables, and keeps all match results in their own environment.
- The provider can revoke a partner's access at any time by deleting the Delta Share recipient - future authorized reads can be denied without changing the customer's cloud account. Previously downloaded data cannot be recalled. Test revocation, cached access, and temporary URL expiry separately. See the recipient access documentation.
System Architecture
Two components with separate responsibilities. New reference-data reads and access provisioning still depend on the provider services being available:
- Admin server (reference data provider side): license management, recipient/share provisioning against Unity Catalog, and access revocation. Runs as a single, centrally hosted service (proof-of-concept used Cloud Run).
- Data-owner container: a self-contained image with a registration wizard, patient data upload, the Delta Sharing client, and the matching engine. The same container image deploys unmodified to either AWS (ECS Fargate) or GCP (Cloud Run) - the only per-deployment difference is the cloud-native storage bucket it reads patient data from and writes results to.
Validating the Cross-Cloud Requirement
The real-world driver for this project was a cross-platform requirement: an internal application was running in GCP, while a partner's data lived in AWS S3, with the requirement that the application be able to read shared reference data without either side needing to move data across cloud boundaries. The proof of concept validated this directly - one data-owner deployment on AWS reading from an S3-backed container, and a second on GCP - both authenticating against the same Delta Sharing catalog with the same client library and credential format, and both able to have their access revoked independently from the single admin server.
Technologies Used
- Databricks Unity Catalog + Delta Sharing as the sole data access path between the reference data provider and each partner's environment - an open protocol (Linux Foundation), so partners need no Databricks license of their own to read shared data.
- AWS ECS Fargate and GCP Cloud Run to run identical container images as the customer-side matching application, proving the design is cloud-agnostic.
- Python / Flask for the registration wizard, matching dashboard, and admin APIs.
- AWS S3 and GCP Cloud Storage as the local, customer-owned storage for patient data and match results - designed to remain outside the reference provider's access.
Lessons Learned
The biggest lesson was architectural: once Delta Sharing is treated as the actual data access protocol rather than an add-on, it provides a common foundation for table access and recipient authorization. Application data boundaries, credential lifecycle, cross-cloud networking, and revocation testing remain engineering responsibilities. The second lesson was practical - proving the design cross-cloud (GCP application reading AWS-hosted data through the shared catalog, and vice versa) early on surfaced integration details around credential provisioning and revocation that would have been much more expensive to discover after a real customer was onboarded. Company and organization names used in this write-up are generic/anonymized; the architecture and technical approach reflect real proof-of-concept work.