AWS Architecture Blog·September 18, 2026Designing a Scalable Serverless Genomic Variant Querying System on AWS
This article details sBeacon, a serverless solution for querying genomic variant data built by CSIRO on AWS. It leverages services like S3, Lambda, DynamoDB, and Athena to achieve high scalability, cost-effectiveness, and fast query responses for large genomic datasets. The architecture emphasizes direct VCF file consumption, decentralized data ownership, and a zero-trust security model.
Read original on AWS Architecture BlogIntroduction to sBeacon and its Benefits
sBeacon is a production-ready, serverless implementation of the Beacon protocol for securely querying genomic variant data. Built by CSIRO on AWS, it targets clinical and research applications, offering significant advantages over traditional approaches. Key benefits include high scalability to mega-biobank-scale datasets (hundreds of millions of individuals), low operational cost (approximately USD 0.40/month for a 1000 Genomes dataset), and fast query responses (around 5 seconds for real-world queries). It also simplifies data management by directly consuming standard VCF files without heavy ingestion or transformation, enabling rapid onboarding of new data within seconds.
Decentralization and Security Focus
A crucial aspect of sBeacon is its focus on improved privacy, data ownership, and decentralization. By avoiding central databases and supporting federated networks, data remains under the control of original holders, addressing ethical and privacy concerns inherent in sensitive genomic data. The system also implements a zero-trust model with explicit authentication, least-privilege access, ephemeral compute isolation, and strict cloud-native boundary controls.
Architecture Overview: Data Onboarding
The sBeacon architecture is divided into data onboarding and data querying processes. Data onboarding focuses on ingesting and indexing genomic metadata. Importantly, raw genomic data (VCF files) is not copied but referenced from its original location (e.g., Amazon S3), while only metadata is loaded and indexed for efficient querying. This design choice minimizes data movement and supports decentralized data models.
- User submits genomic data location to an API Gateway endpoint.
- An AWS Lambda function handles data indexing.
- Metadata is written to Amazon S3 in ORC format for Athena processing.
- Another Lambda orchestrates the indexing, calling CSIRO Ontoserver for ontology indexing.
- Resulting index files are stored in Amazon S3.
- Amazon Athena runs `CREATE TABLE AS SELECT` (CTAS) queries to build and optimize metadata tables, writing them back to S3 in ORC format.
Architecture Overview: Data Querying
The data querying architecture is designed for flexibility, catering to diverse genomic and medical data queries. It leverages a modular approach, separating querying logic into several Lambda functions. This design allows for specialized querying scopes while maintaining a consistent architectural pattern.
- User query hits an API Gateway endpoint, invoking a Microservice Lambda.
- The Microservice Lambda looks up relevant ontology terms in Amazon DynamoDB.
- Matching descendent ontology terms are returned.
- Athena is queried using these ontology codes to fetch metadata.
- If needed, the Microservice Lambda calls the Variant Querying Module.
- The Variant Querying Module (detailed below) processes raw VCF data.
- Results are formatted according to the Beacon protocol and returned via API Gateway.
Variant Querying Module
For genomic variant queries, an Initiator Lambda function fans out `splitQuery` and `performQuery` Lambda functions. These functions operate across VCF files stored in Amazon S3, fetching and processing specific regions to fulfill the query. This fan-out pattern is critical for parallelizing large-scale genomic data processing and achieving the advertised fast query responses.