How Adaptive Phage Therapeutics Reduced Pipeline Runtime by 80% with AWS Batch and Nextflow*
Adaptive Phage Therapeutics (APT) is a clinical-stage biotech company advancing therapies to combat multi-drug-resistant infections. Traditional antimicrobials are becoming obsolete as pathogens evolve resistance. APT’s innovative PhageBank therapy leverages a growing library of phages, offering evergreen broad-spectrum and polymicrobial coverage. Through a proprietary phage susceptibility assay, commercialized with Mayo Clinic Laboratories, APT is positioned to revolutionize antimicrobial treatment on a global scale. This groundbreaking work is partially funded by the U.S. Department of Defense.
- *Decreased from 5-6 hours to just 1.15 hours, an approximate 80% improvement.
The Challenge
APT had an existing pipeline built with Snakemake, a bioinformatics workflow engine. While the team’s cloud environment was effective, they sought to standardize to a High-Performance Compute (HPC) platform that could:
• Accelerate pipeline runtimes to deliver refined data faster.
• Enable scientists to process data efficiently, reducing pipeline execution times from days to minutes where possible.
• Scale dynamically to meet variable computational demands while managing costs.
Pipeline jobs, originating from wet bench labs, required scientists to submit raw data through the pipeline. The speed of processing directly impacted the pace of research, making optimization critical.
The Solution
PTP’s CloudOps Engineering team designed a modernized architecture to update APT’s pipeline discovery process, leveraging AWS Batch and migrating from Snakemake to Nextflow:
AWS Batch and Nextflow Integration
• Pipelines now begin with a job submitted to AWS Batch, which launches an EC2 instance running Nextflow.
• Docker images handle pipeline execution, ensuring standardization and repeatability.
• Nextflow dynamically provisions instances for parallel processing, improving efficiency and reducing costs.
Automated Workflow with Lambda
• Lambda functions monitor S3 buckets for new data uploads from wet labs and automatically trigger pipeline execution.
• Outputs from the pipeline are saved to S3, triggering additional Lambda functions to initiate subsequent pipelines.
Optimized Data Storage and Organization
• All pipeline data is stored and organized in S3, with outputs arranged for downstream analysis.
• Polished data is processed through additional pipelines as needed, with results consolidated and presented to scientists.
AWS Services Implemented:
Compute and Storage
EC2, AWS Batch, Lambda, ECR, and S3
Governance and Monitoring
CodeCommit, CloudFormation, CloudWatch, CloudTrail, IAM, and AWS Config
Networking
Virtual Private Cloud (VPC)
The Outcome
The implementation of AWS Batch and Nextflow delivered transformative results for APT:
Runtime Reduction
Pipeline runtimes decreased from 5-6 hours to just 1.15 hours using approximately 650 CPUs.
Parallel Processing
Leveraging AWS Batch allowed pipelines to run concurrently, delivering actionable data faster without additional costs.
Enhanced Scalability
The architecture enables APT to handle increasing data volumes efficiently.
Extended Team Support
PTP’s CloudOps Engineering team provides ongoing design, architecture, and cloud management support, acting as an extension of APT’s data science team.
Ready to optimize your HPC environment?
APT’s transition to an AWS Batch and Nextflow-based pipeline architecture underscores the transformative potential of cloud-native solutions. By reducing runtime, scaling dynamically, and enhancing efficiency, APT is better positioned to advance its mission of combating antimicrobial resistance.
Let us help you unlock your potential.
Contact PTP today to learn how we can help accelerate your research and innovation.