Life Sciences

Genomics Data Transfer, Analytics, and Machine Learning using AWS Services AWS Whitepaper 10.An additional step is added to run an AWS Glue workflow to convert the VCF to Apache Parquet, write the Parquet files to a data lake bucket in Amazon S3 and update the AWS Glue Data Catalog. 11.A bioinformatic scientist works with the data in the Amazon S3 data lake using Amazon Athena via a Jupyter notebook, Amazon Athena console, AWS CLI, or an API. Jupyter notebooks can be launched from either Amazon SageMaker or AWS Glue. You can also use Amazon SageMaker to train machine learning models or do inference using data in your data lake. 16

• Genomics Data Transfer, Analytics, and Machine Learning using AWS Services
• Table of Contents
• Genomics Data Transfer, Analytics, and Machine Learning using AWS Services (1)
• Abstract
• Introduction
• Transferring genomics data to the Cloud and establishing data access patterns using AWS DataSync and AWS Storage Gateway for files
• Recommendations
• Reference architecture
• File-based access to Amazon S3
• Running secondary analysis workflows using AWS Step Functions and AWS Batch
• Recommendations
• Reference architecture
• Performing tertiary analysis with data lakes using AWS Glue and Amazon Athena
• Recommendations
• Reference architecture
• Performing tertiary analysis with machine learning using Amazon SageMaker
• Recommendations
• Reference architecture
• Conclusion
• Appendix A: Genomics report pipeline reference architecture
• Appendix B: Research data lake ingestion pipeline reference architecture
• Appendix C: Genomics data transfer, analytics, and machine learning reference architecture
• Appendix D: Compliance resources
• Appendix E: Optimizing data transfer, cost, and performance
• Appendix F: Optimizing storage cost and data lifecycle management
• Appendix G: Optimizing secondary analysis compute cost
• Appendix H: Handling secondary analysis task errors and workflow failures
• Appendix I: Monitoring secondary analysis workflow status, cost, and performance
• Appendix J: Scaling secondary analysis
• Appendix K: Optimizing the performance of data lake queries
• Appendix L: Optimizing the cost of data lake queries
• Contributors
• Document revisions
• Notices