Life Sciences

Genomics Data Transfer, Analytics, and Machine Learning using AWS Services AWS Whitepaper Appendix L: Optimizing the cost of data lake queries The biggest challenge with cost optimizing a data lake in Amazon Simple Storage Service (Amazon S3) is understanding the data lake access patterns to lifecycle the data and optimize for cost. With data lakes, the access patterns are often irregular, making it difficult to capture a common set of access patterns and lifecycle the data. Amazon S3 Intelligent-Tiering solves this problem. For a small monitoring and automation fee, S3 Intelligent-Tiering monitors access patterns and moves objects that have not been accessed for 30 consecutive days to the Infrequent Access tier. If the data is accessed later, it is automatically moved back to the frequent access tier. Enabling S3 Intelligent-Tiering is an easy way to cost optimize your data lake. 28

• Genomics Data Transfer, Analytics, and Machine Learning using AWS Services
• Table of Contents
• Genomics Data Transfer, Analytics, and Machine Learning using AWS Services (1)
• Abstract
• Introduction
• Transferring genomics data to the Cloud and establishing data access patterns using AWS DataSync and AWS Storage Gateway for files
• Recommendations
• Reference architecture
• File-based access to Amazon S3
• Running secondary analysis workflows using AWS Step Functions and AWS Batch
• Recommendations
• Reference architecture
• Performing tertiary analysis with data lakes using AWS Glue and Amazon Athena
• Recommendations
• Reference architecture
• Performing tertiary analysis with machine learning using Amazon SageMaker
• Recommendations
• Reference architecture
• Conclusion
• Appendix A: Genomics report pipeline reference architecture
• Appendix B: Research data lake ingestion pipeline reference architecture
• Appendix C: Genomics data transfer, analytics, and machine learning reference architecture
• Appendix D: Compliance resources
• Appendix E: Optimizing data transfer, cost, and performance
• Appendix F: Optimizing storage cost and data lifecycle management
• Appendix G: Optimizing secondary analysis compute cost
• Appendix H: Handling secondary analysis task errors and workflow failures
• Appendix I: Monitoring secondary analysis workflow status, cost, and performance
• Appendix J: Scaling secondary analysis
• Appendix K: Optimizing the performance of data lake queries
• Appendix L: Optimizing the cost of data lake queries
• Contributors
• Document revisions
• Notices