Life Sciences

Genomics Data Transfer, Analytics, and Machine Learning using AWS Services AWS Whitepaper File-based access to Amazon S3 In this scenario, a run completion tracker is used to monitor the staging folder and start DataSync task runs to transfer the data to Amazon S3. See Optimizing data transfer cost and performance (p. 20) for information about the run completion tracker pattern. For information about getting started with DataSync, see Getting started with AWS DataSync. To learn more about optimizing storage cost, see Optimizing storage cost and data lifecycle management (p. 22). File-based access to Amazon S3 Amazon EC2 file-based access to Amazon S3 starts with setting up an FSx file system that can be mounted on your EC2 instance. Amazon FSx provides two file systems to choose from: Amazon FSx for Windows File Server for business applications and Amazon FSx for Lustre for compute-intensive workloads. For information about setting up Amazon FSx for Windows File Server, see Create Your File System in the Amazon FSx for Windows File Server User Guide. For information about setting up Amazon FSx for Lustre User Guide, see Create Your Amazon FSx for Lustre File System in the Amazon FSx for Lustre Users Guide. On-premises file-based access to Amazon S3 starts with setting up a file gateway on-premises to present objects stored in Amazon S3 as NFS or SMB on-premises. AWS Storage Gateway connects an on-premises software appliance with cloud-based storage to provide seamless integration with data security features between your on-premises IT environment and the AWS storage infrastructure. You can use the service to store data in the AWS Cloud for scalable and cost-effective storage that helps maintain data security. A file gateway supports a file interface into Amazon S3 and combines a service and a virtual software appliance. By using this combination, you can store and retrieve objects in Amazon S3 using industry-standard file protocols such as NFS and SMB. For information about setting up a file gateway, see Creating a file gateway in the AWS Storage Gateway User Guide. Note You will incur egress charges when transferring data to on-premises from Amazon S3. 5

• Genomics Data Transfer, Analytics, and Machine Learning using AWS Services
• Table of Contents
• Genomics Data Transfer, Analytics, and Machine Learning using AWS Services (1)
• Abstract
• Introduction
• Transferring genomics data to the Cloud and establishing data access patterns using AWS DataSync and AWS Storage Gateway for files
• Recommendations
• Reference architecture
• File-based access to Amazon S3
• Running secondary analysis workflows using AWS Step Functions and AWS Batch
• Recommendations
• Reference architecture
• Performing tertiary analysis with data lakes using AWS Glue and Amazon Athena
• Recommendations
• Reference architecture
• Performing tertiary analysis with machine learning using Amazon SageMaker
• Recommendations
• Reference architecture
• Conclusion
• Appendix A: Genomics report pipeline reference architecture
• Appendix B: Research data lake ingestion pipeline reference architecture
• Appendix C: Genomics data transfer, analytics, and machine learning reference architecture
• Appendix D: Compliance resources
• Appendix E: Optimizing data transfer, cost, and performance
• Appendix F: Optimizing storage cost and data lifecycle management
• Appendix G: Optimizing secondary analysis compute cost
• Appendix H: Handling secondary analysis task errors and workflow failures
• Appendix I: Monitoring secondary analysis workflow status, cost, and performance
• Appendix J: Scaling secondary analysis
• Appendix K: Optimizing the performance of data lake queries
• Appendix L: Optimizing the cost of data lake queries
• Contributors
• Document revisions
• Notices