Amazon FSx for Lustre Integrations

Amazon Web Services (AWS) is the world’s most comprehensive cloud platform, trusted by millions of customers across industries. From startups to global enterprises and government agencies, AWS provides on-demand solutions for compute, storage, networking, AI, analytics, and more. The platform empowers organizations to innovate faster, reduce costs, and scale globally with unmatched flexibility and reliability. With services like Amazon EC2 for compute, Amazon S3 for storage, SageMaker for AI/ML, and CloudFront for content delivery, AWS covers nearly every business and technical need. Its global infrastructure spans 120 availability zones across 38 regions, ensuring resilience, compliance, and security. Backed by the largest community of customers, partners, and developers, AWS continues to lead the cloud industry in innovation and operational expertise.

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements. Amazon S3 is designed for 99.999999999% (11 9's) of durability, and stores data for millions of applications for companies all around the world. Scale your storage resources up and down to meet fluctuating demands, without upfront investments or resource procurement cycles. Amazon S3 is designed for 99.999999999% (11 9’s) of data durability.

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment. Amazon EC2 delivers the broadest choice of compute, networking (up to 400 Gbps), and storage services purpose-built to optimize price performance for ML projects. Build, test, and sign on-demand macOS workloads. Access environments in minutes, dynamically scale capacity as needed, and benefit from AWS’s pay-as-you-go pricing. Access the on-demand infrastructure and capacity you need to run HPC applications faster and cost-effectively. Amazon EC2 delivers secure, reliable, high-performance, and cost-effective compute infrastructure to meet demanding business needs.

AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted. With AWS Batch, there is no need to install and manage batch computing software or server clusters that you use to run your jobs, allowing you to focus on analyzing results and solving problems. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute services and features, such as AWS Fargate, Amazon EC2 and Spot Instances. There is no additional charge for AWS Batch. You only pay for the AWS resources (e.g. EC2 instances or Fargate jobs) you create to store and run your batch jobs.

Amazon SageMaker is an advanced machine learning service that provides an integrated environment for building, training, and deploying machine learning (ML) models. It combines tools for model development, data processing, and AI capabilities in a unified studio, enabling users to collaborate and work faster. SageMaker supports various data sources, such as Amazon S3 data lakes and Amazon Redshift data warehouses, while ensuring enterprise security and governance through its built-in features. The service also offers tools for generative AI applications, making it easier for users to customize and scale AI use cases. SageMaker’s architecture simplifies the AI lifecycle, from data discovery to model deployment, providing a seamless experience for developers.

The Lustre file system is an open-source, parallel file system that supports many requirements of leadership class HPC simulation environments. Whether you’re a member of our diverse development community or considering the Lustre file system as a parallel file system solution, these pages offer a wealth of resources and support to meet your needs. The Lustre file system provides a POSIX-compliant file system interface, which can scale to thousands of clients, petabytes of storage, and hundreds of gigabytes per second of I/O bandwidth. The key components of the Lustre file system are the Metadata Servers (MDS), the Metadata Targets (MDT), Object Storage Servers (OSS), Object Server Targets (OST), and the Lustre clients. Lustre is purpose-built to provide a coherent, global POSIX-compliant namespace for very large-scale computer infrastructure, including the world's largest supercomputer platforms. It can support hundreds of petabytes of data storage.

Amazon Elastic Kubernetes Service (Amazon EKS) is a fully managed Kubernetes service. Customers such as Intel, Snap, Intuit, GoDaddy, and Autodesk trust EKS to run their most sensitive and mission-critical applications because of its security, reliability, and scalability. EKS is the best place to run Kubernetes for several reasons. First, you can choose to run your EKS clusters using AWS Fargate, which is serverless compute for containers. Fargate removes the need to provision and manage servers, lets you specify and pay for resources per application, and improves security through application isolation by design. Second, EKS is deeply integrated with services such as Amazon CloudWatch, Auto Scaling Groups, AWS Identity and Access Management (IAM), and Amazon Virtual Private Cloud (VPC), providing you a seamless experience to monitor, scale, and load-balance your applications.

Amazon FSx makes it easy and cost effective to launch, run, and scale feature-rich, high-performance file systems in the cloud. It supports a wide range of workloads with its reliability, security, scalability, and broad set of capabilities. Amazon FSx is built on the latest AWS compute, networking, and disk technologies to provide high performance and lower TCO. And as a fully managed service, it handles hardware provisioning, patching, and backups -- freeing you up to focus on your applications, your end users, and your business.

AWS High Performance Computing (HPC) services empower users to execute large-scale simulations and deep learning workloads in the cloud, providing virtually unlimited compute capacity, high-performance file systems, and high-throughput networking. This suite of services accelerates innovation by offering a broad range of cloud-based tools, including machine learning and analytics, enabling rapid design and testing of new products. Operational efficiency is maximized through on-demand access to compute resources, allowing users to focus on complex problem-solving without the constraints of traditional infrastructure. AWS HPC solutions include Elastic Fabric Adapter (EFA) for low-latency, high-bandwidth networking, AWS Batch for scaling computing jobs, AWS ParallelCluster for simplified cluster deployment, and Amazon FSx for high-performance file systems. These services collectively provide a flexible and scalable environment tailored to diverse HPC workloads.

AWS DataSync is a secure, online service that automates and accelerates moving data between on-premises storage and AWS Storage services. It simplifies migration planning and reduces expensive on-premises data movement costs with a fully managed service that seamlessly scales as data loads increase. DataSync can copy data between Network File System (NFS) shares, Server Message Block (SMB) shares, Hadoop Distributed File Systems (HDFS), self-managed object storage, AWS Snowcone, Amazon Simple Storage Service (Amazon S3) buckets, Amazon Elastic File System (Amazon EFS) file systems, Amazon FSx for Windows File Server file systems, Amazon FSx for Lustre file systems, Amazon FSx for OpenZFS file systems, and Amazon FSx for NetApp ONTAP file systems. It also supports moving data between other public clouds and AWS Storage services, enabling replication, archival, or sharing of application data easily. DataSync provides end-to-end security, including data encryption and data integrity.