Audience
Research institutions requiring a file system solution to manage their large-scale, data-intensive computing environments
About Lustre
The Lustre file system is an open-source, parallel file system that supports many requirements of leadership class HPC simulation environments. Whether you’re a member of our diverse development community or considering the Lustre file system as a parallel file system solution, these pages offer a wealth of resources and support to meet your needs. The Lustre file system provides a POSIX-compliant file system interface, which can scale to thousands of clients, petabytes of storage, and hundreds of gigabytes per second of I/O bandwidth. The key components of the Lustre file system are the Metadata Servers (MDS), the Metadata Targets (MDT), Object Storage Servers (OSS), Object Server Targets (OST), and the Lustre clients. Lustre is purpose-built to provide a coherent, global POSIX-compliant namespace for very large-scale computer infrastructure, including the world's largest supercomputer platforms. It can support hundreds of petabytes of data storage.
Other Popular Alternatives & Related Software
Amazon FSx for Lustre
Amazon FSx for Lustre is a fully managed service that provides high-performance, scalable storage for compute-intensive workloads. Built on the open-source Lustre file system, it offers sub-millisecond latencies, up to hundreds of gigabytes per second of throughput, and millions of IOPS, making it ideal for applications such as machine learning, high-performance computing, video processing, and financial modeling. FSx for Lustre integrates seamlessly with Amazon S3, allowing you to link file systems to S3 buckets. This integration enables transparent access and processing of S3 data from a high-performance file system, with the ability to import and export data between FSx for Lustre and S3. The service supports multiple deployment options, including scratch file systems for temporary storage and persistent file systems for long-term storage, as well as SSD and HDD storage types to optimize cost and performance based on workload requirements.
Learn more
Amazon EC2 UltraClusters
Amazon EC2 UltraClusters enable you to scale to thousands of GPUs or purpose-built machine learning accelerators, such as AWS Trainium, providing on-demand access to supercomputing-class performance. They democratize supercomputing for ML, generative AI, and high-performance computing developers through a simple pay-as-you-go model without setup or maintenance costs. UltraClusters consist of thousands of accelerated EC2 instances co-located in a given AWS Availability Zone, interconnected using Elastic Fabric Adapter (EFA) networking in a petabit-scale nonblocking network. This architecture offers high-performance networking and access to Amazon FSx for Lustre, a fully managed shared storage built on a high-performance parallel file system, enabling rapid processing of massive datasets with sub-millisecond latencies. EC2 UltraClusters provide scale-out capabilities for distributed ML training and tightly coupled HPC workloads, reducing training times.
Learn more
AWS DataSync
AWS DataSync is a secure, online service that automates and accelerates moving data between on-premises storage and AWS Storage services. It simplifies migration planning and reduces expensive on-premises data movement costs with a fully managed service that seamlessly scales as data loads increase. DataSync can copy data between Network File System (NFS) shares, Server Message Block (SMB) shares, Hadoop Distributed File Systems (HDFS), self-managed object storage, AWS Snowcone, Amazon Simple Storage Service (Amazon S3) buckets, Amazon Elastic File System (Amazon EFS) file systems, Amazon FSx for Windows File Server file systems, Amazon FSx for Lustre file systems, Amazon FSx for OpenZFS file systems, and Amazon FSx for NetApp ONTAP file systems. It also supports moving data between other public clouds and AWS Storage services, enabling replication, archival, or sharing of application data easily. DataSync provides end-to-end security, including data encryption and data integrity.
Learn more
MooseFS
MooseFS is a breakthrough concept in the Big Data storage industry. It allows us to combine data storage and data processing in a single unit using commodity hardware thereby providing an extremely high ROI. Through this innovative approach, we provide professional services and expert advisory for storage solutions as well as implementation and support for all your operations.
MooseFS, which was introduced in 2008 as a spin-off of Gemius (a leading European company which measures internet in over 20 countries), went on to become one of the most sought after Data storage software for companies worldwide. It is still being used to store huge amounts of data for Gemius core operations, where over 300 000 events are gathered and processed per second, for 24 hours and 7 days a week. Therefore, any solution we present to our customers has been tested in real-life work environment involving Big Data Analytics.
Learn more
Pricing
Starting Price:
Free
Free Version:
Free Version available.
Integrations
Company Information
OpenSFS and EOFS
United States
www.lustre.org
Videos and Screen Captures
Other Useful Business Software
MongoDB Atlas runs apps anywhere
MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
Product Details
Platforms Supported
Cloud
Windows
Mac
Linux
Training
Documentation
Support
Online