Cloud Parallel File Systemsv1.0

1. Executive Summary

The latest organizational initiatives involving artificial intelligence (AI), machine learning (ML), high performance computing (HPC), and containerized application spaces necessitate a robust storage solution. One of the main selection criteria for storage is the file system and its capabilities.

What a solution for large, high-performance data needs today is a parallel file system. These systems operate on a clustered basis to improve throughput and IOP performance for process workloads and may leverage object-based storage to allow for virtually unlimited scaling and resiliency. However, integrating a parallel file system into a complex production environment may require additional components and storage technologies to support data movement in and out of the HPC cluster. This creates more points of failure, bottlenecks, and architectural and operational complexity.

The WEKA Data Platform aims to mitigate these challenges by simplifying the HPC storage design and supporting workloads, both on-premises and in the public cloud. The WekaFS file system is at the core of the WEKA data platform and contains built-in storage tiering, snapshot and snap-to-object backup capabilities, file system cloning, encryption at rest and in-flight, and support for a host of storage protocols. Organizations looking for a parallel file system, object-based storage, or a high-performance storage option often look to the public cloud for guidance.

We benchmarked the usability, effort, and performance of the WEKA Data Platform against Amazon FSx for Lustre on AWS. In this hands-on benchmark, we found that WEKA provided comparable or superior usability and outperformed FSx for Lustre at similar capacities by up to 300% or more. On some of our tests, WekaFS IO latency was less than 30% that of FSx for Lustre. Our usability tests also found WEKA to be a mature and easily deployed and operated solution in AWS specifically.