The Business Case for Object Storage

Object storage is a data storage architecture tuned to the unique demands of unstructured data such as documents, email, images, video, and other media files. It differs from file storage, which keeps data in hierarchical folders, and block storage, which stores data across discrete, addressable blocks. It works by storing data as objects that contain data, metadata, and a unique ID to enable distributed access. By abstracting data as objects, it is easier to manage large amounts of unstructured data.

Object storage is abstracted from administrators and applications. It uses RESTful API and HTTP/S protocols to access data, organized into unique “objects,” stored in a flat namespace with rich metadata. RESTful APIs make it simpler to manage and gain access to data without having to create bespoke code or applications.

Object storage offers a host of benefits over other approaches, not least of which is its ability to store and access the vast—and rapidly growing—amounts of unstructured data in enterprises. Among the notable benefits:

• Durability: Based on resilient architecture, it can withstand multiple disk failures
• Availability: Due to RESTful APIs, data is accessible from anywhere
• Scalability: Object storage can scale to almost any size; services such as Facebook have hundreds of petabytes of data
• Affordability: Commodity hardware makes object storage affordable

Object storage makes managing large amounts of data easier, especially unstructured data such as video, and the data management allows easier scaling. In fact, most of the world’s large-scale applications run using object storage, often via the main three cloud storage providers, including Amazon, which first popularized object storage with its S3 service. Netflix, Amazon Prime, Disney+, Spotify, Dropbox, and Facebook all use it.

Emerging applications fueling uptake of object storage include big data analytics and large-scale artificial intelligence/machine learning (AI/ML) data lakes, as well as streaming data applications and data storage for Internet of Things (IoT). Object storage also continues for its original application—archival storage, due to speed and performance issues—but new, high-performance object storage is becoming more common.

File storage remains an alternative for legacy applications that still use a Portable Operating System Interface (POSIX), in which case a network-attached storage gateway should be set in front of the object store. This can increase cost and impair performance for some types of workloads. In some cases, primary storage vendors provide the S3 object storage protocol for their appliances, making it easier to manage small amounts of data without building an object storage system.

For large capacities, the alternative to object storage remains scale-out file storage with S3 interface, though the less feature-rich API can limit the number of use cases. The advantage of mating a file storage system with an S3 interface is that data can be accessed via two different protocols.

Because object storage allows easy encryption at rest and in transit, file storage could be less secure as well as more costly and less scalable. That said, object storage is not a panacea. On high-demand use cases, such as AI, managing metadata in real time can result in systems being overwhelmed. In addition, because object stores are traditionally set up to handle large files, they may handle smaller files inefficiently. Finally, because many object stores still save data chunks on a local file system in the back end, this can contribute to the creation of additional latency and performance issues.

30 Days: Start with a Plan
Evaluate major use cases and applications that can take advantage of the technology, both in the cloud and on-premises. Prioritize use cases to build an object storage infrastructure that can start small and grow to accommodate additional data and workloads.

60 Days: Develop a Proof of Concept
Implement a simple infrastructure and deploy a “canary” application. Work to build the skills necessary to manage a large object storage infrastructure and how to integrate it with the rest of the infrastructure.

90 Days: Proceed with Migration
Work on a migration and integration plan for all applications that can take advantage of object storage. Consolidate data on the object storage infrastructure and review capacity planning for the rest of the storage infrastructure after important data migrations.

• Enterprise Analytic Solutions 2020
• Blog: The Status of Object Storage
• Blog: Object Storage – Time for Differentiation and Diversification