Last updated on

Object Storage

💾 Object Storage: Concept and Comparison

Object storage is a modern storage mechanism that has gained prominence, particularly in cloud environments (e.g., AWS S3, Google Cloud Storage, Azure Blob Storage).

  • Comparison to File Systems: Unlike traditional file systems, object storage stores data in a flat structure. There is no true hierarchy (no folders or directory trees). While services like S3 may offer the illusion of folders, this is typically achieved by making the “folder name” part of the object’s unique key/name.
  • Comparison to Databases: Object storage is more comparable to a file system than a transactional database. Databases are optimized for structured data, complex querying, filtering, and searching. Object storage is optimized for storing, retrieving, and managing large, unstructured data items.
  • Predecessor: The concept of an “object” evolves from BLOB (Binary Large Object) storage. The term “object” in object storage is synonymous with “blob.”

🔑 Core Characteristics of Object Storage

Data Structure and Access

  • Flat Structure: All data objects are stored at the same level—there is no path-based navigation.
  • Key-Value Access: Each object has a globally unique key (name). This allows for direct, immediate access to the object, much like retrieving a value from a hash map using its unique key. This direct access provides a performance advantage.
  • No Duplicates: The requirement for a unique key means that duplicate names are not allowed, even across different tenants of the service (a requirement for globally unique names in services like S3).

The Object Itself

  • Content: Objects are typically large files or media, such as:
    • Images and Videos
    • Large media files
    • Database backups
    • Other files intended for long-term storage

Object Immutability (Write/Read/Update/Delete)

  • Optimized for Write-Once, Read-Many: A critical characteristic is the inability to update or edit a file in place.
  • Allowed Operations: You can write (add) a new object and read (retrieve) an existing object.
  • Tradeoff: If a file needs to be “updated,” the old object must be deleted, and an entirely new object with the same key (or a new key) is written. This simplifies the underlying storage architecture and improves scalability.

🌐 Object Storage Interface and Use Cases

Interface

The primary interface for reading and writing files to object storage is HTTP (or HTTPS).

  • Simple Network Request: To retrieve a file, the application makes a direct network request (e.g., an HTTP GET request) using the object’s unique name.
  • No Querying: This eliminates the need for complex SQL queries, filtering, or traversing file trees.
  • Direct Delivery: This allows the application to offload the serving of large files. For instance, an application can simply provide the object storage URL to a user’s browser, allowing the user to download the video or image directly from the object store, bypassing the application server entirely.

System Design Use Cases

Object storage is the standard solution for handling large files in modern application design.

  • Storing Large Files/Media: It is optimized for storing all application media (images, videos, documents). Storing these in a database would slow down queries, inflate storage costs, and increase read/write load on the database server.
  • Long-Term Storage and Archiving: Due to its high durability and scalability, it’s ideal for backups, logs, and other data requiring long-term persistence.

In system design interviews, the rule of thumb is: any time you deal with large, relatively static files, use object storage.