Learn Ceph - Object Storage with RGW

#Introduction

Welcome to episode 7 of the Learn Ceph series! You've mastered the first two interfaces: RBD for block and CephFS for file. Now for the third: object storage with RGW (RADOS Gateway). This is Ceph's entrance into the S3 world that dominates modern cloud.

RGW provides an API compatible with Amazon S3 and OpenStack Swift, so the tooling, SDKs, and libraries already in the S3 ecosystem can be used directly without changes. This object storage is ideal for backup, media, data lakes, and archives.

In this episode we'll cover the RGW concepts, setting up realms, zones, and zonegroups, managing users, buckets, and objects, and integration with S3 clients and object lifecycle.

#RGW Concepts and S3/Swift

#What Is the RADOS Gateway

RGW is a daemon that provides object storage with an HTTP interface. It translates REST S3/Swift requests into RADOS operations: each object is stored as a RADOS object, metadata is stored in the rgw_metadata pool, and bucket indexes live in dedicated pools.

A single RGW deployment can serve many tenants and many users. Because it's stateless, RGW can scale horizontally behind a load balancer, and for multi-site we'll cover that in episode 13.

#S3 and Swift Compatibility

RGW supports most of the S3 API: bucket operations, object PUT/GET/DELETE, multipart upload, versioning, lifecycle, and access policies. For Swift, an endpoint with different authentication but the same basic object functionality is available.

S3   : https://rgw.example.com/  (port 443/7480)
Swift: https://rgw.example.com/swift/v1/

#Setting Up Realms, Zones, and Zonegroups

#The Logical Separation Structure

RGW organizes deployments in a hierarchy: realm → zonegroup → zone. A realm is the top-level isolation unit, a zonegroup is a collection of zones with replication policies, and a zone is the physical data unit.

For a simple single-site deployment, cephadm automatically creates the default realm, zonegroup, and zone when the first RGW is deployed:

ceph orch apply rgw myrgw --placement="3 node1 node2 node3"
ceph orch ps --daemon-type rgw

ceph orch apply rgw myrgw deploys three RGW instances with the realm name myrgw. We'll dissect the full zone structure for multi-site in episode 13.

#Viewing the Realm Structure

Checking the current configuration:

radosgw-admin realm list
radosgw-admin zonegroup list
radosgw-admin zone list

radosgw-admin realm list shows the existing realms. The zonegroup list and zone list commands show the hierarchy beneath them. For single-site deployments, one default realm is enough.

#User, Bucket, and Object Management

#Creating a User and Access Keys

Every RGW client is represented by a user. Create a user to access the S3 API:

radosgw-admin user create --uid=alice \
  --display-name="Alice User" \
  --email=alice@example.com

The command output includes an access_key and secret_key — these are the credentials S3 clients use. Store them securely; RGW doesn't show the full secret key on subsequent calls.

#Creating Buckets and Uploading Objects

With the credentials above, you can use the aws cli for bucket and object operations:

aws configure --profile rgw
aws --endpoint-url https://rgw.example.com s3 mb s3://first-bucket
aws --endpoint-url https://rgw.example.com s3 cp report.pdf s3://first-bucket/

aws s3 mb s3://first-bucket creates a bucket, and aws s3 cp uploads a file. Note that every command must include --endpoint-url pointing to RGW.

#Creating Buckets with radosgw-admin

Buckets can also be created directly from the admin side:

radosgw-admin bucket create --bucket=first-bucket --uid=alice
radosgw-admin bucket list

radosgw-admin bucket list shows all buckets along with their metadata. Access credentials still follow the bucket's owner user.

#S3 Client Integration and Lifecycle

#Various S3 Clients

Because the API is S3-compatible, almost any S3 client can use RGW: aws cli, s3cmd, boto3, rclone, and SDK libraries in many languages. As an example with Python:

Upload with boto3

import boto3
 
s3 = boto3.client(
    "s3",
    endpoint_url="https://rgw.example.com",
    aws_access_key_id="ACCESS",
    aws_secret_access_key="SECRET",
)
 
s3.upload_file("report.pdf", "first-bucket", "report.pdf")
print("upload complete")

The boto3.client(..., endpoint_url="https://rgw.example.com") code points the client at RGW instead of AWS. All other boto3 operations work as usual.

#Object Lifecycle

RGW supports lifecycle rules to manage objects automatically — for example moving objects to another storage class or deleting them after a set period:

{
  "Rules": [
    {
      "ID": "delete-old",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/" },
      "Expiration": { "Days": 90 }
    }
  ]
}

The rule above deletes objects with the logs/ prefix after 90 days. RGW lifecycle is evaluated periodically by the rgw daemon, and the configuration is stored as internal metadata.

#Conclusion

In this episode you've understood and practiced RGW object storage: the concepts and S3/Swift compatibility, the realm-zonegroup-zone hierarchy, user and bucket creation, integration with aws cli and boto3, and object lifecycle.

The key takeaways:

• RGW translates S3/Swift requests into RADOS operations.
• Realms, zonegroups, and zones manage logical separation and replication.
• Users are created with radosgw-admin user create, which produces access keys.
• aws cli and boto3 use --endpoint-url or endpoint_url to point at RGW.
• Lifecycle rules automate object expiration and movement.
• RGW is stateless, so it's easy to scale horizontally.

In the next episode, episode 8, we'll cover Ceph client integration — connecting Linux clients to CephFS and RBD, configuring RADOS Gateway clients, integrating with Kubernetes CSI drivers and cloud platforms, and best practices for client authentication and keyrings. Time for all your interfaces to be used from the client side!