Learn Ceph - Object Lifecycle & S3 Features

#Introduction

Welcome to episode 17 of the Learn Ceph series! In episode 7 you created buckets and uploaded basic objects. Now we level up to object lifecycle & S3 features — RGW capabilities that make it comparable to commercial cloud object storage services.

Good object storage doesn't just store objects; it manages their lifecycle too. Objects can have versions, expire automatically, be moved to cheaper storage classes, and become sources of data for analytics. RGW supports all of this through the standard S3 API.

By the end of this episode you'll understand how to configure bucket lifecycle and versioning, build object policies, handle large objects with multipart upload, apply data retention features for compliance, and integrate object storage with analytics pipelines. Let's get started.

#Bucket Lifecycle and Versioning

#Enabling Versioning

Bucket versioning makes every overwrite produce a new version of the object instead of deleting the old one. It's a simple protection against accidental overwrites:

aws --endpoint-url https://rgw.example.com \
  s3api put-bucket-versioning \
  --bucket bucket-pertama \
  --versioning-configuration Status=Enabled

put-bucket-versioning enables versioning on the bucket. Once active, every PUT creates a new version and every DELETE creates a delete marker — the old object can still be recovered.

#Restoring an Old Version

With versioning active, restoring an object to a previous version is straightforward:

aws --endpoint-url https://rgw.example.com \
  s3api get-object \
  --bucket bucket-pertama \
  --key laporan.pdf \
  --version-id <versi-id> restore.pdf

get-object --version-id retrieves a specific version of an object. If a file was overwritten by accident, you can pull the version from before the overwrite and restore it.

#Building a Bucket Lifecycle

Lifecycle rules automate the transition and deletion of objects based on age. Here's a rule that deletes old objects:

{
  "Rules": [
    {
      "ID": "arsip-lama",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/" },
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}

The rule above moves objects with the logs/ prefix to a cheaper storage class after 30 days, then deletes them after 365 days. RGW evaluates this rule periodically through the lifecycle daemon.

#Object Policies and Multipart Upload

#Bucket and Object Policies

An object policy controls who can access a particular object. Policies can be attached to a bucket or to individual objects, and are JSON-based with an S3-compatible format. We covered an example policy granting read access to another user at the bucket level in episode 10; object-level policies work similarly but target a single object.

#Multipart Upload for Large Objects

For large objects, uploading directly in a single request is prone to failure and slow. Multipart upload splits the object into several parts uploaded in parallel, then merges them:

aws --endpoint-url https://rgw.example.com \
  s3 cp video-besar.mkv s3://bucket-pertama/ \
  --multipart-chunksize 64MB

--multipart-chunksize makes the aws cli use multipart upload automatically. For objects over 100 MB, multipart is almost always the right choice — besides being faster, the upload can resume from the parts that succeeded.

#Doing Multipart Manually

Through the API, the multipart process consists of three steps: CreateMultipartUpload, UploadPart, and CompleteMultipartUpload. This flow lets you upload parts from several threads at once and merge them with one final request.

#GDPR, Compliance, and Data Retention

#Retention Policies

For compliance, data sometimes must be kept for a minimum period and can't be deleted earlier. RGW supports this through a combination of versioning, access policies that restrict deletes, and lifecycles that start counting from the object's creation.

aws --endpoint-url https://rgw.example.com \
  s3api put-bucket-policy \
  --bucket bucket-audit \
  --policy '{"Statement":[{"Effect":"Deny","Action":["s3:DeleteObject"],"Resource":["arn:aws:s3:::bucket-audit/*"],"Principal":"*"}]}'

Deny s3:DeleteObject prevents object deletion in the audit bucket. For truly undeletable retention, pair the deny policy with very restricted access to that bucket's resources.

#Data Access for Audits

Enable access logging so every operation is recorded for audit purposes:

radosgw-admin log enable --bucket=bucket-audit

radosgw-admin log enable enables access logging for a specific bucket. This log data becomes compliance evidence when auditors ask for an access trail on sensitive data.

#Integrating Object Storage with Analytics

#Object Storage as a Data Lake

Object storage is the foundation of the modern data lake: cheap, scalable, and able to hold data in any format. RGW can be the backend for query engines like Apache Spark, Trino, or Presto that read data directly from an S3-compatible endpoint.

aws --endpoint-url https://rgw.example.com s3 ls s3://bucket-data/

aws s3 ls lists objects ready to be consumed by analytics engines. As long as the engine supports an S3 endpoint, RGW can be a data source without major pipeline changes.

#Event Notifications for Pipelines

So analytics can react to new data, RGW supports bucket notifications to message queues like Kafka or AMQP. Every new object triggers an event that can start processing — we'll dig deeper into this topic in episode 18 when discussing the Ceph ecosystem.

#Conclusion

In this episode you've understood advanced RGW object storage features: enabling bucket versioning and building lifecycle rules, managing object policies and multipart upload for large objects, applying GDPR and data retention features for compliance, and integrating object storage with analytics engines as a data lake.

The key takeaways:

• Versioning keeps every object version and protects against overwrites.
• Lifecycle rules automate object transitions and deletions based on age.
• Multipart upload is needed for large objects and resumable uploads.
• Deny policies and access logs support GDPR and retention requirements.
• RGW object storage can be a data lake backend for analytics engines.
• Bucket notifications connect new data to processing pipelines.

In the next episode, episode 18, we'll cover extending the Ceph ecosystem — additional services like CephFS snapshots, RBD mirroring, and RGW bucket notifications, integration with Ansible, Terraform, and automation, building custom monitoring tooling, and how to contribute to the Ceph open-source ecosystem. Time to go beyond built-in features!