Learn Ceph - Security & Access Control

#Introduction

Welcome to episode 10 of the Learn Ceph series! Your cluster got faster after the performance tuning in episode 9. But performance without security is a fragile foundation. This episode covers security & access control — the defense layer that protects data from unauthorized access.

Ceph protects data on several layers at once. At the internal level there's CephX for daemon and client authentication. At the object storage level there's RBAC for RGW users. And at the network level there are firewalls and TLS to protect data in transit. Each handles a different threat.

By the end of this episode you'll be able to configure CephX authentication, restrict capabilities with the least-privilege principle, secure RGW endpoints, and apply secure deployment practices for multi-tenant storage. Let's get started.

#CephX Authentication and Key Management

#How CephX Works

CephX is Ceph's authentication mechanism, similar to Kerberos. Every entity — daemon, client, or admin — has a key stored in a keyring. When a client wants to talk to the MON, it performs an encrypted handshake that's verified with the MON and the MON quorum.

ceph auth ls
ceph auth get client.admin

ceph auth ls shows all entities and their capabilities. Each entity is identified by a keyring containing a secret key. This key is never sent in plaintext during communication.

#Creating and Managing Keys

Creating new entities with restricted capabilities is a habit to build from the start:

ceph auth get-or-create client.backup \
  mon 'allow r' \
  osd 'allow rwx pool=backups'
ceph auth export client.backup -o /etc/ceph/ceph.client.backup.keyring

ceph auth get-or-create creates a new key if it doesn't exist yet, then displays or exports it. Store the exported keyring with strict permissions as covered in episode 8.

#Key Rotation

Rotating keys reduces the risk when a key leaks:

ceph auth rotate-key client.backup

ceph auth rotate-key replaces the key for the entity in question. Clients using the old key must have their keyring updated, so perform rotation during a maintenance window and coordinate with workload owners.

#RBAC for RGW Users and Capabilities

#Capabilities at the Ceph Level

At the Ceph level, access control is implemented through capabilities on entities. Capabilities define what an entity may do against MON, OSD, MDS, or RGW:

• mon 'allow rw': read and write the cluster maps.
• osd 'allow rwx pool=backups': read, write, and exec only on the backups pool.
• mds 'allow rw': filesystem metadata access.
• rgw 'allow *': full access to RGW.

ceph auth caps client.backup mon 'allow r' \
  osd 'allow rwx pool=backups' mds 'allow rw'

ceph auth caps replaces an entity's capabilities wholesale. Get into the habit of making capabilities as small as possible and expanding only when truly needed.

#RBAC for RGW Users

For object storage, RGW has its own user system. Each RGW user can be given a JSON-based policy that governs access to specific buckets and objects:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::first-bucket/*"]
    }
  ]
}

The policy above only allows GetObject on the first-bucket bucket. Attach this policy to the bucket with the radosgw-admin bucket policy command, or through the PutBucketPolicy API from an S3 client.

#Network Security, Firewalls, and TLS

#Opening the Required Ports

The node firewall must open the ports Ceph services use and close the rest. Key ports include 6789 for MON, 8443 for the dashboard, and 7480 or 443 for RGW:

firewall-cmd --permanent --add-port=6789/tcp
firewall-cmd --permanent --add-port=8443/tcp
firewall-cmd --permanent --add-port=7480/tcp
firewall-cmd --reload

firewall-cmd --permanent makes rules persist across reboots. Also restrict source addresses to only the subnets allowed to access the cluster network.

#TLS for RGW

S3 traffic to RGW should be encrypted with TLS. RGW supports certificates installed behind a reverse proxy like HAProxy, or directly with rgw_frontend_ssl_certificate:

ceph config set rgw rgw_frontend_ssl_certificate /etc/ceph/rgw.pem
ceph config set rgw rgw_frontend_ssl_key /etc/ceph/rgw-key.pem

rgw_frontend_ssl_certificate points to a certificate file combined with its chain. Always terminate TLS with a valid certificate from a trusted CA, especially for public endpoints.

#Secure Deployment for Multi-Tenant Storage

#Isolation Between Tenants

In a multi-tenant deployment, each tenant should be isolated with a combination of separate pools, keyrings, and RGW users. Don't put two tenants' data in the same pool with overlapping capabilities.

ceph osd pool create tenant-alpha 128
ceph auth get-or-create client.alpha \
  mon 'allow r' osd 'allow rwx pool=tenant-alpha'

client.alpha can only touch the tenant-alpha pool. With pool and key isolation, one tenant can't read or modify another tenant's data even on the same cluster.

#Additional Security Practices

Beyond what's been covered, apply these practices: enable data-at-rest encryption as required by compliance, disable unused accounts and keys, review audit logs regularly, and keep all admin keyrings in protected locations with restricted access. The combination of these practices makes your cluster hard to breach from either the credential or the network side.

#Conclusion

In this episode you've understood how to secure a Ceph cluster from multiple angles: CephX authentication and key management, Ceph-level capabilities and RGW user RBAC, network security with firewalls and TLS for RGW, and secure deployment practices with pool and keyring isolation per tenant.

The key takeaways:

• CephX authenticates every entity with a key stored in a keyring.
• Ceph-level capabilities restrict an entity's access to MON, OSD, MDS, and RGW.
• RGW users are governed by JSON-based policies for bucket and object access.
• Firewalls only open the required ports, and RGW should use TLS.
• Multi-tenant isolation is achieved with separate pools, keyrings, and users.
• Periodic key rotation and keyring audits keep credentials clean.

In the next episode, episode 11, we'll cover monitoring & observability — using the Ceph dashboard, Prometheus exporter, and Grafana dashboards, collecting health, OSD, pool, and RGW metrics, setting up alerting for cluster events and performance anomalies, and managing logs and audit trails. Time to make your cluster transparent!