Learning Kubernetes - Episode 21.1 - PersistentVolume and PersistentVolumeClaim

#Introduction

Note

If you want to read the previous episode, you can click the Episode 21 thumbnail below

In the previous episode, we explored Volumes in Kubernetes and touched briefly on persistentVolumeClaim as a volume type. In this episode (21.1), we'll go deeper and focus entirely on PersistentVolume (PV) and PersistentVolumeClaim (PVC) — the two core objects that Kubernetes uses to decouple storage provisioning from application consumption.

Note: Here I'll be using a Kubernetes Cluster installed through K3s.

Without PVs and PVCs, every team that deploys a stateful application has to manually manage storage backends. With them, the storage lifecycle can be managed independently from the application lifecycle — making it possible to run databases, message queues, and file storage reliably in Kubernetes.

#What Is a PersistentVolume?

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using a StorageClass. It is a cluster-level resource — meaning it exists independently of any Pod or Namespace.

Think of a PV like a physical disk that IT has formatted and made available. It's there waiting to be used. The specifics of where that disk lives (NFS server, cloud block storage, local disk) are abstracted behind the PV API object.

Key characteristics of PersistentVolume:

Cluster-scoped - Not bound to any specific Namespace
Independent lifecycle - Exists beyond the lifecycle of any Pod
Pluggable backends - NFS, iSCSI, cloud volumes (EBS, GCE PD, Azure Disk), local disk, CSI drivers
Access modes - RWO, ROX, RWX, RWOP
Reclaim policies - What happens to the PV when it is released
Storage capacity - Fixed capacity defined at creation time

#What Is a PersistentVolumeClaim?

A PersistentVolumeClaim (PVC) is a request for storage by a user or application. It is Namespace-scoped and specifies requirements such as storage size and access mode. Kubernetes finds a suitable PV that satisfies the claim and binds them together.

Think of PVC like a purchase order. The developer says: "I need 10Gi of read-write storage." Kubernetes then finds a PV that matches and binds the two together.

Key characteristics of PersistentVolumeClaim:

Namespace-scoped - Belongs to a specific Namespace
Declarative request - Specifies size, access mode, StorageClass
Binding - Kubernetes binds it to a matching PV
Pod-consumable - Used in Pod spec as a volume source
Dynamic provisioning - Can trigger PV creation automatically via StorageClass

#PV and PVC Lifecycle

Understanding the full lifecycle prevents data loss and misconfiguration.

#Phase 1: Provisioning

PVs can be provisioned in two ways:

Static provisioning - An admin manually creates PV objects pointing to existing storage
Dynamic provisioning - A StorageClass automatically creates a PV when a matching PVC is submitted

#Phase 2: Binding

Kubernetes control plane watches for unbound PVCs and matches them to available PVs based on:

Requested access mode
Requested storage size (PV must be >= PVC request)
StorageClass name (if specified)
Label selectors (optional)

Once a match is found, both the PV and PVC move to Bound state. A PV can only be bound to one PVC at a time.

#Phase 3: Using

A Pod references the PVC by name. Kubernetes mounts the underlying storage into the container at the specified path.

#Phase 4: Release

When a PVC is deleted, the PV enters the Released state. What happens next depends on the Reclaim Policy:

Retain - PV is kept, data preserved, requires manual cleanup before re-use
Delete - PV and the underlying storage are deleted automatically
Recycle (deprecated) - Basic scrub (rm -rf) and make available again

#Phase 5: Reclaiming

After Released, a PV with Retain policy holds data and cannot be bound to a new PVC without manual intervention. An admin must delete and recreate the PV (or clean the underlying storage) to reuse it.

#PersistentVolume Spec

A PV manifest defines the storage backend and its characteristics.

#Static PV Example (Local Path)

#Static PV Example (NFS)

#PV Spec Fields Explained

Field	Description
`capacity.storage`	The storage size this PV offers
`accessModes`	How the volume can be mounted
`storageClassName`	Links PV to a StorageClass
`persistentVolumeReclaimPolicy`	What happens when PVC is deleted
`volumeMode`	`Filesystem` (default) or `Block`
`nodeAffinity`	Constrain PV to specific nodes (for local volumes)

#PersistentVolumeClaim Spec

A PVC is simpler — it describes what the application needs without specifying how or where.

#Basic PVC Example

#PVC with Label Selector

Tip

Use label selectors when you want to bind a PVC to a specific PV, not just any PV that satisfies the size and access mode requirements. This is useful for ensuring a particular application always gets the same storage.

#Access Modes

Access modes define how a volume can be mounted across nodes. Not all storage backends support all modes.

Mode	Short	Description
`ReadWriteOnce`	RWO	Read-write by a single node
`ReadOnlyMany`	ROX	Read-only by many nodes
`ReadWriteMany`	RWX	Read-write by many nodes
`ReadWriteOncePod`	RWOP	Read-write by a single Pod (K8s 1.22+)

Warning

The access mode is a capability declaration, not an enforcement mechanism at the node level. ReadWriteOnce means the volume can only be mounted as read-write on one node at a time — but multiple Pods on the same node can use it simultaneously.

#Reclaim Policies

Reclaim policy controls what happens to the underlying storage when a PVC is deleted.

#Retain

PV moves to Released state
Data is preserved on the storage backend
PV cannot be rebound to a new PVC automatically
Admin must manually delete and recreate the PV to reuse it
Best for: Production databases where data loss is unacceptable

#Delete

PV and underlying storage resource are deleted automatically
Best for: Dynamic provisioning where ephemeral storage is acceptable (dev/test environments)

#Comparing Retain vs Delete

apiVersion: v1
kind: PersistentVolume
metadata:
    name: db-pv-production
spec:
    capacity:
        storage: 100Gi
    accessModes:
        - ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: fast-ssd
    # ... backend spec
Use Retain for production databases — data survives PVC deletion

apiVersion: v1
kind: PersistentVolume
metadata:
    name: cache-pv-dev
spec:
    capacity:
        storage: 5Gi
    accessModes:
        - ReadWriteOnce
    persistentVolumeReclaimPolicy: Delete
    storageClassName: standard
    # ... backend spec
Use Delete for dev/test — storage is cleaned up automatically

Use Retain for production databases — data survives PVC deletion

#StorageClass and Dynamic Provisioning

With static provisioning, an admin must pre-create PVs. With dynamic provisioning, a StorageClass automatically creates a PV when a matching PVC is submitted — no admin intervention needed.

#Defining a StorageClass

#StorageClass with Cloud Provisioner (AWS EBS)

#Dynamic PVC using StorageClass

When this PVC is created, the fast-ssd StorageClass triggers automatic PV creation and binding. No manual PV creation needed.

Tip

Set a default StorageClass in your cluster so that PVCs that don't specify a storageClassName are automatically handled. In K3s, local-path is the default StorageClass.

#Practical Implementation

#Example 1: PostgreSQL with Static PV

This is the most common pattern for self-managed databases: an admin pre-creates a PV on a fast local disk, and the database workload claims it.

apiVersion: v1
kind: PersistentVolume
metadata:
    name: postgres-pv
    labels:
        app: postgres
spec:
    storageClassName: manual
    capacity:
        storage: 20Gi
    accessModes:
        - ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    hostPath:
        path: /data/k8s/postgres
        type: DirectoryOrCreate

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: postgres-pvc
    namespace: default
spec:
    storageClassName: manual
    accessModes:
        - ReadWriteOnce
    resources:
        requests:
            storage: 20Gi
    selector:
        matchLabels:
            app: postgres

apiVersion: apps/v1
kind: Deployment
metadata:
    name: postgres
    namespace: default
spec:
    replicas: 1
    selector:
        matchLabels:
            app: postgres
    template:
        metadata:
            labels:
                app: postgres
        spec:
            containers:
                - name: postgres
                  image: postgres:15
                  ports:
                      - containerPort: 5432
                  env:
                      - name: POSTGRES_DB
                        value: mydb
                      - name: POSTGRES_USER
                        value: admin
                      - name: POSTGRES_PASSWORD
                        valueFrom:
                            secretKeyRef:
                                name: postgres-secret
                                key: password
                      - name: PGDATA
                        value: /var/lib/postgresql/data/pgdata
                  volumeMounts:
                      - name: postgres-storage
                        mountPath: /var/lib/postgresql/data
            volumes:
                - name: postgres-storage
                  persistentVolumeClaim:
                      claimName: postgres-pvc

#Example 2: Shared NFS Storage for Multiple Pods

When multiple Pods need to read and write the same shared storage (e.g., a legacy file upload service with multiple replicas), use ReadWriteMany with an NFS-backed PV.

#Example 3: Checking PV and PVC Status

After applying manifests, verify binding:

Expected output for a healthy bound PV:

bash

NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   AGE
postgres-pv    20Gi       RWO            Retain           Bound    default/postgres-pvc     manual         5m
nfs-shared-pv  100Gi      RWX            Retain           Bound    production/nfs-shared-pvc nfs           5m

Expected output for a healthy bound PVC:

bash

NAMESPACE    NAME              STATUS   VOLUME          CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default      postgres-pvc      Bound    postgres-pv     20Gi       RWO            manual         5m
production   nfs-shared-pvc    Bound    nfs-shared-pv   100Gi      RWX            nfs            5m

#Example 4: Volume Expansion

If your application outgrows the original PVC size, you can expand it (if the StorageClass has allowVolumeExpansion: true).

Apply the change and Kubernetes will trigger a resize operation on the underlying storage.

Caution

Volume shrinking is not supported. You can only expand a PVC, never reduce it. Plan your initial size conservatively, but not too small — leave room for growth.

#Common Mistakes and Pitfalls

#Mistake 1: PVC Stuck in Pending State

Symptom: PVC stays Pending indefinitely.

Causes and fixes:

Cause	Fix
No PV with matching access mode	Create a PV with correct access mode
PV capacity is smaller than PVC request	Increase PV capacity or reduce PVC request
StorageClass mismatch	Ensure `storageClassName` matches between PV and PVC
No default StorageClass	Add a default StorageClass or specify one explicitly

#Mistake 2: Data Loss After Pod Restart

Problem: Data disappears after Pod restarts.

Root cause: Using emptyDir or hostPath instead of a PVC.

#Mistake 3: Deleting PVCs in Production

Problem: Accidentally deleting a PVC deletes the application's data.

Prevention: Use Retain reclaim policy and protect PVCs with finalizers or RBAC policies that prevent unintended deletion.

Warning

A PVC deletion with Delete reclaim policy will permanently destroy the underlying storage and all its data. There is no undo. Always use Retain for production stateful workloads.

#Mistake 4: Wrong Namespace for PVC

Problem: Pod cannot find the PVC.

Root cause: PVCs are Namespace-scoped. A Pod in namespace: production cannot reference a PVC in namespace: default.

#Mistake 5: Using hostPath PV in Multi-Node Clusters

Problem: Pod is rescheduled to a different node and storage data is no longer available.

Root cause: hostPath PVs are tied to a specific node. When a Pod moves to another node, it doesn't find the data.

Fix: Use nodeAffinity to pin the Pod to the node with the hostPath data, or better — use proper networked storage (NFS, CSI driver, cloud volumes).

#Best Practices

#Always Set Reclaim Policy Explicitly

Don't rely on defaults. Always set persistentVolumeReclaimPolicy explicitly:

#Use StorageClasses for Dynamic Provisioning

Avoid static PV management at scale. Define clear StorageClasses for different tiers:

#Label Your PVs

Label PVs for easy filtering and selector-based binding:

#Enable Volume Expansion on StorageClass

Always enable allowVolumeExpansion so you can grow PVCs without downtime:

#Use `WaitForFirstConsumer` Volume Binding Mode

This delays PV binding until a Pod is scheduled, ensuring the PV is created in the same availability zone as the Pod:

Important

Without WaitForFirstConsumer, a PV can be provisioned in a different availability zone than where the Pod lands, causing a scheduling failure. This is especially important in multi-zone cloud environments.

#Monitor PV and PVC Status

Integrate PV/PVC status into your observability stack:

#When NOT to Use PersistentVolumes

PVs and PVCs add complexity. Sometimes simpler solutions are right:

Truly stateless applications — Don't add a PVC just because you can. If your app stores nothing, use emptyDir or no volume at all.
ConfigMaps and Secrets for configuration — Don't persist config in a PVC. Use configMap and secret volumes.
Short-lived batch jobs — If a Job processes data and the output doesn't need to persist beyond the job, emptyDir is fine.
External managed databases — If your organization runs databases outside Kubernetes (RDS, Cloud SQL, Managed PostgreSQL), don't replicate them inside the cluster with PVCs. Use the managed service and reference it via connection strings.

Note

PVs and PVCs shine when you need to persist application state inside the cluster. For everything else, lean on managed services or simpler volume types.

#Conclusion

In episode 21.1, we've covered PersistentVolume (PV) and PersistentVolumeClaim (PVC) in depth. These two objects are the backbone of stateful application storage in Kubernetes.

Key takeaways:

PersistentVolume is a cluster-scoped storage resource independent of any Pod
PersistentVolumeClaim is a Namespace-scoped request for storage
Kubernetes binds a PVC to a matching PV based on access mode, capacity, and StorageClass
Static provisioning requires manual PV creation; dynamic provisioning uses a StorageClass
Reclaim Policy (Retain vs Delete) controls what happens to data when PVC is deleted
Always use Retain for production databases to prevent accidental data loss
Access modes (RWO, ROX, RWX, RWOP) must be supported by the underlying storage backend
Use WaitForFirstConsumer in multi-zone clusters to avoid zone-mismatch failures
Enable allowVolumeExpansion on StorageClasses for online volume growth

Understanding PV and PVC puts you in control of storage management in Kubernetes — a critical skill for running stateful applications reliably in production.

Are you getting a clearer picture of how Kubernetes manages persistent storage? Keep the momentum going and look forward to the next episode!

Note

If you want to continue to the next episode, you can click the Episode 22 thumbnail below

Episode 22

Learning Kubernetes - Episode 21.1 - PersistentVolume and PersistentVolumeClaim

Related Posts