Configuring Persistent Volumes and Storage Classes in Kubernetes
Overview and What You Will Learn
Containers are ephemeral by design — when a pod restarts, all data written to its filesystem is lost. For databases, message queues, and any stateful workload, this is catastrophic. Kubernetes solves this through PersistentVolumes (PV), PersistentVolumeClaims (PVC), and StorageClasses — a three-layer abstraction that decouples storage provisioning from storage consumption, allowing pods to survive restarts, rescheduling, and node failures without losing data.
By the end of this guide you will be able to:
- Understand the PV, PVC, and StorageClass relationship and provisioning lifecycle
- Create StorageClasses for dynamic volume provisioning on AWS, GCP, and on-prem clusters
- Write PersistentVolumeClaims and mount volumes correctly inside pod specs
- Configure volume access modes and reclaim policies for different production workloads
- Troubleshoot PVC stuck in Pending state and volume mount failures
Why This Matters in Production
Zerodha's trading platform stores order books, trade history, and user portfolio data in PostgreSQL running on Kubernetes. If the PostgreSQL pod restarts without a PersistentVolume, every trade record since the last external backup is lost — a regulatory violation and a catastrophic user trust failure.
At Hotstar, the video transcoding pipeline writes intermediate encoded segments to shared storage that multiple pods must read simultaneously. The wrong access mode on the PVC causes silent data corruption or outright mount failures. Understanding storage configuration is not optional for any engineer running stateful workloads on Kubernetes.
Core Principles
The three-layer storage abstraction and how they compose:CLUSTER ADMIN DEVELOPER APPLICATION ───────────── ───────── ─────────── StorageClass PersistentVolumeClaim Pod spec (defines HOW (requests WHAT (mounts PVC storage is storage is needed: as a volume provisioned: size, access mode, at a path) AWS EBS, GCP PD, storage class) NFS, local disk) │ │ │ └────────────────────────────────┴───────────────────────────┘ │ PersistentVolume (PV) (the actual provisioned storage unit — created automatically by StorageClass or manually by admin for static provisioning)
Access modes — the most misunderstood configuration in Kubernetes storage:ReadWriteOnce (RWO) → One node can mount read-write at a time Used for: databases, single-instance stateful apps Supported by: AWS EBS, GCP Persistent Disk, Azure DiskReadWriteMany (RWX) → Multiple nodes can mount read-write simultaneously Used for: shared file storage, media assets, ML datasets Supported by: NFS, AWS EFS, GCP Filestore, Azure Files NOT supported by: EBS, GCP PD, Azure DiskReadOnlyMany (ROX) → Multiple nodes can mount read-only simultaneously Used for: shared config files, static assets
Detailed Step-by-Step Practical Lab
Step 1 — Inspect Available StorageClasses
NIX
1kubectl get storageclassesExample output on an AWS EKS cluster:2NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE3gp2 (default) ebs.csi.aws.com Delete WaitForFirstConsumer4gp3-encrypted ebs.csi.aws.com Retain WaitForFirstConsumer5efs-sc efs.csi.aws.com Retain ImmediateInspect a specific StorageClass for full configuration6kubectl describe storageclass gp3-encrypted7 8> 📌 **Remember:** `RECLAIMPOLICY: Delete` means the underlying cloud disk is permanently deleted when the PVC is deleted. `RECLAIMPOLICY: Retain` keeps the disk even after PVC deletion — always use Retain for production databases.9 10#### Step 2 — Create Production StorageClasses11 12```yamlstorageclasses.yaml — define storage tiers for different workload typesTier 1: Fast encrypted SSD for databases (AWS gp3)13apiVersion: storage.k8s.io/v114kind: StorageClass15metadata:16name: gp3-encrypted17annotations:18storageclass.kubernetes.io/is-default-class: "false"19provisioner: ebs.csi.aws.com20parameters:21type: gp322iops: "6000" # 6000 IOPS — good for PostgreSQL/MySQL23throughput: "250" # 250 MB/s throughput24encrypted: "true" # Encrypt at rest — required for financial data25kmsKeyId: "arn:aws:kms:ap-south-1:123456789:key/zerodha-ebs-key"26reclaimPolicy: Retain # NEVER auto-delete production database disks27allowVolumeExpansion: true # Allow resizing PVCs without downtime28volumeBindingMode: WaitForFirstConsumer # Provision in same AZ as pod29Tier 2: Shared file storage for media assets (AWS EFS — supports RWX)30apiVersion: storage.k8s.io/v131kind: StorageClass32metadata:33name: efs-shared34provisioner: efs.csi.aws.com35parameters:36provisioningMode: efs-ap37fileSystemId: fs-0a1b2c3d4e5f6789 # Your EFS filesystem ID38directoryPerms: "700"39reclaimPolicy: Retain40volumeBindingMode: Immediate41Tier 3: Fast local NVMe for temporary high-performance workloads42apiVersion: storage.k8s.io/v143kind: StorageClass44metadata:45name: local-nvme46provisioner: kubernetes.io/no-provisioner # Manual provisioning47volumeBindingMode: WaitForFirstConsumer48reclaimPolicy: Delete # Local storage is node-specific — delete on release49 50```bashkubectl apply -f storageclasses.yaml51 52#### Step 3 — Create PersistentVolumeClaims for Different Workloads53 54```yamlpvcs.yaml — storage claims for different production workloadsPVC for PostgreSQL database — single node, high IOPS, encrypted55apiVersion: v156kind: PersistentVolumeClaim57metadata:58name: postgres-data-pvc59namespace: production60labels:61app: postgres62team: platform63spec:64accessModes:65- ReadWriteOnce # Only one node mounts at a time — correct for databases66storageClassName: gp3-encrypted67resources:68requests:69storage: 100Gi # Start with 100GB — can expand later without downtime70PVC for Hotstar video asset storage — multiple pods read/write simultaneously71apiVersion: v172kind: PersistentVolumeClaim73metadata:74name: video-assets-pvc75namespace: transcoding76spec:77accessModes:78- ReadWriteMany # Multiple transcoding pods mount simultaneously79storageClassName: efs-shared80resources:81requests:82storage: 5Ti # 5TB for video asset storage83PVC for Redis cache persistence — small, fast, single node84apiVersion: v185kind: PersistentVolumeClaim86metadata:87name: redis-data-pvc88namespace: production89spec:90accessModes:91- ReadWriteOnce92storageClassName: gp3-encrypted93resources:94requests:95storage: 20Gi96 97```bashkubectl apply -f pvcs.yamlCheck PVC status — should move from Pending to Bound98kubectl get pvc -n production99NAME STATUS VOLUME CAPACITY100postgres-data-pvc Bound pvc-a1b2c3d4-e5f6-7890-abcd-ef1234567890 100Gi101redis-data-pvc Bound pvc-b2c3d4e5-f6a7-8901-bcde-f12345678901 20Gi102 103#### Step 4 — Mount PVCs into Pod Specs104 105```yamldeployment-postgres.yaml — PostgreSQL with persistent storage106apiVersion: apps/v1107kind: StatefulSet108metadata:109name: postgres110namespace: production111spec:112serviceName: postgres113replicas: 1114selector:115matchLabels:116app: postgres117template:118metadata:119labels:120app: postgres121spec:122securityContext:123fsGroup: 999 # PostgreSQL runs as UID 999 — set volume ownership124containers:125- name: postgres126image: postgres:15.4127env:128- name: POSTGRES_DB129value: zerodha_trading130- name: POSTGRES_USER131value: rahul132- name: POSTGRES_PASSWORD133valueFrom:134secretKeyRef:135name: postgres-credentials136key: password137- name: PGDATA138value: /var/lib/postgresql/data/pgdata # Subdirectory avoids lost+found issue139ports:140- containerPort: 5432141resources:142requests:143cpu: "500m"144memory: "1Gi"145limits:146cpu: "2"147memory: "4Gi"148volumeMounts:149- name: postgres-storage150mountPath: /var/lib/postgresql/data # PostgreSQL data directory151volumes:152- name: postgres-storage153persistentVolumeClaim:154claimName: postgres-data-pvc # Reference the PVC by name155 156```bashkubectl apply -f deployment-postgres.yamlVerify volume is mounted inside the pod157kubectl exec -it postgres-0 -n production -- df -h /var/lib/postgresql/data158Filesystem Size Used Avail Use% Mounted on159/dev/nvme1n1 98G 156M 98G 1% /var/lib/postgresql/data160 161> ⚠️ **Security:** Always set `securityContext.fsGroup` to match the UID your application runs as. Without it, the mounted volume is owned by root and your application process may fail to write to it — causing a crash that looks like a storage failure but is actually a permissions issue.162 163#### Step 5 — Expand a PVC Without Downtime164 165When your database grows beyond the initial allocation:166 167```bashVerify the StorageClass supports volume expansion168kubectl get storageclass gp3-encrypted -o jsonpath='{.allowVolumeExpansion}'169trueEdit the PVC to request more storage170kubectl patch pvc postgres-data-pvc -n production 171--type='json' 172-p='[{"op":"replace","path":"/spec/resources/requests/storage","value":"200Gi"}]'Watch the expansion happen173kubectl get pvc postgres-data-pvc -n production -w174NAME STATUS CAPACITY CONDITIONS175postgres-data-pvc Bound 100Gi Resizing...176postgres-data-pvc Bound 200Gi FileSystemResizePending177postgres-data-pvc Bound 200Gi ← expansion completeFor filesystem resize to complete — the pod may need a restart178kubectl rollout restart statefulset/postgres -n production179 180> 💡 **Tip:** Volume expansion only works in one direction — you can increase a PVC's size but never decrease it. Always start with a reasonable baseline and use `allowVolumeExpansion: true` on your StorageClass so you can grow without recreating the PVC.181 182#### Step 6 — Troubleshoot PVC Stuck in Pending State183 184```bashPVC not moving from Pending to Bound185kubectl get pvc postgres-data-pvc -n production186NAME STATUS VOLUME CAPACITY ACCESS MODES187postgres-data-pvc Pending ← stuckStep 1 — Describe the PVC for the reason188kubectl describe pvc postgres-data-pvc -n production189Events:190Warning ProvisioningFailed storageclass.storage.k8s.io "gp3-encrypted" not found191→ StorageClass name is wrong or not installedWarning ProvisioningFailed failed to provision volume:192InvalidParameterValue: The iops parameter is not supported for volume type gp2193→ Wrong parameters for the volume typeWarning WaitForFirstConsumer waiting for first consumer to be created194→ VolumeBindingMode is WaitForFirstConsumer —195PVC will stay Pending until a pod tries to mount it. This is normal.Step 2 — Check if the CSI driver is running196kubectl get pods -n kube-system | grep ebs-csi197ebs-csi-controller-xxx 6/6 Running 0 5d198ebs-csi-node-xxx 3/3 Running 0 5dStep 3 — Check CSI driver logs for provisioning errors199kubectl logs -n kube-system 200-l app=ebs-csi-controller 201-c csi-provisioner 202--tail=50203 204#### Step 7 — Implement Volume Snapshots for Backup205 206```yamlvolume-snapshot.yaml — take a point-in-time snapshot of the PostgreSQL volume207apiVersion: snapshot.storage.k8s.io/v1208kind: VolumeSnapshot209metadata:210name: postgres-snapshot-20250525211namespace: production212spec:213volumeSnapshotClassName: csi-aws-vsc214source:215persistentVolumeClaimName: postgres-data-pvc # Snapshot this PVC216 217```bashkubectl apply -f volume-snapshot.yamlCheck snapshot status218kubectl get volumesnapshot -n production219NAME READYTOUSE SOURCEPVC RESTORESIZE AGE220postgres-snapshot-20250525 true postgres-data-pvc 100Gi 2mRestore from snapshot into a new PVC221kubectl apply -f - <<EOF222apiVersion: v1223kind: PersistentVolumeClaim224metadata:225name: postgres-data-restored226namespace: production227spec:228accessModes:229- ReadWriteOnce230storageClassName: gp3-encrypted231resources:232requests:233storage: 100Gi234dataSource:235name: postgres-snapshot-20250525236kind: VolumeSnapshot237apiGroup: snapshot.storage.k8s.io238EOF239 240> 📌 **Remember:** Volume snapshots are crash-consistent, not application-consistent. For PostgreSQL, always run `pg_dump` or use `pg_basebackup` for application-consistent backups. Use volume snapshots as a fast recovery complement, not as your only backup strategy.241 242### Production Best Practices & Common Pitfalls243 244* Always use `PGDATA=/var/lib/postgresql/data/pgdata` (a subdirectory) for PostgreSQL on Kubernetes. Mounting directly to `/var/lib/postgresql/data` causes PostgreSQL to fail because the volume root contains a `lost+found` directory it cannot handle.245* Tag your PVCs with team and application labels — at scale, identifying which PVC belongs to which application becomes impossible without consistent labelling.246* Set up automated volume snapshot schedules using Velero or the cloud provider's native snapshot scheduler. A 100GB PostgreSQL volume with no snapshots is a single point of failure.247* Monitor PVC usage with `kubectl exec <pod> -- df -h` and alert at 80% full — Kubernetes does not automatically expand volumes and a full disk causes immediate pod failure.248* Never share a single RWO PVC between multiple pods. Only one node can mount it at a time — if a second pod tries to mount it on a different node, it will stay in Pending or ContainerCreating indefinitely.249 250> 🔴 **Common Mistake:** Using `reclaimPolicy: Delete` on production database StorageClasses. When a developer accidentally runs `kubectl delete pvc postgres-data-pvc`, the underlying cloud disk and all its data is permanently deleted within seconds. Always use `Retain` for any storage containing production data.251 252### Quick Reference & Troubleshooting Commands253 254| Command | Purpose |255|:---|:---|256| `kubectl get pvc -n <ns>` | List all PVCs and their binding status |257| `kubectl describe pvc <name> -n <ns>` | Full PVC details and provisioning events |258| `kubectl get pv` | List all PersistentVolumes cluster-wide |259| `kubectl get storageclass` | List available StorageClasses |260| `kubectl describe storageclass <name>` | Full StorageClass configuration |261| `kubectl patch pvc <name> -n <ns> --type='json' -p='[...]'` | Expand PVC size |262| `kubectl exec <pod> -n <ns> -- df -h` | Check disk usage inside a pod |263| `kubectl get volumesnapshot -n <ns>` | List volume snapshots |264| `kubectl logs -n kube-system -l app=ebs-csi-controller -c csi-provisioner` | Debug CSI provisioning failures |265| `kubectl get events -n <ns> --field-selector reason=ProvisioningFailed` | Filter storage provisioning failures |