What is StatefulSet? | DevOps Dictionary

StatefulSet — Running Stateful Apps in Kubernetes

Why Deployments Break for Databases

A standard Deployment treats all pods as identical and disposable. Pod names are random (postgres-7d9f8c-xkqzp), storage is not guaranteed to follow a pod, and all pods can start or die simultaneously.

For a PostgreSQL primary-replica setup at Zerodha, this is catastrophic:

◈ DIAGRAM

+------------------------------------------+
| Deployment (WRONG for databases)         |
|                                          |
| postgres-7d9f8c-xkqzp  <- random name   | <- If pod dies, new pod gets
| postgres-7d9f8c-ab1cd  <- random name   |    a different random name.
|                                          |    Replicas lose their target.
| Shared PVC: postgres-data                | <- Two pods writing to the same
|   ^ both pods mount this                |    disk = data corruption
+------------------------------------------+
 
+------------------------------------------+
| StatefulSet (CORRECT for databases)      |
|                                          |
| postgres-0  <- stable, permanent name   | <- Always the primary.
| postgres-1  <- stable, permanent name   |    Replicas always know where
| postgres-2  <- stable, permanent name   |    to replicate from.
|                                          |
| postgres-data-0  <- dedicated PVC       | <- Each pod owns its own disk.
| postgres-data-1  <- dedicated PVC       |    PVC survives pod restarts.
| postgres-data-2  <- dedicated PVC       |
+------------------------------------------+

The Three Guarantees StatefulSet Provides

1. Stable Network Identity

Each pod gets a permanent DNS name that never changes, even after restarts:

◈ DIAGRAM

+------------------------------------------+
| DNS record format:                       |
|                                          |
| <pod-name>.<headless-svc>.<ns>.svc.cluster.local
|                                          |
| postgres-0.postgres-svc.production.svc.cluster.local
| postgres-1.postgres-svc.production.svc.cluster.local
| postgres-2.postgres-svc.production.svc.cluster.local
+------------------------------------------+

2. Ordered Startup and Shutdown

◈ DIAGRAM

+------------+     +------------+     +------------+
| postgres-0 |     | postgres-1 |     | postgres-2 |
|            |     |            |     |            |
| Starts 1st | --> | Starts 2nd | --> | Starts 3rd |
| (primary)  |     | (Running?) |     | (Running?) |
+------------+     +------------+     +------------+
 
Shutdown order is reversed:
postgres-2 stops -> postgres-1 stops -> postgres-0 stops (primary last)

3. Persistent Volume Per Pod

Each pod gets its own PVC via volumeClaimTemplates. The PVC survives pod deletion — when the pod recreates, it reattaches to the same volume automatically.

A Real StatefulSet Manifest — Redis Cluster

YAML

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: production
spec:
  serviceName: redis-headless           # Must reference a Headless Service
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:7.2-alpine
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: redis-data
              mountPath: /data          # Each pod mounts its own /data volume
          resources:
            requests:
              memory: "256Mi"
              cpu: "100m"
            limits:
              memory: "512Mi"
              cpu: "500m"
  volumeClaimTemplates:                 # This is what makes it a StatefulSet
    - metadata:
        name: redis-data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3-encrypted
        resources:
          requests:
            storage: 10Gi              # Each of 3 pods gets its own 10Gi PVC
                                       # Total: 30Gi provisioned automatically

Headless Service — The Required Companion

StatefulSets require a Headless Service (clusterIP: None) to assign stable DNS records to each individual pod:

YAML

apiVersion: v1
kind: Service
metadata:
  name: redis-headless
  namespace: production
spec:
  clusterIP: None          # "Headless" — no virtual IP, just DNS per pod
  selector:
    app: redis
  ports:
    - port: 6379
      name: redis

◈ DIAGRAM

+------------------------------------------+    +------------------------------------------+
| Regular Service (clusterIP: 10.96.0.5)  |    | Headless Service (clusterIP: None)       |
|                                          |    |                                          |
| DNS: redis-svc.production.svc...         |    | DNS: redis-0.redis-headless.production.. |
| Routes to ANY redis pod (random)         |    | DNS: redis-1.redis-headless.production.. |
| Good for stateless read balancing        |    | Routes to SPECIFIC pod (by ordinal)      |
|                                          |    | Required for primary-replica topology    |
+------------------------------------------+    +------------------------------------------+

REMEMBER THIS
**Remember:** Without the Headless Service, pods don't get their individual DNS records. The `serviceName` in your StatefulSet spec MUST exactly match the Headless Service `metadata.name` — a mismatch means pods start but DNS never resolves.

Managing StatefulSet Pods

Bash

# List all pods — notice the stable ordinal names
kubectl get pods -n production -l app=redis
# NAME      READY   STATUS    RESTARTS   AGE
# redis-0   1/1     Running   0          5d
# redis-1   1/1     Running   0          5d
# redis-2   1/1     Running   0          5d
 
# List the auto-created PVCs (one per pod)
kubectl get pvc -n production -l app=redis
# NAME               STATUS   VOLUME                     CAPACITY
# redis-data-redis-0 Bound    pvc-3a8f2c1d-...           10Gi
# redis-data-redis-1 Bound    pvc-5c2e4b1a-...           10Gi
# redis-data-redis-2 Bound    pvc-7d3f9c2b-...           10Gi
 
# Scale up — redis-3 will be created with its own PVC
kubectl scale statefulset redis --replicas=4 -n production
 
# Rolling update (in reverse order: 2, 1, 0)
kubectl rollout status statefulset/redis -n production

StatefulSet vs Deployment — Decision Guide

Factor	Use Deployment	Use StatefulSet
App stores state to disk	No	Yes
Pods need unique identity	No	Yes
All pods are interchangeable	Yes	No
Startup order matters	No	Yes
Examples	API servers, web frontends	PostgreSQL, Kafka, Elasticsearch, Redis Cluster, ZooKeeper

Troubleshooting Common StatefulSet Problems

Problem	Symptom	Fix
Pod stuck in `Pending`	`redis-1` never starts	`redis-0` is not yet in `Running` state — fix the first pod before others proceed
PVC not binding	`redis-data-redis-0` stuck in `Pending`	StorageClass missing or wrong AZ — check `kubectl describe pvc redis-data-redis-0` events
Pod recreated but lost data	Data empty after restart	Pod reattached to wrong PVC — verify PVC name matches pod ordinal exactly
Headless Service DNS not resolving	`nslookup redis-0.redis-headless` fails	`serviceName` in StatefulSet spec doesn't match the Headless Service name
Scale-down left orphan PVCs	PVCs remain after `replicas: 1`	StatefulSet never auto-deletes PVCs on scale-down — delete manually after confirming data is safe

COMMON MISTAKE / WARNING
**Common Mistake:** Running a database as a Deployment with a single shared PVC. When the pod reschedules to a different node, the `ReadWriteOnce` PVC cannot follow to a different node, leaving the pod permanently stuck in `Pending`.

COMMON MISTAKE / WARNING
**Security:** Never run StatefulSet pods as `root` inside the container. A pod breakout on a database StatefulSet with root access can corrupt data or exfiltrate the entire volume. Always set `securityContext.runAsNonRoot: true` and `readOnlyRootFilesystem: true` on StatefulSet containers.

PLACEMENT PRO TIP
**Tip:** When upgrading StatefulSets (e.g., Redis version bump), use `updateStrategy: RollingUpdate` with `partition: 2` to canary-test the upgrade on only the last pod first. Once verified healthy, set `partition: 0` to roll out to all pods.