etcd — Extended Technical Detail
What is etcd in Simple Terms?
etcd is the memory of your entire Kubernetes cluster. Every object you create — pods, deployments, secrets, configmaps, nodes — is stored in etcd. If etcd goes down or gets corrupted, your entire cluster loses its state. It is the most critical component to protect and back up.
Bash
+------------------------------------------+| kubectl apply -f deployment.yaml | <- You submit a resource+------------------------------------------+ | v+------------------------------------------+| API Server validates the request | <- Authorisation + validation+------------------------------------------+ | v+------------------------------------------+| etcd stores the desired state | <- Single source of truth| "5 replicas of trading-api expected" | persisted to disk+------------------------------------------+ | v+------------------------------------------+| Controller Manager reads from etcd | <- Reconciliation loop begins| Scheduler reads from etcd | to match actual to desired+------------------------------------------+What etcd Stores
◈ DIAGRAM
+-----------------------------+| Kubernetes Secrets | <- Base64 encoded (encrypt at rest in prod)+-----------------------------+| ConfigMaps | <- App configuration data+-----------------------------+| Pod and Deployment specs | <- All workload desired state+-----------------------------+| Node registrations | <- kubelet heartbeats and node status+-----------------------------+| RBAC roles and bindings | <- All access control policies+-----------------------------+| Service endpoints | <- ClusterIP, port mappings+-----------------------------+| CRD objects | <- Custom resources (Argo, Istio, etc.)+-----------------------------+etcd Cluster Architecture — Why 3 or 5 Nodes
etcd uses the Raft consensus protocol. It requires a quorum (majority) of members to be available to accept writes:
◈ DIAGRAM
+------------------------+ +------------------------+ +------------------------+| etcd-node-1 (leader) | | etcd-node-2 (follower) | | etcd-node-3 (follower) || | | | | || mumbai-control-1 | -> | mumbai-control-2 | -> | mumbai-control-3 || Accepts writes | | Replicates from leader | | Replicates from leader |+------------------------+ +------------------------+ +------------------------+ 3-node cluster: quorum = 2. Can survive 1 node failure.5-node cluster: quorum = 3. Can survive 2 node failures.Single node: quorum = 1. Zero fault tolerance — one outage = full cluster loss.How to Check etcd Health
Bash
# Check etcd pod status in the control plane namespacekubectl get pods -n kube-system | grep etcd # Check etcd cluster health using etcdctl (run from a control plane node)ETCDCTL_API=3 etcdctl \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ endpoint health # Expected healthy output:# https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 3.2ms # Check all members of the etcd clusterETCDCTL_API=3 etcdctl \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ member list # Output:# 3a57933972cb5131, started, mumbai-control-1, https://10.0.1.10:2380, https://10.0.1.10:2379# 857c7a8d3a27c8b3, started, mumbai-control-2, https://10.0.1.11:2380, https://10.0.1.11:2379# b6246cfd09d5e09c, started, mumbai-control-3, https://10.0.1.12:2380, https://10.0.1.12:2379How to Back Up etcd
Bash
# Take a snapshot backup — automate this daily in productionETCDCTL_API=3 etcdctl snapshot save \ /backup/etcd-snapshot-$(date +%Y%m%d-%H%M).db \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key # Verify the snapshot integrity immediately after savingETCDCTL_API=3 etcdctl snapshot status \ /backup/etcd-snapshot-20250610-0200.db \ --write-out=table # Output:# +----------+----------+------------+------------+# | HASH | REVISION | TOTAL KEYS | TOTAL SIZE |# +----------+----------+------------+------------+# | fe01cf57 | 198008 | 1521 | 3.7 MB |# +----------+----------+------------+------------+How to Restore etcd from Backup
Bash
# Step 1 — Restore the snapshot to a new data directoryETCDCTL_API=3 etcdctl snapshot restore \ /backup/etcd-snapshot-20250610-0200.db \ --data-dir=/var/lib/etcd-restored \ --name=mumbai-control-1 \ --initial-cluster="mumbai-control-1=https://10.0.1.10:2380" \ --initial-advertise-peer-urls=https://10.0.1.10:2380 # Step 2 — Update the etcd static pod manifest to point to the new data dir# Edit /etc/kubernetes/manifests/etcd.yaml:# --data-dir=/var/lib/etcd-restored # Step 3 — kubelet auto-restarts etcd with the restored stateetcd Encryption at Rest
By default, Kubernetes Secrets are stored in etcd as base64 — not encrypted. Enable encryption using an EncryptionConfiguration:
YAML
# /etc/kubernetes/enc/encryption-config.yamlapiVersion: apiserver.config.k8s.io/v1kind: EncryptionConfigurationresources: - resources: - secrets providers: - aescbc: # AES-CBC encryption keys: - name: key1 secret: <base64-encoded-32-byte-key> - identity: {} # Fallback: existing unencrypted secrets still readable
Bash
# Apply by adding this flag to kube-apiserver static pod manifest:# --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml # Verify a secret is encrypted in etcd (should show gibberish, not base64 text)ETCDCTL_API=3 etcdctl get \ /registry/secrets/production/db-password \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C | headTroubleshooting Common etcd Problems
| Problem | Symptom | Fix |
|---|---|---|
| etcd pod CrashLoopBackOff | All kubectl commands fail with etcdserver: no leader |
Check data directory permissions and disk space — etcd stops writing when disk is full |
| etcd leader election loop | Frequent leader changes in member list output | High network latency between control plane nodes — check inter-node latency, etcd needs <10ms |
| Slow API server responses | kubectl commands take 5-30 seconds | etcd has too many keys — run etcdctl defrag to compact and defragment the database |
| Snapshot restore failed | snapshot file doesn't exist |
Wrong path or snapshot corrupted — always run snapshot status after saving to validate |
| Secrets visible as plain text in etcd | etcdctl get shows base64 text |
Encryption at rest not enabled — add EncryptionConfiguration to kube-apiserver |
COMMON MISTAKE / WARNING**Security:** etcd stores Kubernetes Secrets in base64 encoding — not encrypted by default. Anyone with direct etcd access can read all secrets in the cluster. Enable encryption at rest using `EncryptionConfiguration`. For a platform like Zerodha handling financial transaction data and API keys, unencrypted etcd is a critical compliance gap.
REMEMBER THIS**Remember:** Always run etcd as a 3-node or 5-node cluster in production for high availability. A single etcd node is a single point of failure for your entire cluster — one node crash means no new pods, no config changes, and no service deployments until it recovers.
PLACEMENT PRO TIP**Tip:** Set up an automated daily etcd snapshot job via a CronJob that writes to an off-cluster storage bucket (AWS S3 `ap-south-1`, GCS). The snapshot is only useful if it is stored somewhere other than the cluster itself — a cluster-wide failure would destroy snapshots stored on the same nodes.
COMMON MISTAKE / WARNING**Common Mistake:** Running `etcdctl` without the `ETCDCTL_API=3` prefix. The default API version is v2, which uses a different command syntax and will silently return empty results or errors when your cluster is running etcd v3. Always prefix with `ETCDCTL_API=3`.