etcd ā Extended Technical Detail
What is etcd in Simple Terms?
etcd is the memory of your entire Kubernetes cluster. Every object you create ā pods, deployments, secrets, configmaps, nodes ā is stored in etcd. If etcd goes down or gets corrupted, your entire cluster loses its state. It is the most critical component to protect and back up.
ā DIAGRAM
+------------------------------------------+| kubectl apply -f deployment.yaml | <- You submit a resource+------------------------------------------+ | v+------------------------------------------+| API Server validates the request | <- Authorisation + validation+------------------------------------------+ | v+------------------------------------------+| etcd stores the desired state | <- Single source of truth| "5 replicas of trading-api expected" | persisted to disk+------------------------------------------+ | v+------------------------------------------+| Controller Manager reads from etcd | <- Reconciliation loop begins| Scheduler reads from etcd | to match actual to desired+------------------------------------------+What etcd Stores
ā DIAGRAM
+-----------------------------+| Kubernetes Secrets | <- Base64 encoded (encrypt at rest in prod)+-----------------------------+| ConfigMaps | <- App configuration data+-----------------------------+| Pod and Deployment specs | <- All workload desired state+-----------------------------+| Node registrations | <- kubelet heartbeats and node status+-----------------------------+| RBAC roles and bindings | <- All access control policies+-----------------------------+| Service endpoints | <- ClusterIP, port mappings+-----------------------------+| CRD objects | <- Custom resources (Argo, Istio, etc.)+-----------------------------+etcd Cluster Architecture ā Why 3 or 5 Nodes
etcd uses the Raft consensus protocol. It requires a quorum (majority) of members to be available to accept writes:
ā DIAGRAM
+------------------------+ +------------------------+ +------------------------+| etcd-node-1 (leader) | | etcd-node-2 (follower) | | etcd-node-3 (follower) || | | | | || mumbai-control-1 | -> | mumbai-control-2 | -> | mumbai-control-3 || Accepts writes | | Replicates from leader | | Replicates from leader |+------------------------+ +------------------------+ +------------------------+ 3-node cluster: quorum = 2. Can survive 1 node failure.5-node cluster: quorum = 3. Can survive 2 node failures.Single node: quorum = 1. Zero fault tolerance ā one outage = full cluster loss.How to Check etcd Health
Bash
1# Check etcd pod status in the control plane namespace2kubectl get pods -n kube-system | grep etcd3 4# Check etcd cluster health using etcdctl (run from a control plane node)5ETCDCTL_API=3 etcdctl \6 --endpoints=https://127.0.0.1:2379 \7 --cacert=/etc/kubernetes/pki/etcd/ca.crt \8 --cert=/etc/kubernetes/pki/etcd/server.crt \9 --key=/etc/kubernetes/pki/etcd/server.key \10 endpoint health11 12# Expected healthy output:13# https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 3.2ms14 15# Check all members of the etcd cluster16ETCDCTL_API=3 etcdctl \17 --endpoints=https://127.0.0.1:2379 \18 --cacert=/etc/kubernetes/pki/etcd/ca.crt \19 --cert=/etc/kubernetes/pki/etcd/server.crt \20 --key=/etc/kubernetes/pki/etcd/server.key \21 member list22 23# Output:24# 3a57933972cb5131, started, mumbai-control-1, https://10.0.1.10:2380, https://10.0.1.10:237925# 857c7a8d3a27c8b3, started, mumbai-control-2, https://10.0.1.11:2380, https://10.0.1.11:237926# b6246cfd09d5e09c, started, mumbai-control-3, https://10.0.1.12:2380, https://10.0.1.12:2379How to Back Up etcd
Bash
1# Take a snapshot backup ā automate this daily in production2ETCDCTL_API=3 etcdctl snapshot save \3 /backup/etcd-snapshot-$(date +%Y%m%d-%H%M).db \4 --endpoints=https://127.0.0.1:2379 \5 --cacert=/etc/kubernetes/pki/etcd/ca.crt \6 --cert=/etc/kubernetes/pki/etcd/server.crt \7 --key=/etc/kubernetes/pki/etcd/server.key8 9# Verify the snapshot integrity immediately after saving10ETCDCTL_API=3 etcdctl snapshot status \11 /backup/etcd-snapshot-20250610-0200.db \12 --write-out=table13 14# Output:15# +----------+----------+------------+------------+16# | HASH | REVISION | TOTAL KEYS | TOTAL SIZE |17# +----------+----------+------------+------------+18# | fe01cf57 | 198008 | 1521 | 3.7 MB |19# +----------+----------+------------+------------+How to Restore etcd from Backup
Bash
1# Step 1 ā Restore the snapshot to a new data directory2ETCDCTL_API=3 etcdctl snapshot restore \3 /backup/etcd-snapshot-20250610-0200.db \4 --data-dir=/var/lib/etcd-restored \5 --name=mumbai-control-1 \6 --initial-cluster="mumbai-control-1=https://10.0.1.10:2380" \7 --initial-advertise-peer-urls=https://10.0.1.10:23808 9# Step 2 ā Update the etcd static pod manifest to point to the new data dir10# Edit /etc/kubernetes/manifests/etcd.yaml:11# --data-dir=/var/lib/etcd-restored12 13# Step 3 ā kubelet auto-restarts etcd with the restored stateetcd Encryption at Rest
By default, Kubernetes Secrets are stored in etcd as base64 ā not encrypted. Enable encryption using an EncryptionConfiguration:
YAML
1# /etc/kubernetes/enc/encryption-config.yaml2apiVersion: apiserver.config.k8s.io/v13kind: EncryptionConfiguration4resources:5 - resources:6 - secrets7 providers:8 - aescbc: # AES-CBC encryption9 keys:10 - name: key111 secret: <base64-encoded-32-byte-key>12 - identity: {} # Fallback: existing unencrypted secrets still readable
Bash
1# Apply by adding this flag to kube-apiserver static pod manifest:2# --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml3 4# Verify a secret is encrypted in etcd (should show gibberish, not base64 text)5ETCDCTL_API=3 etcdctl get \6 /registry/secrets/production/db-password \7 --endpoints=https://127.0.0.1:2379 \8 --cacert=/etc/kubernetes/pki/etcd/ca.crt \9 --cert=/etc/kubernetes/pki/etcd/server.crt \10 --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C | headTroubleshooting Common etcd Problems
| Problem | Symptom | Fix |
|---|---|---|
| etcd pod CrashLoopBackOff | All kubectl commands fail with etcdserver: no leader |
Check data directory permissions and disk space ā etcd stops writing when disk is full |
| etcd leader election loop | Frequent leader changes in member list output | High network latency between control plane nodes ā check inter-node latency, etcd needs <10ms |
| Slow API server responses | kubectl commands take 5-30 seconds | etcd has too many keys ā run etcdctl defrag to compact and defragment the database |
| Snapshot restore failed | snapshot file doesn't exist |
Wrong path or snapshot corrupted ā always run snapshot status after saving to validate |
| Secrets visible as plain text in etcd | etcdctl get shows base64 text |
Encryption at rest not enabled ā add EncryptionConfiguration to kube-apiserver |
ā ļø Security: etcd stores Kubernetes Secrets in base64 encoding ā not encrypted by default. Anyone with direct etcd access can read all secrets in the cluster. Enable encryption at rest using EncryptionConfiguration. For a platform like Zerodha handling financial transaction data and API keys, unencrypted etcd is a critical compliance gap.š Remember: Always run etcd as a 3-node or 5-node cluster in production for high availability. A single etcd node is a single point of failure for your entire cluster ā one node crash means no new pods, no config changes, and no service deployments until it recovers.
š” Tip: Set up an automated daily etcd snapshot job via a CronJob that writes to an off-cluster storage bucket (AWS S3 ap-south-1, GCS). The snapshot is only useful if it is stored somewhere other than the cluster itself ā a cluster-wide failure would destroy snapshots stored on the same nodes.š“ Common Mistake: Runningetcdctlwithout theETCDCTL_API=3prefix. The default API version is v2, which uses a different command syntax and will silently return empty results or errors when your cluster is running etcd v3. Always prefix withETCDCTL_API=3.