CNI — The Networking Foundation of Every Kubernetes Cluster
What is CNI in Simple Terms?
When Kubernetes creates a pod, it needs to give that pod a unique IP address and connect it to the cluster network so it can talk to every other pod — even on different nodes. CNI is the standard that defines how that networking is set up. Kubernetes itself has no built-in networking — it delegates 100% of pod networking to whichever CNI plugin you install.
Popular CNI plugins used in Indian production clusters:
| Plugin | Used By | Key Strength |
|---|---|---|
| Calico | Zerodha, banking platforms | Network policy enforcement, BGP routing |
| Cilium | High-traffic APIs, Hotstar-scale | eBPF-based, no iptables, best performance |
| Flannel | Dev/test clusters | Simple overlay, easy to set up |
| Weave | Legacy clusters | Simple mesh networking |
How CNI Works — Pod Creation Flow
◈ DIAGRAM
+--------------------------------------------------+| Pod scheduled on mumbai-prod-node-1 | <- Step 1: Scheduler assigns node+--------------------------------------------------+ | v+--------------------------------------------------+| Kubelet creates a network namespace for the pod | <- Step 2: Isolated network env+--------------------------------------------------+ | v+--------------------------------------------------+| Kubelet calls CNI plugin binary (/opt/cni/bin/) | <- Step 3: CNI takes over+--------------------------------------------------+ | v+--------------------------------------------------+| CNI assigns pod IP 10.244.1.15 via IPAM | <- Step 4: Unique IP allocated+--------------------------------------------------+ | v+--------------------------------------------------+| CNI creates veth pair: | <- Step 5: Virtual cable created| eth0 (inside pod ns) <-> veth0abc (cni0) |+--------------------------------------------------+ | v+--------------------------------------------------+| Routing rules added for 10.244.1.15 | <- Step 6: All nodes can reach pod+--------------------------------------------------+ | v+--------------------------------------------------+| Pod can now communicate with every pod | <- Step 7: Networking ready+--------------------------------------------------+What is a veth Pair?
A veth (virtual ethernet) pair is like a virtual network cable with two ends. CNI uses it to connect the isolated pod network namespace to the node's main network:
◈ DIAGRAM
+------------------------------+ +------------------------------+| Pod | | Node || | | || eth0 (10.244.1.15) | <------> | veth0abc -> cni0 bridge || (inside pod namespace) | | (node network namespace) |+------------------------------+ +------------------------------+Traffic leaving the pod goes through eth0 -> veth pair -> node bridge -> routed to destination.
CNI Plugin Comparison — Choosing for Production
| Feature | Calico | Cilium | Flannel |
|---|---|---|---|
| Network Policies | Full support | Full support | Not supported |
| Performance | Good | Excellent (eBPF) | Moderate |
| Observability | Basic | Deep (Hubble UI) | None |
| Encryption | WireGuard | WireGuard/IPSec | None |
| Complexity | Medium | High | Low |
| Best for | Policy-heavy prod | High-throughput prod | Dev/test |
How to Check Which CNI is Running
Bash
1# List CNI config files on any worker node2ls /etc/cni/net.d/3 4# Common output examples:5# 10-calico.conflist (Calico CNI)6# 10-flannel.conflist (Flannel CNI)7# 05-cilium.conf (Cilium CNI)8 9# View the full CNI config10cat /etc/cni/net.d/10-calico.conflist11 12# Check CNI plugin binaries installed13ls /opt/cni/bin/Installing Cilium CNI (Recommended for Production)
Bash
1# Install Cilium CLI2curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz3tar xzvf cilium-linux-amd64.tar.gz4sudo mv cilium /usr/local/bin5 6# Pin to a tested version in production — never use @latest on live clusters7cilium install --version 1.15.08 9# Verify all Cilium pods are running10cilium status --wait11 12# Run connectivity test13cilium connectivity testTroubleshooting CNI Issues
| Symptom | Likely Cause | Fix |
|---|---|---|
Pod stuck in ContainerCreating |
CNI config missing or broken | Check /etc/cni/net.d/ on the node |
All pods Pending after fresh cluster |
CNI plugin never installed | Install Calico, Cilium, or Flannel |
networkPlugin cni failed to set up pod |
CNI binary missing | Check /opt/cni/bin/ |
| Pods can't reach each other cross-node | Pod CIDR routing broken | Verify kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}' |
Bash
1# Describe pod to read CNI error detail2kubectl describe pod <pod-name> -n production3 4# Check CNI plugin logs (Calico example)5kubectl logs -n kube-system -l k8s-app=calico-node --tail=506 7# Confirm pod CIDRs are assigned per node8kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'9 10# SSH into node and confirm CNI config exists11ssh rahul@10.0.1.5012ls /etc/cni/net.d/💡 Tip: For production clusters running high-traffic workloads like Zerodha's trading engine or Hotstar's streaming nodes, use Cilium — it replaces iptables with eBPF for significantly better throughput and built-in observability via the Hubble UI.
📌 Remember: Kubernetes does NOT include any CNI plugin by default. On a bare kubeadm cluster, every pod will stay in Pending indefinitely until you install one. This is the single most common reason a fresh cluster appears broken.⚠️ Security: CNI plugins like Calico and Cilium support Kubernetes NetworkPolicy objects that restrict pod-to-pod traffic. Without a NetworkPolicy-capable CNI, any pod in your cluster can freely reach any other pod — a serious risk in multi-tenant environments at Razorpay or PhonePe.
🔴 Common Mistake: Switching CNI plugins on a live cluster without draining all nodes first. Migrating from Flannel to Cilium mid-flight will break all existing pod networking and cause a full cluster outage. Always plan CNI migrations during a maintenance window.