What is the career path for learning Debugging Kubernetes Networking with kubectl and CNI Plugins?

Mastering Debugging Kubernetes Networking with kubectl and CNI Plugins enables engineering opportunities in DevOps, SRE, and cloud platform automation.

Debugging Kubernetes Networking with kubectl and CNI Plugins | DevOps Network

Q: How long does it take to learn Debugging Kubernetes Networking with kubectl and CNI Plugins?

Most students gain core proficiency in Debugging Kubernetes Networking with kubectl and CNI Plugins in 2–3 weeks of active hands-on labs.

Overview and What You Will Learn

Kubernetes networking failures are among the hardest production issues to diagnose — a pod can be running perfectly but completely unable to reach another service due to a CNI misconfiguration, a missing NetworkPolicy rule, a DNS resolution failure, or a kube-proxy iptables issue. This lab walks you through a systematic debugging methodology for every layer of Kubernetes networking using real kubectl commands, the netshoot debug container, and CNI-specific diagnostic tools.

By the end of this guide you will be able to:

Diagnose pod-to-pod, pod-to-service, and pod-to-external connectivity failures systematically
Debug DNS resolution failures using CoreDNS logs and nslookup inside pods
Identify CNI plugin misconfiguration causing pods to stay in ContainerCreating state
Trace kube-proxy iptables rules to verify Service routing is correctly programmed
Write and debug Kubernetes NetworkPolicies that restrict inter-pod traffic

Why This Matters in Production

At Zerodha, a silent network partition between the order matching engine and the risk management service caused trades to execute without risk checks for 4 minutes during a CNI upgrade — not detected by application logs because the service call was timing out silently and falling back to a cached value. The incident was only discovered through a networking-level trace.

Kubernetes networking has five distinct layers that can fail independently — the pod network interface, CNI plugin, kube-proxy service rules, CoreDNS, and NetworkPolicies. Engineers who cannot debug each layer independently will waste hours on production incidents that a systematic approach resolves in minutes.

Core Principles

The five networking layers in Kubernetes and their failure modes: Layer 1 — Pod Network Interface (veth pair + CNI assignment) Failure: Pod stuck in ContainerCreating, no IP assigned Tool: kubectl describe pod, ip addr inside pod Layer 2 — CNI Plugin (Calico / Cilium / Flannel overlay) Failure: Pod has IP but cannot reach pods on OTHER nodes Tool: CNI logs, calicoctl, cilium status Layer 3 — kube-proxy (iptables / IPVS Service rules) Failure: Pod can reach pod IPs directly but Service IP fails Tool: iptables -L -n, kubectl get endpoints Layer 4 — CoreDNS (cluster DNS resolution) Failure: Service IPs work but DNS names fail Tool: kubectl logs coredns, nslookup inside pod Layer 5 — NetworkPolicy (ingress/egress firewall rules) Failure: Connection refused or timeout with no obvious cause Tool: kubectl get networkpolicies, policy trace tools

Always debug from Layer 1 upward — never assume the problem is DNS when the pod might not have a network interface at all.

Detailed Step-by-Step Practical Lab

Step 1 — Deploy the netshoot Debug Container

netshoot is the standard Swiss Army knife for Kubernetes network debugging — it contains curl, nslookup, dig, tcpdump, netstat, traceroute, and dozens of other tools:

Bash

# Run netshoot as a temporary debug pod in the same namespace as the failing service
kubectl run netshoot \
  --image=nicolaka/netshoot \
  -it --rm \
  -n production \
  -- bash
 
# Run netshoot sharing the SAME network namespace as a specific pod
# (sees the exact same network interfaces and routes as that pod)
kubectl debug -it \
  --image=nicolaka/netshoot \
  --target=order-service-7d9f8b-xkp2q \
  order-service-7d9f8b-xkp2q \
  -n production \
  -- bash

REMEMBER THIS
**Remember:** `kubectl debug --target=` shares the pod's network namespace — it sees the same IP, the same routes, and the same DNS config as the application pod. This is the most accurate way to reproduce networking issues exactly as the application experiences them.

Step 2 — Debug Layer 1: Verify Pod Has a Network Interface and IP

Bash

# Check if pod has an IP assigned
kubectl get pod order-service-7d9f8b-xkp2q -n production -o wide
# NAME                           READY   STATUS    IP            NODE
# order-service-7d9f8b-xkp2q   0/1     ContainerCreating   <none>   mumbai-worker-1
 
# If IP is <none> — CNI has not assigned an address. Describe for the reason:
kubectl describe pod order-service-7d9f8b-xkp2q -n production
# Look for in Events:
# Failed to create pod sandbox: rpc error: code = Unknown
# desc = failed to set up sandbox container network: plugin type="calico"
# failed (add): error getting ClusterInformation
 
# Inside the pod (if it started) — verify network interface exists
ip addr show eth0
# 2: eth0@if45: <BROADCAST,MULTICAST,UP,LOWER_UP>
#     inet 10.244.2.15/24 brd 10.244.2.255 scope global eth0
 
# Check routing table
ip route show
# default via 169.254.1.1 dev eth0
# 10.244.0.0/16 via 10.244.2.1 dev eth0   ← cluster pod CIDR route

Step 3 — Debug Layer 2: Test Pod-to-Pod Connectivity Across Nodes

Bash

# Get IPs of pods on DIFFERENT nodes
kubectl get pods -n production -o wide | grep order-service
# order-service-7d9f8b-xkp2q   10.244.2.15   mumbai-worker-1
# order-service-7d9f8b-mn3lp   10.244.3.22   mumbai-worker-2  ← different node
 
# From inside netshoot — ping the pod on the other node directly by IP
ping 10.244.3.22
# PING 10.244.3.22: 56 data bytes
# 64 bytes from 10.244.3.22: icmp_seq=0 ttl=62 time=0.8ms  ← working
# Request timeout for icmp_seq 0                             ← CNI overlay broken
 
# If ping fails — check CNI pods on both nodes
kubectl get pods -n kube-system -o wide | grep -E "calico|cilium|flannel"
 
# Check CNI logs on the node where the failing pod lives
kubectl logs -n kube-system calico-node-xxxxx | tail -50
 
# For Calico — check node BGP peering status
kubectl exec -n kube-system calico-node-xxxxx -- calicoctl node status

Step 4 — Debug Layer 3: Verify kube-proxy Service Rules

Bash

# Test direct pod IP (bypasses kube-proxy completely)
# Inside netshoot:
curl http://10.244.2.15:8080/health
# 200 OK  ← pod is healthy
 
# Test Service ClusterIP (goes through kube-proxy iptables)
kubectl get service order-service -n production
# NAME            TYPE        CLUSTER-IP      PORT(S)
# order-service   ClusterIP   10.96.45.200    8080/TCP
 
curl http://10.96.45.200:8080/health
# Connection refused  ← kube-proxy rule is broken
 
# Verify the Service has endpoints (pods are registered as backends)
kubectl get endpoints order-service -n production
# NAME            ENDPOINTS                                   AGE
# order-service   10.244.2.15:8080,10.244.3.22:8080           5d  ← healthy
# order-service   <none>                                      5d  ← no pods matching selector
 
# If endpoints show <none> — the Service selector doesn't match pod labels
kubectl get service order-service -n production -o jsonpath='{.spec.selector}'
# {"app":"order-service"}
 
kubectl get pods -n production --show-labels | grep order-service
# order-service-7d9f8b-xkp2q   app=orders-service  ← typo! "orders" not "order"

Bash

# SSH onto the node and verify iptables rules were programmed correctly
ssh rahul@mumbai-worker-1
 
# Check if kube-proxy created rules for the Service ClusterIP
sudo iptables -t nat -L KUBE-SERVICES -n | grep 10.96.45.200
# KUBE-SVC-XYZ  tcp -- 0.0.0.0/0  10.96.45.200  tcp dpt:8080
 
# Check kube-proxy is running correctly
kubectl get pods -n kube-system | grep kube-proxy
kubectl logs -n kube-system kube-proxy-xxxxx | grep -i error

PLACEMENT PRO TIP
**Tip:** If the Service has correct endpoints but the ClusterIP still fails, the kube-proxy iptables rules may be stale. Restarting the kube-proxy DaemonSet pod on the affected node forces a full iptables resync: `kubectl delete pod kube-proxy-xxxxx -n kube-system`.

Step 5 — Debug Layer 4: CoreDNS and DNS Resolution Failures

Bash

# Inside netshoot — test DNS resolution step by step
# Test 1: Can we resolve the short service name?
nslookup order-service
# Server: 10.96.0.10  ← CoreDNS ClusterIP
# Address: 10.96.0.10#53
# ** server can't find order-service: NXDOMAIN  ← DNS failure
 
# Test 2: Try the full qualified name
nslookup order-service.production.svc.cluster.local
# 10.96.45.200  ← works with FQDN but not short name
 
# Test 3: Check the pod's DNS search domains
cat /etc/resolv.conf
# nameserver 10.96.0.10
# search production.svc.cluster.local svc.cluster.local cluster.local
# options ndots:5
 
# Test 4: Test external DNS (is it just internal or everything?)
nslookup google.com
# ** server can't find google.com: SERVFAIL  ← CoreDNS cannot reach upstream

Bash

# Check CoreDNS pod status
kubectl get pods -n kube-system | grep coredns
 
# Check CoreDNS logs for errors
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100
 
# Common CoreDNS error signatures:
# [ERROR] plugin/errors: 2 google.com. A: read udp → upstream timeout
# [ERROR] plugin/errors: 2 SERVFAIL → CoreDNS cannot reach upstream DNS
 
# Check CoreDNS ConfigMap for upstream DNS configuration
kubectl get configmap coredns -n kube-system -o yaml

YAML

# Fix — update CoreDNS ConfigMap to use reliable upstream resolvers
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4 {    # Use Google DNS as upstream
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

Bash

kubectl apply -f coredns-configmap.yaml
 
# Restart CoreDNS to pick up new config
kubectl rollout restart deployment/coredns -n kube-system

Step 6 — Debug Layer 5: NetworkPolicy Blocking Traffic

Bash

# Check if any NetworkPolicies exist in the namespace
kubectl get networkpolicies -n production
 
# If NetworkPolicies exist — describe them to see what they allow/deny
kubectl describe networkpolicy payments-isolation -n production
 
# Common symptom: curl to a pod IP works, curl to the same pod from
# a different namespace times out → NetworkPolicy is blocking cross-namespace traffic

YAML

# networkpolicy-debug.yaml — allow order-service to call payments-service
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-order-to-payments
  namespace: payments-production
spec:
  podSelector:
    matchLabels:
      app: payments-service       # This policy applies TO payments pods
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: orders-production   # Allow from orders namespace
          podSelector:
            matchLabels:
              app: order-service  # Only from order-service pods specifically
      ports:
        - protocol: TCP
          port: 4000

Bash

kubectl apply -f networkpolicy-debug.yaml
 
# For Cilium clusters — use Hubble for real-time policy trace
cilium hubble observe \
  --namespace production \
  --type drop \
  --last 100
# Shows every dropped packet with the policy rule that caused it
 
# For Calico clusters — use policy trace tool
kubectl exec -n kube-system calico-node-xxxxx -- \
  calicoctl policy trace \
  --src-pod production/order-service-7d9f8b-xkp2q \
  --dst-pod payments-production/payments-service-6c9d4f-mn3lp

COMMON MISTAKE / WARNING
**Security:** Once you create any NetworkPolicy in a namespace, all traffic not explicitly allowed is denied by default. This means adding your first NetworkPolicy to a namespace can instantly break all existing connectivity if you don't also add allow rules for every legitimate traffic flow.

Step 7 — Run a Full Connectivity Matrix Test

For a comprehensive network health check across all services:

Bash

# Deploy a test pod that checks connectivity to every service in the namespace
kubectl run connectivity-test \
  --image=nicolaka/netshoot \
  -n production \
  --restart=Never \
  -- sh -c "
    echo '=== Testing order-service ===' && \
    curl -s -o /dev/null -w '%{http_code}' http://order-service:8080/health && \
    echo '=== Testing payments-service ===' && \
    curl -s -o /dev/null -w '%{http_code}' http://payments-service.payments-production.svc.cluster.local:4000/health && \
    echo '=== Testing external DNS ===' && \
    nslookup google.com && \
    echo '=== All tests complete ==='
  "
 
# View results
kubectl logs connectivity-test -n production
 
# Clean up
kubectl delete pod connectivity-test -n production

Production Best Practices & Common Pitfalls

Always test cross-namespace connectivity after adding any NetworkPolicy. A policy in the target namespace affects all inbound traffic including from other namespaces that previously worked without any policy.
Use Cilium with Hubble in production — the real-time policy trace and drop visibility is worth the migration cost. Debugging NetworkPolicies without Hubble is guesswork.
Label your namespaces explicitly with kubernetes.io/metadata.name — this label is auto-applied in Kubernetes 1.21+ and is required for reliable namespace-based NetworkPolicy selectors.
Monitor CoreDNS with Prometheus and alert on coredns_dns_response_rcode_count_total{rcode="SERVFAIL"} — a spike indicates upstream DNS failure that will cause cascading service discovery failures across the entire cluster.
Never run tcpdump directly on a node in production without approval — packet capture on a financial services cluster is a compliance event that must be logged and justified.

COMMON MISTAKE / WARNING
**Common Mistake:** Checking pod logs to diagnose networking issues. Application logs say "connection refused" or "timeout" — they cannot tell you whether the failure is at Layer 2 (CNI), Layer 3 (kube-proxy), Layer 4 (DNS), or Layer 5 (NetworkPolicy). Always use network-level tools like netshoot, not application logs, for network debugging.

Quick Reference & Troubleshooting Commands

Command	Purpose
`kubectl run netshoot --image=nicolaka/netshoot -it --rm -n <ns> -- bash`	Launch network debug container
`kubectl debug -it --image=nicolaka/netshoot --target=<pod> <pod> -n <ns> -- bash`	Debug sharing pod's network namespace
`kubectl get endpoints <service> -n <ns>`	Verify pods are registered as Service backends
`kubectl get networkpolicies -n <ns>`	List all NetworkPolicies in a namespace
`kubectl get pods -n kube-system \| grep coredns`	Check CoreDNS pod health
`kubectl logs -n kube-system -l k8s-app=kube-dns`	CoreDNS logs for DNS failure diagnosis
`nslookup <service>.<ns>.svc.cluster.local`	Test DNS resolution from inside a pod
`curl -v http://<clusterip>:<port>/path`	Test Service IP directly bypassing DNS
`kubectl get pods -n production -o wide`	Show pod IPs and which node they are on
`kubectl logs -n kube-system kube-proxy-<id>`	kube-proxy logs for Service routing issues

Asset Tracker Update:

Debugging Kubernetes Networking with kubectl and CNI Plugins

Overview and What You Will Learn

Why This Matters in Production

Core Principles

Detailed Step-by-Step Practical Lab

Step 1 — Deploy the netshoot Debug Container

Step 2 — Debug Layer 1: Verify Pod Has a Network Interface and IP

Step 3 — Debug Layer 2: Test Pod-to-Pod Connectivity Across Nodes

Step 4 — Debug Layer 3: Verify kube-proxy Service Rules

Step 5 — Debug Layer 4: CoreDNS and DNS Resolution Failures

Step 6 — Debug Layer 5: NetworkPolicy Blocking Traffic

Step 7 — Run a Full Connectivity Matrix Test

Production Best Practices & Common Pitfalls

Quick Reference & Troubleshooting Commands

Resources

Explore More in Kubernetes Networking and Traffic Management

Configuring Ingress Controllers with NGINX for Production Traffic

Kubernetes Network Policies for Pod-Level Traffic Control

Implementing Kubernetes Deployment Strategies - Rolling, Blue-Green, and Canary

Configuring Kubernetes Resource Requests, Limits, and QoS Classes

Debugging Kubernetes Networking with kubectl and CNI Plugins

Overview and What You Will Learn

Why This Matters in Production

Core Principles

Detailed Step-by-Step Practical Lab

Step 1 — Deploy the netshoot Debug Container

Step 2 — Debug Layer 1: Verify Pod Has a Network Interface and IP

Step 3 — Debug Layer 2: Test Pod-to-Pod Connectivity Across Nodes

Step 4 — Debug Layer 3: Verify kube-proxy Service Rules

Step 5 — Debug Layer 4: CoreDNS and DNS Resolution Failures

Step 6 — Debug Layer 5: NetworkPolicy Blocking Traffic

Step 7 — Run a Full Connectivity Matrix Test

Production Best Practices & Common Pitfalls

Quick Reference & Troubleshooting Commands

Related Content to Explore

Foundations

Version Control (Git) for devops

Build & Package Management

Resources

Explore More in Kubernetes Networking and Traffic Management

Configuring Ingress Controllers with NGINX for Production Traffic

Kubernetes Network Policies for Pod-Level Traffic Control

Implementing Kubernetes Deployment Strategies - Rolling, Blue-Green, and Canary

Configuring Kubernetes Resource Requests, Limits, and QoS Classes