What is Toleration? | DevOps Dictionary

Toleration — Extended Technical Detail

What is a Toleration in Simple Terms?

A Toleration is the pod's way of saying "I am allowed on that restricted node." It is the matching key to the Taint lock on the node. Without a Toleration, a pod is automatically repelled by any node that carries a matching Taint.

◈ DIAGRAM

+------------------------------------------+
| Node: gpu-node-1                         |
| Taint: workload=gpu:NoSchedule           | <- Node repels all pods by default
+------------------------------------------+
 
+------------------------------------------+
| Regular API Pod                          |
| No Toleration                            | <- REPELLED — stays on untainted nodes
+------------------------------------------+
 
+------------------------------------------+
| GPU Transcoding Pod                      |
| Toleration: workload=gpu:NoSchedule      | <- ALLOWED — can land on tainted node
+------------------------------------------+

Toleration in a Pod Spec

YAML

# Pod spec for a GPU encoding job that tolerates the GPU node taint
apiVersion: v1
kind: Pod
metadata:
  name: video-encoder
  namespace: streaming-prod
spec:
  tolerations:
    - key: "workload"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"   # Must match the taint effect exactly
  containers:
    - name: encoder
      image: registry.hotstar.in/video-encoder:v3.1.0
      resources:
        limits:
          nvidia.com/gpu: 1

Toleration Operators — Equal vs Exists

◈ DIAGRAM

+------------------------+          +------------------------------+
| operator: Equal        |          | operator: Exists             |
|                        |          |                              |
| key=workload           | <------> | key=workload                 |
| value=gpu              |          | (value is ignored)           |
| effect=NoSchedule      |          | effect=NoSchedule            |
|                        |          |                              |
| Matches ONLY taints    |          | Matches ANY taint with       |
| where key=workload     |          | key=workload regardless      |
| AND value=gpu          |          | of its value                 |
+------------------------+          +------------------------------+

Bash

# Equal — key AND value must both match the taint exactly
tolerations:
  - key: "workload"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
 
# Exists — only the key needs to match, value is ignored
tolerations:
  - key: "workload"
    operator: "Exists"
    effect: "NoSchedule"
 
# Universal — tolerates ALL taints on ALL nodes (empty key + Exists)
tolerations:
  - operator: "Exists"   # No key — matches every taint in the cluster

Toleration with tolerationSeconds (for NoExecute)

When a node gets a NoExecute taint (e.g., during a node failure), Kubernetes evicts pods that don't tolerate it. You can delay eviction using tolerationSeconds:

YAML

spec:
  tolerations:
    - key: "node.kubernetes.io/not-ready"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 300   # Stay on the node for 5 minutes before eviction
                               # Useful for Zerodha's trading pods during brief node blips

◈ DIAGRAM

+------------------------------------------+
| Node becomes NotReady                    | <- Hardware hiccup on mumbai-worker-3
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| NoExecute taint auto-applied             | <- Kubernetes adds system taint
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Pod with tolerationSeconds: 300          | <- Waits 5 min before evicting
| Pod without tolerationSeconds            | <- Evicted immediately
+------------------------------------------+

Toleration + NodeAffinity — The Complete Pattern

A Toleration only allows a pod onto a tainted node — it does not guarantee the pod lands there. To ensure the pod goes exclusively to the intended node, combine Toleration with NodeAffinity:

YAML

# deployment.yaml — video transcoder exclusively on GPU nodes at Hotstar
spec:
  tolerations:
    - key: "workload"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"       # Step 1: Allow past the taint repel
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: workload
                operator: In
                values: ["gpu"]  # Step 2: Pin to GPU nodes only
  containers:
    - name: transcoder
      image: registry.hotstar.in/transcoder:v4.2.1
      resources:
        limits:
          nvidia.com/gpu: "1"

◈ DIAGRAM

+------------------------------------------+
| Toleration on pod                        | <- Unlocks the tainted node
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| NodeAffinity on pod                      | <- Forces pod to GPU nodes only
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| GPU node exclusively runs GPU workloads  | <- Isolation achieved
+------------------------------------------+

System Tolerations on DaemonSets

Critical system DaemonSets (kube-proxy, CNI, log collectors) automatically carry built-in Tolerations so they run on every node including tainted ones:

Bash

# Check the built-in tolerations on a system DaemonSet
kubectl get daemonset kube-proxy -n kube-system -o yaml | grep -A 20 tolerations
 
# Output — kube-proxy tolerates everything
# tolerations:
# - operator: Exists    <-- matches ALL taints, runs everywhere

Troubleshooting Common Toleration Problems

Problem	Symptom	Fix
Pod stuck in Pending	`0/5 nodes available: node(s) had taint`	Pod is missing a Toleration — add `tolerations` block with matching key, value, and effect
Toleration set but pod still Pending	Pod has Toleration but won't schedule	Toleration `effect` doesn't match taint `effect` — they must be identical
Pod not landing on intended node	Pod schedules on random untainted nodes	Toleration allows the node but doesn't pin the pod — add `nodeAffinity` or `nodeSelector`
Non-GPU pods landing on GPU nodes	Cost spike on GPU instance billing	Pod has `operator: Exists` with no key — wildcard Toleration bypasses all taints
Pod evicted unexpectedly after node event	Running pod disappears during node hiccup	Missing `tolerationSeconds` on `NoExecute` Toleration — add a grace period

PLACEMENT PRO TIP
**Tip:** Use `operator: Exists` with an empty key to create a pod that tolerates ALL taints on ALL nodes — useful for critical system DaemonSets like log collectors (Fluentd, Filebeat) that must run on every node including tainted GPU and dedicated nodes.

REMEMBER THIS
**Remember:** A Toleration is permission, not a directive. It tells Kubernetes "this pod is allowed on tainted nodes" — but without a `nodeAffinity` or `nodeSelector`, the scheduler may still place the pod on a cheaper untainted node. Always pair Toleration with affinity rules for hard node dedication.

COMMON MISTAKE / WARNING
**Security:** At Razorpay, PCI-DSS compliance requires that card-processing pods run only on dedicated, isolated nodes. Set `operator: Equal` with the exact taint key and value — never `operator: Exists` — to prevent these pods from accidentally tolerating unintended taints and landing on shared infrastructure.

COMMON MISTAKE / WARNING
**Common Mistake:** Using `operator: Exists` without a `key` in a Toleration. This is a wildcard that matches every taint on every node in the cluster — effectively defeating all taint-based isolation for that pod. Always scope Tolerations to a specific `key` unless you explicitly need universal placement (system DaemonSets only).