What is Taint? | DevOps Dictionary

Taint — Extended Technical Detail

What is a Taint in Simple Terms?

A taint is like putting a "No Entry" sign on a node. By default, no pod will be scheduled on a tainted node. Only pods that explicitly say "I am allowed to go here" (via a Toleration) will be scheduled there.

◈ DIAGRAM

+------------------------------------------+
| Without Taint                            |
|                                          |
| gpu-node-1  <- any pod can land here    | <- Regular API pods, batch jobs,
| gpu-node-2  <- any pod can land here    |    and GPU workloads all compete
|                                          |    for expensive GPU nodes
+------------------------------------------+
 
+------------------------------------------+
| With Taint: workload=gpu:NoSchedule      |
|                                          |
| gpu-node-1  [TAINTED]                   | <- Only pods with matching
| gpu-node-2  [TAINTED]                   |    Toleration can land here.
|                                          |    API pods are repelled.
+------------------------------------------+

Real Use Case

At Hotstar, GPU nodes for video transcoding are expensive (10x the cost of regular nodes). Tainting them ensures only transcoding pods run there — not regular API pods accidentally consuming GPU capacity during a traffic spike.

Taint Effects — The Three Modes

◈ DIAGRAM

+------------------------+    +------------------------+    +------------------------+
| NoSchedule             |    | PreferNoSchedule       |    | NoExecute              |
|                        |    |                        |    |                        |
| New pods without       |    | Kubernetes will TRY    |    | New pods repelled AND  |
| Toleration will NOT    |    | to avoid this node     |    | existing pods without  |
| be scheduled here      |    | but may use it if no   |    | Toleration are EVICTED |
|                        |    | other node is free     |    |                        |
| Existing pods: safe    |    | Existing pods: safe    |    | Use for: maintenance   |
| Use for: isolation     |    | Use for: soft hints    |    | and node draining      |
+------------------------+    +------------------------+    +------------------------+

How to Add a Taint to a Node

Bash

# Taint a node — only GPU workloads allowed
kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule
# Taint format: key=value:effect
 
# Taint for node maintenance — evicts non-tolerating pods immediately
kubectl taint nodes mumbai-worker-3 maintenance=patching:NoExecute
 
# Taint without a value (key-only taint)
kubectl taint nodes gpu-node-1 dedicated:NoSchedule
 
# Verify the taint was applied
kubectl describe node gpu-node-1 | grep -A 5 Taints
# Taints: workload=gpu:NoSchedule

How to Remove a Taint

Bash

# Add a minus sign at the end to remove — exact key=value:effect match required
kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule-
 
# Remove a key-only taint
kubectl taint nodes gpu-node-1 dedicated:NoSchedule-
 
# Verify taint is gone
kubectl describe node gpu-node-1 | grep Taints
# Taints: <none>

The Matching Toleration in Pod Spec

A taint does nothing without a matching Toleration in the pod spec:

YAML

# deployment.yaml — GPU transcoding pod that tolerates the taint
apiVersion: apps/v1
kind: Deployment
metadata:
  name: video-transcoder
  namespace: streaming-prod
spec:
  template:
    spec:
      tolerations:
        - key: "workload"
          operator: "Equal"
          value: "gpu"
          effect: "NoSchedule"    # Must match the taint effect exactly
      nodeSelector:
        workload: gpu             # Also add nodeSelector to PREFER GPU nodes
      containers:
        - name: transcoder
          image: registry.hotstar.in/transcoder:v4.2.1
          resources:
            limits:
              nvidia.com/gpu: "1"  # Request 1 GPU

How to View Taints on All Nodes

Bash

# Custom column output — quick overview
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
 
# Full taint details including key, value, effect
kubectl get nodes -o json | \
  jq '.items[] | {name: .metadata.name, taints: .spec.taints}'
 
# Check a specific node
kubectl describe node mumbai-worker-3 | grep -A 10 Taints

Taint + Toleration + NodeAffinity — The Complete Pattern

For hard node dedication (only GPU pods on GPU nodes, and GPU pods ONLY on GPU nodes), combine all three:

YAML

spec:
  tolerations:
    - key: "workload"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: workload
                operator: In
                values: ["gpu"]

◈ DIAGRAM

+------------------------------------------+
| Taint on node                            | <- Repels non-GPU pods
+------------------------------------------+
| Toleration on pod                        | <- Allows GPU pod past the repel
+------------------------------------------+
| NodeAffinity on pod                      | <- Ensures GPU pod goes ONLY to
|                                          |    GPU nodes (not just tolerates)
+------------------------------------------+

Troubleshooting Common Taint Problems

Problem	Symptom	Fix
Pod stuck in `Pending` with taint	`0/5 nodes are available: 5 node(s) had taint`	Pod missing Toleration — add matching `tolerations` block to pod spec
Toleration set but pod still Pending	Pod has Toleration but won't schedule	Toleration `effect` doesn't match taint `effect` — they must be identical
Non-GPU pods landing on GPU nodes	Cost spike on GPU instance billing	Taint is set but pods have a catch-all Toleration (`operator: Exists`) — scope the Toleration to a specific key and value
NoExecute evicting critical pods	Running pods evicted unexpectedly	Wrong node tainted — verify `kubectl get nodes` names before applying `NoExecute`
Taint not removed after maintenance	Node still repelling pods after patch	Forgot the trailing `-` in `kubectl taint` remove command — re-run with `-` suffix

REMEMBER THIS
**Remember:** Taints and Tolerations work together. A taint on a node does nothing unless the pod also has the matching Toleration. Forgetting the Toleration means the pod stays in `Pending` forever — and `kubectl describe pod` will show `node(s) had taint` in the Events section.

PLACEMENT PRO TIP
**Tip:** Use `NoExecute` taint effect during node maintenance. It automatically evicts all non-tolerating pods, so you can safely patch the node without running `kubectl drain` manually. The system daemonsets (kube-proxy, CNI) always have built-in Tolerations for system taints and will not be evicted.

COMMON MISTAKE / WARNING
**Security:** On PCI-DSS workloads like Razorpay's card processing pods, use `NoSchedule` taints on dedicated nodes to ensure no other workload can co-reside on those nodes. Co-tenancy on the same node means potential side-channel attacks via shared CPU caches — taint-based isolation is a critical compliance control.

COMMON MISTAKE / WARNING
**Common Mistake:** Using `operator: Exists` without a `key` in a Toleration. This creates a wildcard Toleration that matches ALL taints on ALL nodes — effectively defeating every taint in the cluster for that pod. Always scope Tolerations to a specific `key`.