What is DaemonSet? | DevOps Dictionary

DaemonSet — One Agent Per Node, Always

What is a DaemonSet in Simple Terms?

A DaemonSet is a standing order to the scheduler: "This pod must always run on every node — one copy per node, no more, no less." Not one total replica. One per node. When a new node joins the cluster, the DaemonSet pod is automatically placed on it. When a node is removed, the pod is cleaned up automatically.

DaemonSet vs Deployment — The Key Difference

◈ DIAGRAM

+------------------------------------------+     +------------------------------------------+
|             Deployment                   |     |              DaemonSet                   |
|                                          |     |                                          |
|  You control replica count               |     |  Cluster controls replica count          |
|  replicas: 3 (on any 3 nodes)            |     |  1 pod per node (on ALL nodes)           |
|                                          |     |                                          |
|  Use for: application workloads          |     |  Use for: infrastructure agents          |
+------------------------------------------+     +------------------------------------------+

One Pod Per Node — How It Looks

◈ DIAGRAM

+---------------+  +---------------+  +---------------+  +---------------+
|  mumbai-node-1|  |  mumbai-node-2|  |  mumbai-node-3|  |  mumbai-node-4|
|               |  |               |  |               |  |               |
|  [fluentd-0]  |  |  [fluentd-1]  |  |  [fluentd-2]  |  |  [fluentd-3]  |
+---------------+  +---------------+  +---------------+  +---------------+
        ^                                                          ^
        |                                                          |
        New node added to cluster -----> DaemonSet auto-schedules pod here

When to Use a DaemonSet

Use DaemonSet for infrastructure agents:

Log collection — Fluentd, Filebeat, Promtail (must read log files from every node's disk)
Metrics scraping — Node Exporter (must collect CPU/memory/disk from every node)
Network plugins — Calico, Cilium CNI agents (must configure networking on every node)
Security agents — Falco, CrowdStrike (must inspect every node's syscalls and processes)
Storage drivers — CSI node drivers that attach and mount volumes

Do NOT use DaemonSet for:

Application workloads (use Deployment)
Batch processing (use Jobs or CronJobs)
Anything where you want explicit control over replica count

A Real DaemonSet — Node Exporter for Prometheus

YAML

# node-exporter-daemonset.yaml
# Runs Prometheus Node Exporter on every node in mumbai-prod-cluster
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true              # Uses the node's network namespace directly
      hostPID: true                  # Sees all processes on the node (required for metrics)
      tolerations:
        - operator: Exists           # Tolerate ALL taints — run on control-plane nodes too
          effect: NoSchedule
      containers:
        - name: node-exporter
          image: prom/node-exporter:v1.7.0
          ports:
            - containerPort: 9100
              hostPort: 9100         # Binds directly to the node's port 9100
          args:
            - '--path.procfs=/host/proc'
            - '--path.sysfs=/host/sys'
          volumeMounts:
            - name: proc
              mountPath: /host/proc
              readOnly: true
            - name: sys
              mountPath: /host/sys
              readOnly: true
      volumes:
        - name: proc
          hostPath:
            path: /proc              # Mounts the node's /proc filesystem
        - name: sys
          hostPath:
            path: /sys               # Mounts the node's /sys filesystem

Targeting Specific Nodes — Not Always Every Node

You do not always want a DaemonSet on every node. A GPU monitoring agent should only run on GPU nodes:

YAML

# nodeSelector — simple label match
spec:
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-gpu      # Only schedule on nodes labelled as GPU nodes

For more complex targeting, use nodeAffinity:

YAML

# nodeAffinity — skip control-plane, run only on worker nodes
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: node-role
              operator: In
              values:
                - worker             # Excludes control-plane nodes explicitly

Tolerations — Getting onto Tainted Nodes

Control-plane nodes carry a default taint: node-role.kubernetes.io/control-plane:NoSchedule. Without a matching toleration, your DaemonSet will skip them. The operator: Exists toleration bypasses ALL taints — use it for critical agents that must run everywhere like Falco or CNI plugins.

YAML

tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule              # Specific — only bypass this one taint

Key DaemonSet Commands

Task	Command
List DaemonSet status	`kubectl get ds -n monitoring`
See which nodes have the pod	`kubectl get pods -o wide -n monitoring`
Check rollout status	`kubectl rollout status ds/node-exporter -n monitoring`
Force restart all pods	`kubectl rollout restart ds/node-exporter -n monitoring`
Describe DaemonSet events	`kubectl describe ds/node-exporter -n monitoring`
Check pod count vs node count	`kubectl get ds node-exporter -n monitoring`

Bash

# Verify a DaemonSet pod is running on every node
kubectl get pods -n monitoring -l app=node-exporter -o wide
 
# Output should show one pod per node:
# NAME                  READY   NODE
# node-exporter-4xk2p   1/1     mumbai-node-1
# node-exporter-7hq9r   1/1     mumbai-node-2
# node-exporter-m2pzn   1/1     mumbai-node-3

COMMON MISTAKE / WARNING
**Security:** Setting `hostNetwork: true` and `hostPID: true` gives the container full visibility into the host's network stack and every process running on the node. Only use these flags for trusted infrastructure agents like Node Exporter or Falco — never for application workloads. In Hotstar or PhonePe production clusters, DaemonSet pods with host access should be reviewed as part of every security audit.

REMEMBER THIS
**Remember:** DaemonSet pod count equals node count. If your cluster has 12 nodes and your DaemonSet shows 10 pods, two nodes have issues — either they are tainted without a matching toleration, or the pods are failing on those nodes. Use `kubectl get pods -o wide` to identify which nodes are missing coverage.

COMMON MISTAKE / WARNING
**Common Mistake:** Using a Deployment with `replicas` matching your node count as a substitute for a DaemonSet. If a node is added later, the Deployment will not automatically place a pod on it — you are back to manual scaling. Always use a DaemonSet for anything that must run on every node.

PLACEMENT PRO TIP
**Tip:** In Zerodha or Swiggy-scale clusters, DaemonSets for log collection (Fluentd/Promtail) are often the highest-volume pods in the cluster. Set proper `resources.requests` and `resources.limits` on them — an unthrottled Fluentd pod can consume enough CPU to starve application pods on the same node during log bursts.