From console.log to Kibana: How Your Pod's Logs Actually Reach Elasticsearch

Almost every backend engineer has heard of the ELK stack. Elasticsearch stores the logs, Kibana lets you search them, and something in the middle ships them.

But here’s a question that trips up most people: your code runs inside a container, inside a pod, on a node somewhere in a cluster you’ll never SSH into. So how does a single console.log("payment failed") line actually make it all the way to Elasticsearch?

There’s no magic. It’s a clean, four-hop pipeline:

Your app (stdout/stderr)
   ↓
Container runtime writes logs to a file on the node
   ↓
Fluentd (a DaemonSet running on every node) tails that file
   ↓
Fluentd ships the log to Elasticsearch  →  Kibana reads it

In my DevOps Essentials post I promised I’d cover the DaemonSet-based logging pattern in detail. This is that post. Let’s walk each hop.

---

Hop 1: Your app just writes to stdout — nothing else

The most important rule in containerized logging is also the most counter-intuitive:

Your application should not manage log files. It should only write to stdout and stderr.

No app.log. No log rotation logic. No shipping code inside your service. Just print.

// This is all your app needs to do.
console.log(JSON.stringify({ level: "info", msg: "payment processed", orderId: 123 }));
console.error(JSON.stringify({ level: "error", msg: "gateway timeout", orderId: 456 }));

This is one of the 12-Factor App principles: treat logs as event streams. Your app is a producer that shouts events into the void. It has no idea where they end up — and that’s the point. The routing, storage, and searching are somebody else’s job.

Why does this matter? Because it decouples your app from your logging infrastructure. You can swap Elasticsearch for Loki, or Fluentd for Fluent Bit, and not touch a single line of application code.

Tip: log as structured JSON, not plain strings. {"level":"error","orderId":456} is trivially filterable in Kibana. "error processing order 456" forces you into fragile regex later.

---

Hop 2: The container runtime captures stdout into a file

So your app printed to stdout. Where does that stream go?

When a container writes to stdout/stderr, the container runtime (Docker, containerd, CRI-O) captures those streams and writes them to a log file on the node’s filesystem. With Docker’s default json-file logging driver, each line becomes a JSON object:

{"log":"{\"level\":\"error\",\"msg\":\"gateway timeout\"}\n","stream":"stderr","time":"2026-07-18T10:22:01.5Z"}

Notice the runtime wraps your log line inside its own envelope — adding which stream it came from (stdout vs stderr) and a time stamp.

On a Kubernetes node, these files live in predictable places:

/var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log
/var/log/containers/<pod>_<namespace>_<container>-<id>.log   # symlinks into the above

That /var/log/containers/ directory is the golden path — every container on that node dumps its logs there, and the filename itself encodes the pod, namespace, and container name. Hold that thought; it becomes important in the next hop.

This is also, by the way, exactly what kubectl logs <pod> reads. That command isn’t doing anything fancy — it’s just showing you the contents of these node-local files. Which is why kubectl logs loses history when a pod is rescheduled or the node’s logs rotate. Node-local files are ephemeral. That’s the whole reason we need to ship them somewhere durable.

---

Hop 3: Fluentd — one collector per node, running as a DaemonSet

Now the real question: who reads those /var/log/containers/*.log files and forwards them?

A log collector. The two you’ll hear about are Fluentd and its lighter cousin Fluent Bit. Their job is to tail the log files, parse them, enrich them, and push them to a destination.

But how do you make sure the collector can see the log files of every container on every node? You can’t run it as a normal Deployment — a Deployment might place 3 replicas on a 10-node cluster, and 7 nodes’ worth of logs would go uncollected.

This is the exact problem a DaemonSet solves.

A DaemonSet guarantees that one copy of a pod runs on every node in the cluster. Add a node, Kubernetes automatically schedules the collector onto it. Remove a node, the collector goes with it.

That’s precisely the guarantee logging needs. One Fluentd pod per node, each responsible only for the logs of containers on its own node.

        Node A                    Node B                    Node C
 ┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
 │  app pods...      │     │  app pods...      │     │  app pods...      │
 │  /var/log/        │     │  /var/log/        │     │  /var/log/        │
 │  containers/*.log │     │  containers/*.log │     │  containers/*.log │
 │        ▲          │     │        ▲          │     │        ▲          │
 │   ┌────┴─────┐    │     │   ┌────┴─────┐    │     │   ┌────┴─────┐    │
 │   │ fluentd  │    │     │   │ fluentd  │    │     │   │ fluentd  │    │
 │   │(DaemonSet│    │     │   │(DaemonSet│    │     │   │(DaemonSet│    │
 │   │   pod)   │    │     │   │   pod)   │    │     │   │   pod)   │    │
 │   └────┬─────┘    │     │   └────┬─────┘    │     │   └────┬─────┘    │
 └────────┼──────────┘     └────────┼──────────┘     └────────┼──────────┘
          └──────────────────────────┼──────────────────────────┘
                                      ▼
                              Elasticsearch

How does a Fluentd pod read files that belong to the node and not to itself? Through a hostPath volume — the node’s /var/log directory is mounted straight into the Fluentd container:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    matchLabels: { app: fluentd }
  template:
    metadata:
      labels: { app: fluentd }
    spec:
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1-elasticsearch
          env:
            - name: FLUENT_ELASTICSEARCH_HOST
              value: "elasticsearch.logging.svc.cluster.local"
            - name: FLUENT_ELASTICSEARCH_PORT
              value: "9200"
          volumeMounts:
            - name: varlog
              mountPath: /var/log            # the node's log dir, read-only
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

Once it can see the files, Fluentd does three things:

1. Tail — it follows every /var/log/containers/*.log file, picking up new lines as they’re written (and remembering its position so a restart doesn’t re-send everything).
2. Enrich — remember the filename encodes pod/namespace/container? Fluentd parses that and attaches Kubernetes metadata — pod_name, namespace, labels, node_name — so in Kibana you can filter by namespace: payments or pod: checkout-7f9c.
3. Buffer & forward — it batches records into a buffer and ships them onward. If Elasticsearch is slow or down, the buffer holds the logs and retries instead of dropping them.

A trimmed-down Fluentd config for that flow:

# 1. TAIL — read every container log file on this node
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kube.*
  <parse>
    @type json
  </parse>
</source>

# 2. ENRICH — add pod/namespace/labels from the K8s API
<filter kube.**>
  @type kubernetes_metadata
</filter>

# 3. FORWARD — batch and ship to Elasticsearch, buffering on failure
<match kube.**>
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
  logstash_format true          # daily indices: logstash-2026.07.18
  <buffer>
    flush_interval 5s
    retry_max_times 10
  </buffer>
</match>

---

Hop 4: Elasticsearch stores it, Kibana shows it

Fluentd pushes each enriched record to Elasticsearch over HTTP. Elasticsearch indexes it — usually into a daily index like logstash-2026.07.18 (that logstash_format true line above). Daily indices make retention trivial: to delete last month’s logs, you just drop those indices.

Then Kibana connects to Elasticsearch and gives you the search UI. Now that console.log("payment failed") from Hop 1 is:

• searchable by full-text (msg: "payment failed")
• filterable by the metadata Fluentd added (namespace: payments AND level: error)
• correlatable across every pod and node in the cluster, in one place

The full journey, end to end:

console.log(...)                        ← your code, Hop 1
   ↓ stdout
/var/log/containers/checkout-*.log      ← container runtime, Hop 2
   ↓ tailed by
fluentd DaemonSet pod on that node      ← Hop 3
   ↓ enrich + buffer + HTTP POST
Elasticsearch  (index: logstash-2026.07.18)
   ↓
Kibana  ← you, searching at 2am during an incident

---

The “L” in ELK isn’t always Logstash

One naming confusion worth clearing up: ELK stands for Elasticsearch, Logstash, Kibana. But in Kubernetes you’ll rarely see Logstash on the nodes. Logstash is heavy (it runs on the JVM) and was designed as a central processing pipeline, not a per-node agent.

So the modern per-node collector is almost always Fluentd or Fluent Bit:

	Fluentd	Fluent Bit
Written in	Ruby (+ C core)	Pure C
Memory footprint	~40 MB+	~1 MB
Plugins	Huge ecosystem	Smaller, growing
Typical role	Aggregator / node agent	Lightweight node agent

A very common production setup is Fluent Bit as the tiny per-node DaemonSet forwarding to a central Fluentd aggregator that does the heavy parsing before Elasticsearch. Same pipeline shape — just split across two tiers. (When people say “EFK stack”, the F is exactly this Fluentd/Fluent Bit substitution.)

---

Gotchas I’ve hit (so you don’t have to)

• Don’t write logs to a file inside the container. If your app writes to /app/logs/app.log, the collector — which only watches stdout — never sees it, and the file dies with the pod. Print to stdout.
• Multiline stack traces get split. Each line of a Java/Node stack trace is a separate stdout write, so the collector treats each as a separate log record. You need a multiline parser to stitch them back into one event.
• Log rotation is real. The kubelet rotates container log files (default ~10 MB). A good collector follows the rotation via its position file; a naive one can double-send or miss lines around the rotation boundary.
• Backpressure will bite you. If Elasticsearch can’t keep up, the collector’s buffer fills. Size your buffers and set retry limits — otherwise a slow Elasticsearch can OOM your logging pods or silently drop logs.
• kubectl logs is not your log store. It reads node-local files that vanish on reschedule. It’s for quick debugging, never for history or auditing. That’s Elasticsearch’s job.

---

Wrapping up

The ELK stack feels like a black box until you trace the one path that matters:

Your app prints to stdout → the container runtime writes it to a file on the node → a Fluentd DaemonSet (one per node) tails that file, enriches it with pod metadata, and ships it → Elasticsearch stores it → Kibana searches it.

The elegant part is the separation of concerns. Your application does the dumbest possible thing — print. The DaemonSet guarantees a collector on every node with zero per-app wiring. And swapping any piece of the pipeline never touches your code.

That’s the whole trick. No magic — just stdout, a file, and a collector that’s everywhere.