The data itself is basically static for that moment. It is the same post, the same numbers, the same content for everyone. So the interesting question is, do we really need to hit the database once for every single one of these requests? Clearly not. And once you start pulling on that thread, it leads to a nice little chain of optimizations that ends in a pub/sub trick. Let me walk through the whole thinking.

Step 1: put Redis in front, the obvious win

The first thing anyone does is add a cache. Put Redis in front of the database and use the classic cache aside pattern.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
async function getPost(id) {
  const key = `post:${id}`;

  // 1. try the cache
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  // 2. miss, go to the database
  const post = await db.query("SELECT * FROM posts WHERE id = ?", [id]);

  // 3. put it in the cache for next time
  await redis.set(key, JSON.stringify(post), "EX", 60);

  return post;
}

This already solves most of the problem. The first request fills the cache, and after that everyone reads from Redis. And Redis is comfortable with this kind of load, a single node handles well over a hundred thousand ops per second, and with Redis Cluster you are into the millions of reads per second range without breaking a sweat. So for pure reads of hot data, Redis absorbs the storm and the database barely notices.

So far so good. If this was the whole story, there would be no blog post. The problem is hiding in one specific moment.

Step 2: the moment it all breaks (cache stampede)

Look closely at what happens when the key is not in the cache. There are two very normal times this is true:

• The very first time anyone asks for this post (cold cache).
• The instant the key expires. Our EX 60 means every 60 seconds the key vanishes for a moment.

Now replay the viral scenario at exactly that instant. A million concurrent requests come in. They all run redis.get, they all miss (because the key is not there yet), and so they all fall through to the database at the same time. Then they all try to write the result back into Redis.

1
2
3
4
5
6
7
   million concurrent requests, key just expired
   │  │  │  │  │  │  │  │  │  │  │  │  │  │  │
   ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼  ▼
        all miss the cache at the same time
                     │
                     ▼
        all hammer the DATABASE at once   ← the stampede

This is called a cache stampede (also thundering herd, or dogpile). The cache was supposed to protect the database, but at the exact moment of a miss it protects nothing, and the database gets hit by thousands of identical, and often expensive, queries all at once. If that query is heavy (a big join, an aggregation), your database can fall over from a load it was never actually required to do, because remember, the answer is the same for everyone. We only needed to run it once.

And it is worse than just reads. In many designs that “populate the cache” step is not a plain set, it is a write into the database too, or an insert of a computed/derived row. Writes and inserts are much heavier than reads for a database, and now you have thousands of concurrent inserts for the same data racing each other. That is a lot of wasted, duplicated, expensive work.

So the real problem statement is, how do we make sure that when many requests want the same missing data, only one of them actually does the work, and everyone else just waits for that one result?

Step 3: single flight, only one request does the work

The idea has a nice name, single flight. For a given key, only allow one in flight computation at a time. Everyone else who wants the same key while it is being computed should not start their own, they should attach to the one already running.

Within a single Node.js process, you get this almost for free with a map of in flight promises:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
const inFlight = new Map();

async function getPostCoalesced(id) {
  const key = `post:${id}`;

  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  // is someone already fetching this key right now?
  if (inFlight.has(key)) {
    return inFlight.get(key); // attach to the existing work, do not start your own
  }

  const promise = (async () => {
    const post = await db.query("SELECT * FROM posts WHERE id = ?", [id]);
    await redis.set(key, JSON.stringify(post), "EX", 60);
    return post;
  })();

  inFlight.set(key, promise);
  try {
    return await promise;
  } finally {
    inFlight.delete(key); // clear it once done, so the next miss can refetch
  }
}

Now if a thousand requests hit the same process during that miss window, the first one starts the database query and the other 999 simply await the same promise. One database call, one cache write, a thousand happy responses.

This is exactly what Go’s singleflight package does, and it is a genuinely underused pattern.

Step 4: but we have many pods, not one process

Here is the catch. In real life your app is not one process, it is 40 pods behind a load balancer. That in memory Map only coalesces requests inside one pod. Across 40 pods you still get up to 40 concurrent database calls at the miss instant, one per pod. Much better than a million, but still not the “exactly once” we want, and if the query is heavy even 40 at once can hurt.

To coordinate across all pods, we need a shared point of truth, and we already have one sitting right there, Redis.

The trick is a lock. The first request to arrive grabs a short lock in Redis, and only the holder of that lock is allowed to go to the database. SET key value NX EX ttl is perfect for this, because NX means “only set if it does not already exist”, so exactly one request across the whole fleet wins it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
async function getPostGlobal(id) {
  const key = `post:${id}`;
  const lockKey = `lock:${key}`;

  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  // try to become THE one who does the work (NX = only if not set)
  const gotLock = await redis.set(lockKey, "1", "NX", "EX", 10);

  if (gotLock) {
    // I am the chosen one. Do the heavy work exactly once.
    const post = await db.query("SELECT * FROM posts WHERE id = ?", [id]);
    await redis.set(key, JSON.stringify(post), "EX", 60);
    await redis.del(lockKey);
    return post;
  }

  // someone else is already doing it. I just have to wait for the result.
  return waitForResult(key);
}

Now exactly one request in the whole cluster touches the database. Everyone else falls into waitForResult. The only question left is, how do the waiters get the answer?

Step 5: polling vs pub/sub for the waiters

The simplest way to wait is to poll. Every waiting request checks Redis every so often to see if the value has appeared yet.

1
2
3
4
5
6
7
8
async function waitForResult(key) {
  for (let i = 0; i < 50; i++) {
    const cached = await redis.get(key);
    if (cached) return JSON.parse(cached);
    await sleep(50); // check again in 50ms
  }
  // fallback: still nothing, do the work ourselves so we never hang forever
}

You can even make this richer by storing a small status value, so waiters can see the stage the work is in (“pending”, “fetching”, “ready”) instead of guessing. That was actually my first instinct too, push the request state into Redis and let everyone poll it.

Polling works, but it has two annoyances. You are still making a Redis call on every poll from every waiting request, so a million waiters polling every 50ms is its own little load. And there is wasted latency, if the work finishes 1ms after your last check, you still sit idle until your next 50ms tick.

This is where pub/sub is much nicer. Instead of everyone repeatedly asking “is it ready yet?”, the waiters subscribe to a channel for that key and go quiet. When the one worker finishes, it publishes a message on that channel. Every waiting request wakes up at that exact moment and reads the now filled value.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// the worker, after it fills the cache:
await redis.set(key, JSON.stringify(post), "EX", 60);
await redis.publish(`ready:${key}`, "done"); // wake everyone up
await redis.del(lockKey);

// a waiter:
async function waitForResult(key) {
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached); // maybe it is already ready

  await subscribeOnce(`ready:${key}`); // sleep until the worker signals
  const value = await redis.get(key);
  return JSON.parse(value);
}

1
2
3
4
5
6
                    ┌─────────────────────┐
   worker (1 req) ─►│  DB query + cache    │─► publish "ready:post:42"
                    └─────────────────────┘            │
                                                        ▼
   waiter  waiter  waiter  waiter  ... all subscribed, all wake up together
      └───────┴───────┴───────┴───── then read the value from Redis, once

No busy polling, no wasted Redis calls, and the waiters get the result the instant it is ready.

Step 6: the tradeoff, and why it is fine

Now the honest part. Have we actually made anything faster for the user? Not really. The waiting requests still have to wait for the first request to finish its database query. If that query takes 800ms, everyone waits roughly 800ms.

But here is the thing, and this is the whole point. That 800ms was going to be paid anyway. The data genuinely takes 800ms to produce. In the naive version, every one of the million requests paid its own 800ms and piled 800ms of load onto the database a million times over. In the coalesced version, everyone pays the same single 800ms, and the database does the work once.

So we did not remove the wait, we removed the duplication. One expensive operation instead of a million, and the user experience is identical, they were going to wait that long either way. That is a fantastic trade, because the cost we cut (database meltdown) is huge and the cost we kept (a wait that was unavoidable) is something we could not have avoided anyway.

The sharp edges (do not skip these)

A few things that will bite you if you ship the simple version:

• The worker can die holding the lock. That is why the lock has a TTL (EX 10). If the holder crashes, the lock expires and another request can take over. Without a TTL, one crash locks that key forever.
• Waiters need a timeout and a fallback. Never wait forever for a signal that might never come (the publish could be missed if you subscribe a hair too late). If the wait times out, fall back to doing the work yourself. Correctness first, coordination second.
• There is a tiny race between subscribing and the publish. Always re-check the cache right after subscribing, in case the value landed in the gap. The snippet above does this.
• Consider stale-while-revalidate. An even smoother pattern is to serve the slightly old value while one background request refreshes it. Then nobody waits at all, they just get data that is a few seconds stale. Great when a little staleness is acceptable, which for a viral read-heavy page it usually is.
• Add jitter to your TTLs. If a lot of keys share the exact same expiry, they all stampede at the same second. A little randomness in the TTL spreads the misses out.

Wrapping up

The chain of reasoning is the nice part here, so let me lay it out one more time:

1. A million requests want the same data, so put Redis in front and most of them are served from cache.
2. But at a cold start or the moment the key expires, they all miss together and stampede the database, and duplicated writes and inserts are even worse than reads.
3. So use single flight, only one request does the real work. In one process a promise map is enough.
4. Across many pods, coordinate with a Redis lock (SET NX EX) so exactly one request in the whole cluster does the work.
5. Let the other requests wait, and use pub/sub instead of polling so they wake up the instant the result is ready.
6. The wait was unavoidable anyway, so the real win is turning a million expensive operations into one.

None of this is exotic. It is just Redis being used as a coordination point and not only as a cache, which is a theme worth internalizing. If you want, I have a separate deep dive on how Redis actually works on the inside, single thread, expiry, pub/sub and all, which pairs well with this one.

If you have used a different approach for cache stampedes, tell me in the comments, I would like to hear it.

Let me go top to bottom through MySQL, or more correctly InnoDB, the storage engine almost everyone uses. We start from the smallest unit on disk, the page, and build up to B+Tree indexes, clustered vs secondary indexes, how the optimizer decides whether to even use an index, when a normal index fails and you reach for spatial indexing, and finally how WAL, the binlog and replication keep everything durable and copied. By the end it should read as one connected story, not a bag of features.

The page: the unit of everything

Before indexes and trees, there is one idea that everything else sits on top of, the page.

(Quick word on InnoDB, since I keep saying it. InnoDB is MySQL’s default storage engine, the part that actually stores rows on disk and handles indexes, transactions and crash recovery. When people say “MySQL internals” today, they almost always mean InnoDB.)

InnoDB does not read or write single rows from the disk. It reads and writes in fixed size blocks called pages, and by default a page is 16 KB. Even if you want one small row, InnoDB pulls the whole 16 KB page that contains it into memory.

Why work in blocks like this? Because disk access is slow and it is slow in a very specific way. The expensive part is finding the data (the seek), not reading a bit more once you are there. So reading one row and reading 16 KB around it costs almost the same. It is much cheaper to read a decent block once than to go back to the disk again and again for tiny pieces.

These pages live on disk, but the hot ones are cached in memory in something called the buffer pool. The buffer pool is just a big chunk of RAM where InnoDB keeps the pages it is using. When you query a row, InnoDB first checks if its page is already in the buffer pool. If yes, no disk access at all. If not, it reads the page from disk into the buffer pool and then uses it. This is why the second run of the same query is often much faster than the first.

What one page actually holds

A page is not just a bag of rows. Crack one open and you find, top to bottom:

• a small header — what page this is, its type, and pointers to the previous and next page at the same level (this is what chains the leaves together),
• the rows themselves, kept in key order as a linked list; on a clustered-index leaf each row also carries two hidden fields, DB_TRX_ID and DB_ROLL_PTR, which are what make MVCC and rollback work,
• some free space that new rows grow into,
• a tiny page directory — a few slots pointing into the rows, so a lookup inside the page is a binary search and not a linear scan,
• and a trailer with a checksum, to catch a page that was only half written before a crash.

You do not need to memorise the layout. The one thing worth keeping is that a page is a self-contained little unit: its rows in sorted order, plus pointers to its neighbours. That is exactly what lets the tree on top of it work.

So the mental model to carry for the rest of this post is simple. The database is a pile of 16 KB pages, some on disk and some cached in memory, and almost everything is about reading as few pages as possible.

How the data is arranged: B-Tree, then B+Tree

Now, if the table is just a pile of pages, how do we find the right page quickly? If we had to scan every page to find id = 42, a large table would be painfully slow. This is the whole reason indexes exist, and the data structure behind them is the B+Tree.

Let me build up to it, because the “B+” part actually matters.

First, the B-Tree

A B-Tree is a balanced tree made for disk. Unlike a normal binary tree where each node has 2 children, a B-Tree node can have many children, hundreds of them. Each node holds a bunch of sorted keys, and between those keys are pointers to child nodes.

Why so many children per node? This goes straight back to the page. One tree node is stored in one page. A page is 16 KB, and a single entry (one key plus a child pointer) is only a handful of bytes, so a node can hold hundreds of entries. This “how many children per node” number is called the fan out, and keeping the fan out high is the whole game.

A quick clarification, because this genuinely trips people up. The 16 in “16 KB page” is the page size, it is not the number of children. A node does not have 16 children, it has as many as physically fit in that 16 KB. For a typical integer key that is roughly 16 KB / (key + pointer), which lands somewhere around 1000 children per node in practice (after row overhead, and because pages are not kept 100% full). So the fan out is in the hundreds to a thousand, not sixteen.

Now the height, which is the payoff. With a fan out of about 1000, the tree stays incredibly short:

1
2
3
height 1  →  ~1,000 rows
height 2  →  ~1,000,000 rows
height 3  →  ~1,000,000,000 rows

So even a table with hundreds of millions of rows is only 3 or 4 levels deep. Since the height is exactly how many pages you must read to reach any row, that means any row is 3 or 4 page reads away, and the top one or two levels are almost always sitting in the buffer pool already. That is the magic. And because these trees are always kept balanced (every leaf at the same depth), that shallow number is not the best case, it is the worst case, the same for every single row.

Two things set that fan out, and both are levers you can feel. One is the page size — a 16 KB page simply fits more entries than a 4 KB one. The other is the size of each entry — a compact INT key with a small child pointer packs far tighter than a fat random UUID. Shrink the entry and the node gets wider, the tree gets flatter, and every lookup costs one fewer page read. That is a good part of why 16 KB is the default (flat tree, but reads still cheap) and why a small integer primary key is worth reaching for: both keep the fan out high.

Here is the catch with a plain B-Tree, and it is exactly what your fan out depends on. In a plain B-Tree, the data itself (the row, or the value) is stored inline in every node, including the internal ones. That data takes up space. So each entry becomes key + data + pointer, which is fat, and far fewer entries fit in the 16 KB page. Fewer entries per node means a lower fan out, a lower fan out means a taller tree, and a taller tree means more page reads for every lookup. The very thing that made the tree fast, packing lots of children into one page, gets spoiled by carrying data around in the internal nodes.

Run a query on a B-Tree: every node stores rows, and there are no links between the bottom nodes. (open the B-Tree visualizer)

Now, the B+Tree (what databases actually use)

A B+Tree is a small but important variation:

• All the real data lives only in the leaf nodes, the bottom level. The internal nodes hold only keys and pointers, they act purely as a directory to guide you down.
• The leaf nodes are linked together like a linked list, left to right in sorted order.

These two changes give databases exactly what they need:

• Range queries become trivial. For WHERE id BETWEEN 40 AND 90, you walk down to the leaf holding 40 once, then just follow the leaf to leaf links to the right until you pass 90. No going back up the tree. A plain B-Tree cannot do this cleanly.
• The internal nodes stay tiny, because they hold only keys and pointers, no data. This is the direct fix for the B-Tree problem above. With no data bloating them, far more entries fit in each 16 KB page, so the fan out is high again and the tree is short again. All the fat data sits down in the leaves, where it does not hurt the fan out of the directory above it.
• Full ordered scans are just walking the leaf chain from left to right.

So the one line summary is, MySQL stores your indexes (and as we will see, your table itself) as B+Trees, and a B+Tree is basically a very short, very wide tree, tuned so that finding any row takes only a handful of page reads.

The same query on a B+Tree: only the leaves store rows, and they are linked — so a range scan glides straight along the leaf chain. (open the B+Tree visualizer)

Clustered vs secondary index (this is the part people miss)

Here is something that surprises a lot of people. In InnoDB, the table itself is a B+Tree. There is no separate “heap” of rows sitting somewhere with indexes pointing into it. The table is an index.

The clustered index

Every InnoDB table is physically stored as one big B+Tree, sorted by the primary key. This is called the clustered index. The leaf nodes of this tree do not just hold the key, they hold the entire row. So when you look up by primary key, you walk down the tree and the moment you reach the leaf, the full row is right there. One traversal, done.

This is also why the choice of primary key matters so much in InnoDB. Since the whole table is sorted and stored by it:

• A small, ever increasing primary key (like an auto increment integer) is ideal. New rows just get appended to the end, pages fill up neatly.
• A large or random primary key (like a random UUID) is worse. Inserts land in random places in the tree, causing pages to split and fragment, and every secondary index gets bigger too (I will explain why in a second).

If you do not define a primary key, InnoDB quietly creates a hidden one for you, so a clustered index always exists.

The secondary index

Now what about an index on some other column, say email? That is a secondary index. It is also a B+Tree, sorted by email this time. But here is the twist, its leaf nodes do not store the full row. They store the indexed column plus the primary key of that row.

So a lookup by email is actually two lookups:

1
2
1. Search the email B+Tree  ──►  find the leaf  ──►  it gives you the primary key
2. Search the clustered index by that primary key  ──►  finally get the full row

This second hop, from the secondary index back to the clustered index to fetch the rest of the row, is often called a bookmark lookup or “index back to table”. It is usually cheap, but it is not free, and it explains a couple of things:

• Why secondary indexes store the primary key, and therefore why a fat primary key makes every secondary index fatter.
• Why a covering index is so nice. If your query only needs columns that are already inside the secondary index, InnoDB can answer it entirely from that index and skip the second hop completely. For example, an index on (email, name) can answer SELECT name FROM users WHERE email = ? without ever touching the clustered index. This is a very common and very effective optimization.

So the picture is, one clustered index that is the table sorted by primary key, and any number of secondary indexes that point back to it using the primary key.

The optimizer: should it even use the index?

Here is a myth worth killing early. Adding an index does not guarantee it will be used. MySQL has a cost based optimizer, and for every query it estimates the cost of different plans and picks the cheapest one. Sometimes the cheapest plan is to ignore your index and just scan the whole table.

That sounds wrong at first, but think about it with the page model.

Using a secondary index means, for each matching row, do a bookmark lookup back into the clustered index. Those lookups jump around the tree, so they are close to random page reads. A full table scan, on the other hand, reads the clustered index leaves in order, which is a nice sequential sweep.

Now imagine your query matches most of the table, for example WHERE is_active = 1 where 90% of users are active. Using the index would mean doing a random bookmark lookup for 90% of all rows, which is far more expensive than just sweeping the table once in order. So the optimizer correctly chooses the full scan.

The idea behind this decision is selectivity (or cardinality). Selectivity is how many distinct values a column has, or put simply, how good the column is at narrowing things down:

• High selectivity (like email, almost unique) means an index lookup returns very few rows, so the index is a big win.
• Low selectivity (like a status column with 3 possible values, or a boolean) means the index returns a huge chunk of the table, and past a certain tipping point the optimizer will skip it.

How does the optimizer even know the selectivity without running the query? It keeps statistics, rough histograms and cardinality estimates for your columns, sampled from the pages. This is also why stale statistics can lead to bad plans, and why ANALYZE TABLE (to refresh stats) sometimes fixes a query that suddenly went slow.

The practical takeaways:

• Do not index low selectivity columns on their own and expect magic. A boolean index is rarely useful by itself.
• Composite index order matters. Put the most selective, most equality filtered column first.
• Use EXPLAIN to see what the optimizer actually decided. If it says it is doing a full scan when you expected an index, the answer is usually selectivity or stale statistics, not a broken index.

When a normal (scalar) index fails

Everything so far assumes we are indexing a single scalar value that has a natural order, a number, a date, a string. B+Trees are perfect for that, because the whole idea of the tree is “keep the keys sorted in one dimension”.

But that single dimension assumption is exactly where B+Trees break down.

Think about location data, a latitude and a longitude, and a query like “find all restaurants within 2 km of me”. This is a two dimensional question. You care about latitude and longitude together.

Suppose you put a B+Tree index on (latitude, longitude). The tree sorts primarily by latitude. So it can quickly narrow down “all points in this latitude band”, but within that band the longitudes are all over the place. To find a small 2 km box you would still have to scan a huge strip of the earth at the right latitude and filter longitude by hand. The index helps with one axis and is useless on the other.

The core problem is this. A B+Tree can only order data along one line. Space is not a line, it is a plane. There is no way to lay out 2D points on a single ordered line so that points near each other on the map are always near each other in the ordering. This is not a MySQL limitation, it is a property of the data structure, and to index space you need one that understands more than one dimension.

Geospatial indexing: quad-trees and R-trees

The fix is to divide space itself instead of ordering values on a line.

The quad-tree, just for intuition

The easiest way to picture space partitioning is a quad-tree. Take the whole map as one square. If a square holds too many points, split it into four quadrants, and keep splitting the crowded ones. A city ends up finely split, an ocean stays one big square.

Drop a latitude/longitude and press Run — watch the grid keep splitting into the dense area. (open the quadtree visualizer)

“Find everything within 2 km of me” becomes a tree walk: start at the root and only descend into the squares that overlap your search box, skipping the rest of the map. That pruning is exactly what a scalar index could not do in 2D.

What MySQL actually uses: the R-tree

A quad-tree is a great mental picture, but MySQL’s SPATIAL index (on GEOMETRY, POINT and similar) is built on an R-tree, not a quad-tree. An R-tree does not carve space up on a fixed grid. Instead it groups nearby objects together and stores a minimum bounding rectangle (MBR) around each group, the smallest box that encloses everything inside it. The leaves hold the actual geometries; the internal nodes hold boxes around boxes. To search, you start at the root and only walk down into the boxes that overlap your query box, pruning the rest.

The same location on an R-tree — far fewer boxes opened, and it stays shallow. (open the R-Tree visualizer)

Quad-tree vs R-tree

Both prune space to answer “near me” queries, so why does a database reach for the R-tree? It comes down to what each one actually divides.

The last two rows are the whole reason MySQL picks an R-tree. A quad-tree splits space blindly, so its depth follows the data, a dense city block pushes one branch very deep while an ocean stays shallow. That lopsided shape does not map neatly onto fixed-size disk pages. An R-tree instead splits an overflowing node to stay balanced and wide, exactly the short, page-per-node shape the rest of this post has been leaning on, so every leaf is the same few page reads from the root. That is what makes it a database index and not just a nice diagram.

The one price the R-tree pays for grouping data instead of tiling space is that its boxes can overlap. A quad-tree’s quadrants never overlap, so a search there follows a clean path down. In an R-tree, two sibling boxes can cover the same patch of map, so your query box might land inside several of them at once and the search has to walk down more than one branch. That is the trade for staying balanced and disk-friendly.

In practice you rarely touch any of this directly:

1
2
3
4
5
6
7
-- a spatial index on a POINT column
ALTER TABLE places ADD SPATIAL INDEX (location);

-- "find places whose location is inside this box"
SELECT name
FROM places
WHERE MBRContains(@search_area, location);

So the takeaway: a B+Tree orders values along one line and is perfect for = and range queries on a single dimension, while an R-tree carves space into (possibly overlapping) boxes and is what you reach for the moment the question is really “near me” or “inside this region”. When a scalar index falls apart on location data, the R-tree is the tool you were missing.

Durability: the Write Ahead Log (WAL)

We have covered how MySQL finds your data. Now the other half, how it does not lose your data when the power goes off mid write.

Here is the tension. All the changes happen on pages in the buffer pool, in memory. Writing those changed (dirty) pages back to disk immediately, for every single commit, would be terrible, because those page writes are scattered randomly across the file and random disk writes are slow. But if we keep changes only in memory and the server crashes, the changes are gone.

The way out is Write Ahead Logging, or WAL. The rule is simple:

Before a change is considered committed, write a small record of that change to a log first. Only then acknowledge the commit. The actual data pages can be flushed to disk later, lazily.

In InnoDB this log is the redo log (the ib_logfile files). It works like this:

1
2
3
4
5
6
COMMIT
  1. Change the page in the buffer pool (in memory)         [fast]
  2. Append what changed to the redo log, then fsync it     [fast, sequential]
  3. Acknowledge the commit to the client                   [done]
  ...later...
  4. Flush the dirty page to its real place on disk         [lazy, batched]

The important trick is that the redo log is written sequentially, appended to the end. Sequential disk writes are dramatically faster than random ones. So instead of doing slow random page writes on every commit, InnoDB does one fast sequential log append on commit, and pushes the slow random page writes to the background where they can be batched.

Now the crash safety. If the server dies after step 2 but before step 4, the data page on disk is stale, but the change is safely in the redo log. On restart, InnoDB does crash recovery: it reads the redo log and replays any changes that had not yet made it to the data pages. So a committed transaction survives even though its page was never written. The redo log is a fixed size ring that is reused in a circle, since once a dirty page is safely flushed, its old log records are no longer needed.

There is a matching undo log as well, which stores the previous version of rows. It is used to roll back a transaction that did not commit, and to give other transactions a consistent older snapshot to read (this is how MVCC and consistent reads work). But the one big idea to hold onto is WAL: log the change first, flush the pages later, replay the log after a crash.

The binlog, and how it is different from the redo log

Here is a point that confuses almost everyone the first time, because MySQL has two logs that sound similar but do completely different jobs.

The redo log we just saw is an InnoDB, storage engine level thing. It is physical (“page 5 byte 100 changed to this”), it is circular (overwritten once flushed), and its only purpose is crash recovery. Nobody outside InnoDB reads it.

The binlog (binary log) is a server level log, above the storage engine. It records the changes as logical events, and it is append only, kept around for as long as you configure. Its purpose is completely different, it is for:

• Replication, shipping changes to replica servers (the big one).
• Point in time recovery, restore last night’s backup, then replay the binlog up to 2 seconds before the bad DELETE.

The binlog can be in different formats:

• STATEMENT based, it logs the actual SQL (UPDATE users SET ...). Compact, but risky for non deterministic statements (think NOW() or RAND()).
• ROW based, it logs the actual before and after of each changed row. Safer and the common default today, at the cost of more log volume.
• MIXED, use statement where safe, fall back to row where not.

Because there are two logs, a commit has to make sure they agree, otherwise after a crash the redo log and the binlog could disagree about whether a transaction happened, and a replica would drift from the primary. InnoDB solves this with an internal two phase commit between the redo log and the binlog: prepare in the redo log, write the binlog, then mark the redo log committed. If a crash happens in the middle, recovery uses the presence of the binlog entry to decide whether to roll the transaction forward or back, so both logs always end up telling the same story.

Quick way to remember the two:

• Redo log, InnoDB, physical, circular, for surviving a crash.
• Binlog, server layer, logical, append only, for replication and recovery.

Replication: copying the data to other servers

Now the binlog pays off. Once you have a log of every change in commit order, you can hand that stream to another server and have it apply the same changes, and now you have a replica that stays in sync with the primary.

The basic flow:

1
2
3
4
5
6
   PRIMARY                                 REPLICA
 ┌──────────┐    binlog events    ┌────────────────────────┐
 │ writes   │ ──────────────────► │ I/O thread writes them  │
 │ + binlog │                     │ to a local relay log    │
 └──────────┘                     │ SQL thread replays them │
                                  └────────────────────────┘

The primary writes to its binlog as usual. Each replica has an I/O thread that pulls those binlog events over the network into a local relay log, and one or more SQL threads that replay the events to apply the exact same changes. The replica ends up with the same data.

The one tradeoff you must understand is how synchronous it is:

• Asynchronous (the default). The primary commits and tells the client “done” without waiting for any replica. Fast, but if the primary dies before a replica has caught up, those last few transactions can be lost. There is also a small replication lag, so a read from a replica can be a little behind the primary.
• Semi synchronous. The primary waits until at least one replica has received (not necessarily applied) the change before acknowledging the commit. Safer against data loss on failover, slightly slower.
• Group replication / fully synchronous setups go further and coordinate a group of nodes for stronger guarantees, at more cost and complexity.

What replication buys you:

• Read scaling. Send heavy read traffic to replicas and keep the primary for writes. Just remember the lag, a value you wrote a moment ago might not be on the replica yet, so read your own writes from the primary when it matters.
• High availability. If the primary dies, promote a replica to be the new primary (failover).
• Backups and analytics off a replica, so you do not load the primary.

This is also the exact same binlog machinery that tools like Debezium tap into for change data capture, streaming every row change out to systems like Kafka. So the humble binlog is not just for MySQL to MySQL replication, it is the source of truth for a whole world of downstream pipelines.

Locking: optimistic, pessimistic, and what InnoDB does by default

The moment two people touch the same row at the same time, something has to decide who waits. There are two broad philosophies.

Pessimistic locking assumes a clash is likely, so it locks the row up front and makes everyone else wait. In MySQL you ask for this with a locking read:

1
2
3
-- take an exclusive lock on the row until my transaction ends
SELECT * FROM accounts WHERE id = 7 FOR UPDATE;
-- ...now update it safely; anyone else touching row 7 blocks until I commit

FOR UPDATE takes an exclusive lock, FOR SHARE a shared one, and the lock is held until you commit or roll back. Great when contention is real and transactions are short. The cost is that blocked transactions sit and wait, and if two of them wait on each other you get a deadlock (InnoDB detects it and kills one).

Optimistic locking assumes clashes are rare, so it takes no lock at all. You read the row along with a version number (or an updated_at), do your work, and only at write time check that nobody changed it underneath you:

1
2
UPDATE accounts SET balance = 900, version = version + 1
WHERE id = 7 AND version = 42;

If that UPDATE touches 0 rows, someone else bumped the version first, so you re-read and retry. No waiting, no lock held across your think-time; the cost is the occasional retry. Note this is a pattern you build, a column plus a check, MySQL has no built-in optimistic mode (though ORMs like Rails and Hibernate ship a helper for it).

So what does MySQL default to? InnoDB is pessimistic for writes. Every UPDATE, DELETE and INSERT automatically takes row-level locks on the rows it touches (and, under the default REPEATABLE READ isolation, gap / next-key locks to keep phantoms out) — you do not ask for it, it just happens, and it is lock-based, not version-based. Plain SELECTs are the happy exception: thanks to MVCC (the undo log from earlier), a normal read takes no lock and simply sees a consistent snapshot, so readers never block writers and writers never block readers. Optimistic locking is never automatic, if you want it you add the version column yourself.

Putting it all together

If I compress the whole journey into a few lines:

Everything sits on 16 KB pages, read in blocks because disk seeks are the expensive part, and cached hot in the buffer pool. Data and indexes are stored as B+Trees, short and wide so any row is only a few page reads away, with linked leaves that make range scans cheap. The table itself is the clustered index sorted by primary key, and secondary indexes point back to it through the primary key, which is why covering indexes are so nice. The optimizer does not blindly use your index, it weighs selectivity and can prefer a full scan when an index would cause too many random lookups. A normal B+Tree only orders one dimension, so for “near me” location queries it fails, and you switch to a spatial (R-tree / quad-tree style) index that divides space into boxes. Durability comes from WAL, log the change to the redo log first and flush pages lazily, then replay the log after a crash. The binlog is a separate, logical, append only log that powers replication and point in time recovery, copying your data to replicas for read scaling and failover. And when two writers reach for the same row, row-level locks decide who waits, pessimistic by default, while MVCC lets plain reads skip locking altogether.

None of these are really separate features. They are all consequences of two simple facts, disk is slow and works in blocks, and memory is fast but not durable. Once you see MySQL through those two facts, most of its design stops being mysterious.

If you would rather poke at these ideas than read about them, I built a set of interactive data-structure simulators to go with this post: run a query through a B-Tree and a B+Tree, or hand a latitude/longitude to a quad-tree and an R-Tree and watch which one gets to you faster.

If you want me to go deeper into any one of these, MVCC and how reads stay consistent, or how EXPLAIN plans actually read, tell me in the comments and I will write a focused follow up.

So in this post I want to do three things, in order. First explain the entities in plain words. Then look at a real, high scale use case, an uptime monitoring service. And finally, the part most tutorials skip, how do you actually decide the numbers, how many partitions, how many consumers, how many brokers, and how to set up the consumer group.

Here is the problem we will design for the whole way through.

We run an uptime monitoring service (think of something like Pingdom or UptimeRobot). We have 10 million monitors in our database, each one is a URL or endpoint that we must check regularly, say every 5 minutes, to see if it is up. Each check is a slow outbound HTTP call. How do we fan this huge amount of work out across many worker machines, reliably, without hammering one box or losing checks?

This is a textbook Kafka job. Let’s build up to it.

The entities, in plain words

Think of Kafka as a durable log of events that many services can write to and read from. Here are the pieces, tied to our monitoring service.

• Event (message): a single record. For us, one “check job”, like {"monitorId":123,"url":"https://acme.com/health"}. Kafka does not care what is inside, it just stores the bytes.

• Topic: a named stream of events, for us health.checks. Producers write to it, consumers read from it. A topic is really just an append only log.

• Partition: the important one. A topic is split into one or more partitions, and each partition is an ordered, append only sequence of events. Partitions are how Kafka scales, because different partitions can live on different machines and be read in parallel. Order is guaranteed within a partition, not across partitions.

• Offset: the position of an event inside a partition, just a number that keeps increasing. A consumer remembers “I have read up to offset 42 in partition 0”, and that is how it knows where to resume.

• Broker: a single Kafka server. A broker holds some of the partitions and serves reads and writes for them. A cluster is a group of brokers working together.

• Producer: whoever writes events into a topic. For us, the scheduler that decides which monitors are due and enqueues a check job for each. When it writes, it can pick a key (the monitor id), and Kafka uses that key to decide which partition the event goes to.

• Consumer: whoever reads events and does the work. For us, a worker that takes a check job, makes the HTTP call, and records whether the site was up.

• Consumer group: a set of consumers that share the work of reading a topic. Kafka gives each partition to exactly one consumer inside a group. So if the topic has 200 partitions and the group has 50 consumers, each consumer reads 4 partitions. Add more consumers and the work gets rebalanced. This is how we scale the workers horizontally, up to the number of partitions.

• Replication: each partition can be copied to multiple brokers. One copy is the leader and the others are followers that stay in sync. If the broker holding the leader dies, a follower takes over, so we do not lose queued check jobs. The number of copies is the replication factor (3 is common in production).

The use case: designing a health monitoring system

Let’s design a small uptime monitoring system, the kind of thing that watches a set of websites or endpoints and tells you the moment one goes down. A user adds the URLs they want watched, and the system’s job is to check each one on a schedule (say every 5 minutes), record whether it was up or down, and alert someone if it was down.

At a small scale you could do this with a cron job and a simple loop. The interesting part is scale. Imagine we are watching 10 million monitors, each one due for a check every 5 minutes, and each check is a slow outbound HTTP call that can take a second or more. Now we have to spread that work across many machines, not lose any checks if a machine dies, and absorb the bursts when a lot of monitors fall due at the same instant. That combination, high fan-out, slow per item work, and no data loss, is exactly what Kafka is good at. So we will put Kafka at the center of the design.

Here is the shape of the pipeline.

1
2
3
4
5
6
7
 Scheduler (producer)                         Worker pods (consumers)

  every tick, find monitors due  ──►  ┌────────────────┐  ──►  worker pulls a job
  publish one job per monitor    ──►  │  health.checks  │  ──►  makes the HTTP call
  (keyed by monitor id)               │  (Kafka topic,  │  ──►  records up / down in DB
                                       │   many partitions)│
                                       └────────────────┘

Two moving parts:

• A scheduler (the producer) wakes up on each tick, queries the DB for monitors that are due for a check, and publishes one check job per monitor onto the health.checks topic. On a busy tick this can be hundreds of thousands of jobs at once.
• A pool of worker pods (the consumers, all in one group) read those jobs, make the actual HTTP call to each target, and write the result back.

Why put Kafka in the middle instead of just having the scheduler call the workers directly?

• It absorbs bursts. If a tick dumps 300k jobs and the workers can only chew through them over the next couple of minutes, the jobs simply wait in the topic. Nothing is dropped, the workers just catch up.
• It decouples scheduling from checking. The scheduler does not know or care how many workers exist. You can scale workers up and down freely.
• It survives worker crashes. If a worker dies mid batch, those jobs are not lost, another worker picks them up from the last committed offset.
• You can replay. If a bug made you record bad results for an hour, you can reset the offset and reprocess.

Here is the producer (the scheduler), publishing jobs keyed by monitor id, using kafkajs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
const { Kafka } = require("kafkajs");

const kafka = new Kafka({ clientId: "scheduler", brokers: ["broker1:9092", "broker2:9092"] });
const producer = kafka.producer();

async function enqueueDueChecks(dueMonitors) {
  await producer.send({
    topic: "health.checks",
    messages: dueMonitors.map((m) => ({
      key: String(m.id),                       // same monitor -> same partition
      value: JSON.stringify({ monitorId: m.id, url: m.url }),
    })),
  });
}

And the worker (a consumer), doing the actual check and recording the result. Note the manual offset commit, more on that below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
const { Kafka } = require("kafkajs");

const kafka = new Kafka({ clientId: "health-worker", brokers: ["broker1:9092", "broker2:9092"] });
const consumer = kafka.consumer({ groupId: "health-workers" });

async function start() {
  await consumer.connect();
  await consumer.subscribe({ topic: "health.checks", fromBeginning: false });

  await consumer.run({
    autoCommit: false, // we commit ourselves, only after the check is done
    eachMessage: async ({ topic, partition, message }) => {
      const { monitorId, url } = JSON.parse(message.value.toString());

      const result = await runHealthCheck(url);   // the slow HTTP call
      await db.results.upsert({ monitorId, result, at: Date.now() });

      // only now do we move the bookmark forward
      await consumer.commitOffsets([
        { topic, partition, offset: (Number(message.offset) + 1).toString() },
      ]);
    },
  });
}

A couple of things to notice, and they matter a lot at this scale. I keyed jobs by monitor id so two overlapping checks for the same monitor cannot run out of order or in parallel on different workers. And recording the result is an upsert, because Kafka delivers “at least once”, so the same job can occasionally be processed twice, and doing it twice should be safe.

A common misconception: is Kafka pub/sub or a message queue?

Worth clearing up, because it trips up almost everyone. People often say “Kafka is a pub/sub system, not a message queue.” That is only half right. Kafka is really a distributed log, and depending on how you use consumer groups it can behave as either one.

• Like a queue (competing consumers). Put all your consumers in the same group, and Kafka splits the partitions among them, so each message is handled by exactly one consumer in that group. That is a work queue, and it is exactly what our monitoring workers do, the check jobs get divided across the one health-workers group.
• Like pub/sub (fan-out). Add a second consumer group on the same topic, and that group independently reads all the messages too. So the health-workers group can be doing the checks, while a separate audit group reads the same health.checks topic to log every job. Each group gets the full stream.

The other thing that makes Kafka different from a classic queue like RabbitMQ or SQS, it does not delete a message when it is consumed. A traditional queue removes a message once a consumer acknowledges it. Kafka keeps every message for its retention period, and each consumer group just tracks its own offset. That is what lets multiple groups read the same data independently, and what lets you replay by rewinding an offset. In a classic queue, once a message is consumed, it is gone.

So the accurate way to say it is, Kafka is a log that gives you queue semantics within a group and pub/sub semantics across groups. Our monitoring design leans on the queue side.

Now the real question: how do you pick the numbers?

This is where most explanations stop, but it is the part that actually matters when you build it. Let’s start with the load.

• 10 million monitors, each checked every 5 minutes (300 seconds).
• So the pipeline must sustain 10,000,000 / 300 ≈ 33,000 checks per second.

Hold on to that number, ~33k/sec. Everything below flows from it.

Producers

On the producer side (the scheduler), three decisions matter.

• Partition key. Key every job by the monitor id. Kafka hashes the key to choose a partition, so all jobs for a given monitor land on the same partition and are processed in order by one worker. That stops two workers checking the same monitor at the same instant. (If you did not care about that, you could skip the key and let jobs spread evenly, but here keying is the safer choice.)
• acks (durability). Set acks=all so a job is only acknowledged once the leader and its in sync replicas have it. You do not want to lose a batch of queued checks because a broker restarted.
• Idempotent producer. Turn on enable.idempotence=true so an internal retry does not enqueue the same job twice.

You do not need many producer instances, a single scheduler (or a small number) can push tens of thousands of small messages per second through a few connections.

Brokers and replication factor

A broker is one Kafka server, and your partitions physically live on brokers. The replication factor is how many copies of each partition Kafka keeps on different brokers for fault tolerance.

The rule that confuses people: your replication factor cannot exceed the number of brokers. The reason is simple once you see it, each copy of a partition has to sit on a different broker (two copies on the same machine would both die together, which defeats the point). So a replication factor of 3 needs at least 3 brokers.

For production you almost always want replication factor 3 (plus min.insync.replicas=2), because then you can lose one entire broker and still have the data on two others and keep accepting writes.

Now bring in our numbers. Say we settle on 200 partitions (why, in a second). With replication factor 3 that is 200 × 3 = 600 partition copies that have to be spread across the cluster. On a cluster of, say, 6 brokers, that is about 100 partition copies per broker, comfortable. If you only had 3 brokers, each would carry all 200, which is heavier and leaves no room when one broker dies. So “more brokers” buys you two things, headroom for the replication factor, and spreading the partitions (and their load) across more machines. As an app team you usually do not own the broker count, you just make sure the cluster has enough brokers for RF 3 and your partition count.

Partitions (the most important choice)

Partitions are your unit of parallelism, because a partition is read by only one consumer in a group. So the partition count is the ceiling on how many workers can run in parallel. This is the number to decide first.

Here is the key insight for this use case. Our messages are tiny (a monitor id and a URL, maybe 200 bytes), so at 33k/sec we are only moving about 6 MB/s, which is nothing for Kafka. The partition count here is not about byte throughput at all. It is about processing parallelism, because each job is a slow outbound HTTP call.

So size partitions by how much work one consumer can do:

1
partitions ≈ target rate / rate one consumer can handle

Say one worker keeps around 200 HTTP checks in flight and an average check (including slow and timing out ones) takes about a second. That is roughly 200 checks/sec per worker. To hit 33k/sec:

1
33,000 / 200 ≈ 165 workers needed

So you want at least ~165 partitions, and you would round up with headroom to something like 200 partitions. That headroom matters because increasing partitions later is possible but a bit painful (it changes how keys map to partitions).

This is the lesson worth stealing, when each message is slow to process, you need a lot of partitions even though the data volume is tiny. Partition count tracks your parallelism, not your bytes.

Consumers and the consumer group

All your workers share one consumer group, say health-workers. The single rule that governs everything:

1
number of consumers ≤ number of partitions

With 200 partitions you can run up to 200 worker consumers. Beyond 200 the extra ones just sit idle, they get no partition. So you would run something like 20 worker pods with 10 consumer instances each (200 consumers total), or 40 pods of 5, whatever fits your pod sizing.

Where do the consumers run? As their own worker deployment, separate from any API service. Spread the consumers across multiple pods so that if one pod dies, the others keep draining the topic (after a quick rebalance), and Kafka just redistributes that pod’s partitions to the survivors.

How are partitions assigned, and what happens when a worker restarts? That is the assignment strategy and rebalancing, which get their own section next.

A sanity check on the things downstream

One trap, do not forget what the workers actually hit. At 33k checks/sec you are making 33k outbound HTTP calls per second and writing 33k results/sec to your database. Both can become the real bottleneck long before Kafka does. So batch the result writes, size your DB connection pools, and make sure the targets (and your own egress) can take that rate. It is worth doing this math early, because at this scale Kafka is rarely the part that falls over first.

Auto-commit vs manual commit

There is one consumer setting that quietly decides whether you lose work: when does the consumer commit its offset.

Committing an offset just means the consumer telling Kafka “I have processed up to here”, so that on a restart or rebalance it resumes from that point. There are two ways to do it.

• Auto-commit (enable.auto.commit=true, the default in most clients): the client commits the latest fetched offsets automatically on a timer. Simple, but dangerous for us, because it commits based on what you pulled, not what you finished. If the timer commits a batch of jobs and then the worker crashes before the HTTP checks actually run, Kafka thinks those checks are done and they are silently never run. For a monitoring service, that means a monitor quietly stops being checked.

• Manual commit (enable.auto.commit=false): you commit after the work succeeds, like in the worker code above. If the worker crashes after the check but before the commit, the job just runs again on restart (a duplicate), which is fine because recording the result is an idempotent upsert.

The rule of thumb, if losing a message is acceptable (metrics, low value logs), auto-commit is fine and simpler. If losing a message is not acceptable (a health check that must run, a payment, a sync), use manual commit and commit only after you have really done the work.

Choosing an assignment strategy, and surviving rebalances

Two things about consumer groups trip people up once real traffic hits, how partitions get assigned to consumers, and what happens every time the group changes.

How partitions get assigned

When consumers join a group, Kafka decides which consumer reads which partition. That decision is the assignment strategy:

• Range (the default): for each topic it hands out contiguous ranges of partitions to the consumers. Fine for a single topic like ours, but can get uneven when one group consumes several topics.
• Round robin: spreads partitions one by one across all consumers, so the load is more even, which helps when a group subscribes to many topics.
• Sticky / cooperative sticky: tries to keep each consumer on the partitions it already had, so a rebalance moves as few partitions as possible. On modern Kafka (2.4+), cooperative sticky is usually the best choice, especially with 200 consumers where a full reshuffle is expensive.

What a rebalance is, and why it can hurt

A rebalance is Kafka re dividing the partitions among the consumers whenever group membership changes, a worker joins, leaves, dies, or partitions are added.

The catch is the old eager, stop the world style, where on any change every consumer drops all its partitions and waits for a fresh assignment. With 200 workers, one pod restarting could briefly stall the entire fleet, and checks pile up. That is why the modern cooperative rebalancing matters, only the partitions that actually need to move are handed over, and everyone else keeps working.

Practical tips to keep rebalances calm

• Keep the consumer count stable. Every scale event and crash is a rebalance, so do not autoscale workers aggressively on a spiky signal.
• Tune the timeouts. Set session.timeout.ms and heartbeat.interval.ms so a short GC pause or a slow HTTP check does not get a healthy worker wrongly kicked out.
• Use static membership. A stable group.instance.id per worker lets Kafka recognise a restarting pod as the same member, so a rolling deploy does not reshuffle all 200 partitions.
• Shut down gracefully so a worker commits its offsets and leaves the group cleanly instead of timing out.

A few gotchas worth knowing

• Order is only per partition. Keying by monitor id keeps each monitor’s checks ordered, but there is no ordering across different monitors, which is exactly what we want here.
• Make handlers idempotent. At least once delivery means a job can run twice. Upserting the result (rather than inserting) keeps that safe.
• Watch consumer lag. Lag is how far behind the workers are from the newest offset. Growing lag here has a very concrete meaning, your monitors are being checked late. If lag climbs, you need more partitions and workers, or faster checks.
• Handle poison jobs. A URL that always times out should not be retried forever and block a partition. Cap the retries and push it to a dead letter topic for later inspection.
• Mind the downstream. At 33k/sec the HTTP targets and your database are the likely bottlenecks, not Kafka. Batch writes and pool connections.

Wrapping up

If you strip Kafka down to one idea, it is this. A topic is a log, split into partitions, and partitions are your unit of parallelism. Almost every sizing decision flows from that. In our monitoring pipeline the whole design fell out of two numbers, 33k checks per second and about 200 checks per second per worker, which told us we needed roughly 200 partitions, up to 200 worker consumers spread across pods, on a cluster with enough brokers to hold replication factor 3.

And notice the theme that kept coming back, partitions track your processing parallelism, not your data volume. Our messages were tiny, but the work behind each one was slow, so we needed a lot of partitions. Get that one idea right and the rest of Kafka is mostly variations on it.

If you want, I can write a follow up on the scheduler side (how to avoid a thundering herd every tick), or on failure handling with retries and a dead letter topic. Tell me in the comments.

In this post I want to go in and out of Redis. How it stays this fast on a single thread, how it deletes expired keys in the background, how persistence works, what Lua scripting gives you, and finally the part that confuses most people, when to use Sentinel and when to use Cluster.

Let’s start from the very beginning.

What is Redis, really

Redis (REmote DIctionary Server) is an in memory data store. At its core it is basically a giant key value dictionary that lives in RAM, and that single fact explains most of its behaviour.

Because the data sits in memory and not on disk, reads and writes are extremely fast. A normal database has to go to the disk, deal with page caches, indexes and so on. Redis just looks up a key in an in memory hash table, which is an O(1) operation most of the time.

But Redis is not only strings. The values can be proper data structures:

• String for simple values, counters, JSON blobs
• Hash for objects with fields, like a user record
• List for queues and stacks
• Set for unique items
• Sorted Set (ZSet) for leaderboards and ranking, where each item has a score
• and a few special ones like Streams, Bitmaps, HyperLogLog and Geo

This is why people call Redis a “data structure server”. You are not just caching strings, you can push to a list, increment a counter or update a leaderboard, all on the server side and all atomic.

The part that surprises everyone: Redis is single threaded

Here is the fact that trips people up. The core of Redis that runs your commands is single threaded. One thread, processing one command at a time.

The first reaction is usually, “how can something single threaded be so fast in 2026 when my laptop has 10 cores?”. It feels backwards. But there are good reasons for it.

Why single threaded actually works here

The data is in memory, so the CPU is rarely the bottleneck. Most Redis commands are simple hash table operations that finish in microseconds. The real cost in a system like this is usually the network and memory, not the CPU doing the work. So adding more threads does not help as much as you would think.

No locks, no race conditions. Because only one thread touches the data, Redis never has to lock anything. There is no risk of two threads corrupting the same key. This makes every single command atomic for free. When you run INCR counter, no other command can sneak in the middle of it. In a multi threaded design you would need locks everywhere, and locks bring their own slowness and bugs.

No context switching. Threads constantly getting scheduled on and off the CPU is not free. A single thread doing tight, small operations avoids all of that overhead.

But then how does it handle thousands of clients at once?

This is the clever part. Redis uses an event loop with I/O multiplexing (epoll on Linux, kqueue on BSD/Mac). Instead of one thread per connection, a single thread watches all the connections and only wakes up for the ones that actually have data ready.

1
2
3
4
5
6
7
8
9
10
11
12
        many clients
   ┌────┬────┬────┬────┐
   │    │    │    │    │
   ▼    ▼    ▼    ▼    ▼
 ┌─────────────────────┐
 │   epoll / kqueue     │   "which sockets are ready?"
 └──────────┬──────────┘
            ▼
   ┌──────────────────┐
   │  single event     │   process one ready command,
   │  loop thread      │   then move to the next
   └──────────────────┘

So one thread can serve tens of thousands of open connections, because at any given moment it is only doing a tiny bit of work for whichever connection has a command ready.

A quick note on the word “client”, because it confused me at first. In Redis, a “client” just means a connection, one open socket. It is not per key, and it is not per application. Your app usually keeps a small connection pool, say 10 connections, and from Redis’s side that pool is 10 clients. Scale out to 40 app instances with 10 connections each, and Redis now sees 400 clients, all multiplexed by that one event loop thread. (The default limit is maxclients 10000, which is where the “tens of thousands” number comes from.) So one connection can read and write any number of keys, there is no such thing as a connection per key.

One small note, because “Redis is single threaded” is only half true today. Since Redis 6 there is optional multi threaded I/O, where extra threads read and write the raw bytes from and to the sockets, and newer versions have made this more effective and turned it on more aggressively. But this only parallelises the network part. The actual command execution, the bit that touches your data, is still one thread. So the atomicity guarantee is exactly the same, no locks and no race conditions, you just get more help draining the sockets when the network is the bottleneck. So the honest one liner is, Redis runs your commands on a single thread, but it is no longer strictly single threaded end to end.

How Redis deletes expired keys

This is one of my favourite parts, because it shows how the single threaded design shapes everything.

When you set a key with an expiry, like SET session:123 abc EX 60, Redis has to remove it after 60 seconds. The obvious way would be to run a timer for every key, but with millions of keys that would be very expensive. So Redis does something smarter, it uses two strategies together.

1. Lazy (passive) expiration

When you try to access a key, Redis first checks if it has already expired. If yes, it deletes it right then and behaves as if the key does not exist.

This is cheap, but it has a problem. If a key expires and nobody ever touches it again, it will just sit there in memory forever, wasting space. So lazy expiration alone is not enough.

2. Active expiration (the background process)

This is the background job people usually do not know about. Redis runs a periodic task (part of its serverCron, roughly 10 times a second by default, controlled by the hz setting) that actively hunts for expired keys.

Each cycle it does roughly this:

1
2
3
4
5
1. Take a sample of ~20 random keys that have an expiry set.
2. Delete the ones that are already expired.
3. If more than 25% of the sample was expired,
   assume there are many more, and repeat immediately.
4. Otherwise stop for this cycle.

It is a probabilistic approach. Redis does not scan every key, that would be too slow. It samples, and if it keeps finding lots of expired keys it keeps going, otherwise it waits for the next cycle.

And here is where single threaded matters again. This cleanup runs on the same single thread that serves your commands. So Redis deliberately time boxes this work. It will not spend too long deleting keys in one cycle, because if it did, your normal commands would be blocked waiting behind it. It is a careful balance between reclaiming memory and staying responsive.

What about under heavy traffic? Since this cleanup runs on the same main thread as your commands, it is fair to worry that it will steal time from real work during a spike. Redis guards against this by capping how long the active cycle is allowed to run each time (roughly a quarter of its small time budget), so your client commands never get starved by expiry work. The one case to watch out for is an “expiry storm”, where a very large number of keys all expire at the exact same instant. Those deletes still happen on the main thread, so they can cause a short latency spike. If you ever set the same TTL on millions of keys at once, it helps to add a little random jitter to the expiry so they do not all die together.

So the real answer to “how does Redis delete expired keys” is, a little bit lazily when you touch them, and a little bit actively in the background, always being careful not to block the main thread.

Persistence: Redis is in memory, but not only in memory

If everything is in RAM, what happens when the server restarts? By default you would lose all the data. That is why Redis has two persistence options.

RDB (snapshots)

RDB takes a point in time snapshot of the whole dataset and writes it to a single file on disk (dump.rdb). You can configure it to snapshot every few minutes, or trigger it with BGSAVE.

The neat trick is how it avoids blocking. Redis fork()s a child process, and the child writes the snapshot while the parent keeps serving traffic. Thanks to copy on write memory in the OS, the child sees a frozen view of the data without actually copying all of it up front.

• Good for backups and fast restarts.
• Downside, if the server crashes between two snapshots, you lose whatever changed in between.

AOF (Append Only File)

AOF logs every write command to a file. On restart, Redis replays the log to rebuild the dataset. You can control how often it flushes to disk with appendfsync:

• always, safest but slowest (flush on every write)
• everysec, flush once a second, a good balance and the common choice
• no, let the OS decide, fastest but least safe

Since the log keeps growing, Redis periodically rewrites it into a compact form that represents the same final state.

Which one?

In practice, many people run both. Modern Redis even supports a hybrid file where an RDB snapshot is used as the base and AOF records the recent changes on top of it. RDB gives you fast restarts and backups, AOF gives you a much smaller window of possible data loss.

What happens on a restart

Since the data lives in memory, a restart has to load it back from disk first, either from the RDB snapshot or by replaying the AOF. This is not instant. The load time grows with the size of your dataset, so a few hundred MB comes back quickly, but many GB can take anywhere from seconds to a few minutes, and AOF replay is usually slower than loading an RDB snapshot. During that load the node is not serving traffic yet, so yes, there is a real cold start cost.

Two things are worth knowing here. In a Sentinel or Cluster setup, a replica keeps serving while one node restarts, so clients are not fully down. And if you run Redis as a pure cache with no persistence at all, a restart is instant, but the cache comes back empty, which can send a sudden flood of misses to your database (a “thundering herd”). So turning persistence off does not remove the restart cost, it just moves it onto your database.

What the managed services hide

If you are on AWS ElastiCache, GCP Memorystore, Redis Cloud, Railway, Upstash and so on, most of this RDB and AOF machinery is handled for you. You mostly just toggle things like “automatic backups” or “enable AOF” in a console, and replication and failover are managed too. It still runs underneath, and the choices still affect your durability and your cost, so it really helps to know what those toggles are actually doing.

Lua scripting: doing many things atomically

So far we have looked at single commands. But sometimes you need a few commands to run together as one unit, and that is where scripting comes in. Redis lets you run Lua scripts on the server using EVAL (and EVALSHA for a cached script). The script runs on the server, right next to the data.

Why is this useful? Two reasons.

Atomicity. Remember Redis is single threaded, so while your Lua script is running, nothing else runs. The whole script executes as one atomic unit. This is perfect for “read a value, decide something, then write” logic where you cannot afford another client to jump in the middle.

Fewer round trips. Instead of sending five commands from your app and paying the network cost each time, you send one script that does all five on the server.

A fair question here, “but I am still calling redis.call five times inside the script, so how is that fewer round trips?”. The trick is where those calls happen. The network cost is between your app and Redis, not inside Redis. Five separate commands from your app means five round trips over the network. One EVAL is a single round trip, and the five redis.calls inside it run right inside the Redis process, in memory, with no network involved. So the number of redis.calls in the script does not cost you any network at all.

A classic example is an atomic rate limiter:

1
2
3
4
5
6
7
8
9
-- KEYS[1] = the rate limit key, ARGV[1] = limit, ARGV[2] = ttl seconds
local current = redis.call("INCR", KEYS[1])
if current == 1 then
  redis.call("EXPIRE", KEYS[1], ARGV[2])
end
if current > tonumber(ARGV[1]) then
  return 0   -- blocked
end
return 1     -- allowed

The increment, the expiry and the check all happen together, with no chance of another request slipping in between.

One important caution, because the script blocks the single thread, a slow Lua script blocks the entire server. So keep scripts short and never put slow loops in them. (Newer Redis also has Functions via the FUNCTION command, which is basically a more structured evolution of the same idea.)

Making Redis reliable and big: replication, Sentinel, Cluster

A single Redis node is great until it dies, or until your data no longer fits in one machine’s memory. This is where the interesting architecture decisions come in.

Replication first

Redis supports replication, where one master node has one or more replicas that keep a copy of its data. Replication is asynchronous, the master keeps serving writes and streams the changes to the replicas.

Replicas are useful for two things, spreading out read traffic, and having a standby copy in case the master fails. But replication by itself does not give you automatic failover. If the master dies, someone (or something) still has to promote a replica. That is exactly the gap Sentinel fills.

Redis Sentinel: high availability

Sentinel is a separate process whose only job is to watch your Redis master and replicas and handle failover automatically.

1
2
3
4
5
6
        ┌───────────┐   monitors    ┌──────────┐
        │ Sentinel  │──────────────▶│  Master   │
        │ Sentinel  │──────────────▶│ Replica 1 │
        │ Sentinel  │──────────────▶│ Replica 2 │
        └───────────┘               └──────────┘
     (usually 3 sentinels for a proper quorum)

What it does:

• Continuously checks if the master is alive.
• If a quorum of sentinels agree the master is down, they elect a replica and promote it to master.
• They update the other replicas to follow the new master.
• Clients ask Sentinel “who is the master right now?”, so they always connect to the correct node even after a failover.

The key thing to understand about Sentinel, it gives you high availability, but not scaling. All your data still lives on one master. Sentinel just makes sure that if the master dies, a replica takes over quickly without a human waking up at 3am.

Redis Cluster: sharding and scaling

Sentinel keeps one dataset alive. Cluster is for when the dataset itself is too big for one machine, or the write throughput is more than one node can handle.

Cluster shards the data across multiple master nodes. It splits the keyspace into 16384 hash slots, and each master owns a range of those slots. The slot for a key is decided by a hash of the key, CRC16(key) % 16384.

1
2
3
4
5
6
      16384 hash slots, split across masters

   Master A            Master B            Master C
  slots 0-5460       slots 5461-10922    slots 10923-16383
     │                   │                   │
  Replica A           Replica B           Replica C

Some important points:

• There is no central proxy. Clients are “cluster aware”. If you send a key to the wrong node, that node replies with a MOVED redirect telling you the right one, and a good client caches this map.
• Each master usually has its own replica, so Cluster also gives you high availability, not just scaling.
• Multi key operations are limited. A command touching multiple keys only works if all those keys live in the same slot. You can force related keys into the same slot using a hash tag, like {user:123}:profile and {user:123}:sessions, where only the part inside {} is hashed.

The throughput this unlocks

This is worth saying plainly, because it is the reason Redis is reached for at scale. A single Redis node already does well over a hundred thousand operations per second. Once you shard with Cluster, throughput scales roughly linearly with the number of master nodes, so a real cluster comfortably handles millions, and up to the ten million operations per second range, of reads and writes.

A normal database simply cannot match this on its own. To get a relational database anywhere near that write throughput, you have to explicitly shard it yourself, split the data across many database servers, route every query to the right shard, and give up easy cross shard joins and transactions, all of which is a big, manual, error prone project. Redis was built around sharding from the start (those 16384 slots), so it hands you horizontal scale as a config choice rather than a re architecture. That combination, everything in memory plus near linear scaling with Cluster, is exactly why Redis sits in front of databases to soak up the traffic they cannot take directly. (If you want to see how this saves the database during a traffic spike, I wrote a separate post on request coalescing and cache stampedes.)

Sentinel or Cluster: how to actually decide

This is the decision people get stuck on, so let me make it simple. Ask yourself two questions.

1. Does my data fit comfortably in one node’s RAM (with room to grow)? 2. Can one node handle my write throughput?

• If yes to both, and you only want protection against the master dying, use Sentinel. It is simpler to run and reason about, and you keep all the multi key commands and transactions working normally, because everything is on one node.

• If no to either, meaning your data is too big for one machine or you need to scale writes horizontally, use Cluster. You get sharding across many masters, and each shard still has its own replica for HA.

A few practical notes:

• Cluster is more complex to operate and it restricts multi key operations to a single slot. Do not reach for it just because it sounds more “scalable”. Complexity has a cost.
• If you are on a managed service like AWS ElastiCache, GCP Memorystore or Redis Cloud, a lot of this Sentinel vs Cluster machinery is handled for you, and you mostly just pick “replication” or “cluster mode” in a dropdown. But it helps a lot to know what is happening underneath.
• A common path is to start with a single node, add replicas plus Sentinel when you need HA, and move to Cluster only when you genuinely outgrow one machine.

My honest rule of thumb, start with Sentinel, move to Cluster only when the data or the write load forces you to. Most applications never actually need Cluster.

Where Redis fits (common use cases)

Just so this does not stay too theoretical, here are the places I actually reach for Redis:

• Caching, the classic one, put it in front of a slow database or API.
• Session store, store user sessions with a TTL so they expire on their own.
• Rate limiting, using counters with expiry (the Lua example above).
• Queues and background jobs, using lists or streams.
• Leaderboards and ranking, using sorted sets.
• Pub/Sub and real time, for chat, notifications and live updates.
• Distributed locks, to coordinate work across many app servers.

Wrapping up

If I had to compress the whole thing into a few lines, it would be this.

Redis is fast because the data lives in memory and one thread runs everything, which also makes every command atomic for free. It uses an event loop to serve thousands of clients on that one thread. It deletes expired keys partly lazily on access and partly through a careful background job that is time boxed so it never blocks you. It stays durable with RDB snapshots and the AOF log. Lua scripts let you run multi step logic atomically right next to the data. And when a single node is not enough, Sentinel keeps it alive and Cluster makes it bigger.

Once you see it this way, Redis stops feeling like a magic fast cache and starts feeling like a very well thought out piece of engineering, where almost every feature traces back to that one decision of keeping the core single threaded.

If there is any part you want me to go deeper into, tell me in the comments and I can write a follow up.

A few years and a lot of maintenance pain later, I now measure a good day by how much code I did not write.

That shift comes down to two principles that sound almost too simple to matter, YAGNI and KISS. These are the first two things I look for in a code review, and also the hardest ones to actually follow, because both of them ask you to do less, and doing less feels like you are not trying hard enough.

Let’s break them down.

YAGNI, You Aren’t Gonna Need It

Build for the requirement in front of you, not for the future you are imagining.

YAGNI says, add functionality only when it is actually needed, not when you think you will need it later. Every “just in case” feature is code that you have to write, test, document and maintain forever, and often for a future that never even arrives.

You are probably violating YAGNI when…

• You catch yourself saying “we might need to support X later”, and you build X right now.
• You add a config flag that only ever has one value.
• You introduce an abstraction (an interface, a base class, a strategy) with exactly one implementation.
• There is a database column that nothing reads yet.

What it actually costs you

The cost is not just the time to build that speculative thing. It is everything that comes after it:

• The time to maintain and test the code that nobody uses.
• The extra complexity that every other engineer now has to read around.
• The bugs hiding inside code paths that never even run in production.

Speculative generality is like a loan you take against your future self, and the interest rate is brutal.

How to apply it

• Ask yourself one question, “is this needed for the story I am shipping today?” If the answer is no, do not build it.
• Delete the speculative code. git will remember it if you ever turn out to be right.
• Build the simple version now, and refactor when the real requirement actually lands.
• Follow the order, make it work, then make it right, then make it fast. Not the other way around.

YAGNI is not an excuse for

This is where people misuse it. YAGNI kills speculative features, it does not kill good engineering. It is never a reason to skip error handling, or ignore security, or write code that is genuinely impossible to extend later. Simplicity and sloppiness are two very different things.

Save this one for your next code review.

KISS, Keep It Simple, Stupid

Simplicity is not a lack of skill. It is actually the hardest skill.

If YAGNI is about how much you build, KISS is about how you build it. It says, the simplest solution that fully solves the problem is the best one. Write for the next person who reads the code, and nine times out of ten that person is you, at 2am, during an incident.

You are probably overcomplicating it when…

• A one-liner slowly grew into a four level deep “clever” chain that nobody can read.
• You reach for a design pattern before you actually have the problem that it solves.
• Your logic is a pyramid of deeply nested if/else blocks.
• You are reinventing something that the standard library already does.
• You need a comment to explain what the code does, not just why it does it.

How to apply it

• Write it the boring, obvious way first. You can always make it clever later (and usually you will not need to).
• Flatten the nesting with early returns and guard clauses.
• One function should do one job. If you cannot give it a clear name, it is doing too much.
• Prefer clear names over clever tricks.
• A simple test, if you cannot explain it in one sentence, simplify it.

The tension with DRY

KISS and DRY (Don’t Repeat Yourself) will sometimes fight each other, and when they do, KISS usually wins:

• A little bit of duplication is cheaper than the wrong abstraction.
• Do not couple two pieces of code together just because they look similar today. Code that looks similar but changes for different reasons is not really duplication, it just rhymes.

Clever code impresses in a PR. Simple code survives in production.

They are two sides of the same coin

If you notice, both principles are really about the same thing, restraint.

• YAGNI stops you from building things that you do not need.
• KISS stops you from over building the things that you do need.

Together they push you towards the same place, the smallest and clearest amount of code that solves the actual problem. And restraint, it turns out, is the senior skill. Anyone can add more code. Knowing what not to add, and being confident enough to delete that speculative abstraction that someone (maybe you) was proud of, is what separates code that ages well from code that slowly becomes a maintenance tax.

As Einstein (supposedly) said, “everything should be made as simple as possible, but not simpler.”

So the next time you are about to add a flag, or an interface, or a clever one liner, just pause for a second and ask, am I solving today’s problem, or a problem that I am only imagining?

The best code is the code that you did not have to write.

But there is one question that confused me for a long time. Your code runs inside a container, inside a pod, on some node in a cluster that you will probably never SSH into. So how does a single console.log("payment failed") actually reach Elasticsearch?

There is no magic here. It is a simple four hop pipeline:

1
2
3
4
5
6
7
Your app (stdout/stderr)
   ↓
Container runtime writes logs to a file on the node
   ↓
Fluentd (a DaemonSet running on every node) tails that file
   ↓
Fluentd ships the log to Elasticsearch  →  Kibana reads it

In my DevOps Essentials post I had promised that I will cover this DaemonSet based logging pattern in detail. So this is that post. Let’s walk through each hop slowly.

Hop 1: Your app only writes to stdout, nothing else

The most important rule in container logging is also the one that feels the most odd in the beginning:

Your application should not manage log files. It should only write to stdout and stderr.

No app.log. No log rotation logic. No shipping code inside your service. Just print.

1
2
3
// This is all your app needs to do.
console.log(JSON.stringify({ level: "info", msg: "payment processed", orderId: 123 }));
console.error(JSON.stringify({ level: "error", msg: "gateway timeout", orderId: 456 }));

This is one of the 12-Factor App principles, treat your logs as event streams. Your app is just a producer that throws events out into the void. It does not know where they finally end up, and that is exactly the point. Routing, storing and searching the logs is somebody else’s job.

Why does this matter? Because it keeps your app separate from your logging setup. You can swap Elasticsearch for Loki, or Fluentd for Fluent Bit, and you do not have to touch a single line of your application code.

One small tip, log as structured JSON instead of plain strings. {"level":"error","orderId":456} is very easy to filter in Kibana. "error processing order 456" will only force you into fragile regex later.

Hop 2: The container runtime writes stdout into a file

So your app printed something to stdout. Where does that stream go?

When a container writes to stdout/stderr, the container runtime (Docker, containerd, CRI-O) picks up those streams and writes them into a log file on the node’s filesystem. With Docker’s default json-file logging driver, every line becomes a JSON object like this:

1
{"log":"{\"level\":\"error\",\"msg\":\"gateway timeout\"}\n","stream":"stderr","time":"2026-07-18T10:22:01.5Z"}

Notice how the runtime wraps your actual log line inside its own envelope. It adds which stream it came from (stdout or stderr) and a time stamp.

On a Kubernetes node these files live in some predictable places:

1
2
/var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log
/var/log/containers/<pod>_<namespace>_<container>-<id>.log   # symlinks into the above

That /var/log/containers/ directory is the important one. Every container on that node dumps its logs there, and the filename itself contains the pod, namespace and container name. Keep this point in mind, it becomes useful in the next hop.

This is also exactly what kubectl logs <pod> reads. That command is not doing anything fancy, it is just showing you the content of these node local files. And that is also why kubectl logs loses the history when a pod gets rescheduled or the node’s logs rotate. These node local files are temporary. That is the whole reason we need to ship them somewhere permanent.

Hop 3: Fluentd, one collector per node running as a DaemonSet

Now the real question. Who reads all those /var/log/containers/*.log files and forwards them?

A log collector. The two names you will hear the most are Fluentd and its lighter cousin Fluent Bit. Their job is to tail the log files, parse them, enrich them and push them to some destination.

But here is the tricky part. How do you make sure the collector can see the logs of every container on every node? You cannot run it as a normal Deployment. A Deployment might place 3 replicas on a 10 node cluster, and then 7 nodes worth of logs will never get collected.

This is exactly the problem that a DaemonSet solves.

A DaemonSet makes sure that one copy of a pod runs on every node in the cluster. Add a new node and Kubernetes automatically schedules the collector on it. Remove a node and the collector goes away with it.

That is exactly the guarantee that logging needs. One Fluentd pod per node, and each one is responsible only for the logs of the containers on its own node.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
        Node A                    Node B                    Node C
 ┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
 │  app pods...      │     │  app pods...      │     │  app pods...      │
 │  /var/log/        │     │  /var/log/        │     │  /var/log/        │
 │  containers/*.log │     │  containers/*.log │     │  containers/*.log │
 │        ▲          │     │        ▲          │     │        ▲          │
 │   ┌────┴─────┐    │     │   ┌────┴─────┐    │     │   ┌────┴─────┐    │
 │   │ fluentd  │    │     │   │ fluentd  │    │     │   │ fluentd  │    │
 │   │(DaemonSet│    │     │   │(DaemonSet│    │     │   │(DaemonSet│    │
 │   │   pod)   │    │     │   │   pod)   │    │     │   │   pod)   │    │
 │   └────┬─────┘    │     │   └────┬─────┘    │     │   └────┬─────┘    │
 └────────┼──────────┘     └────────┼──────────┘     └────────┼──────────┘
          └──────────────────────────┼──────────────────────────┘
                                      ▼
                              Elasticsearch

But how does a Fluentd pod read files that belong to the node and not to itself? Through a hostPath volume. The node’s /var/log directory is mounted straight into the Fluentd container:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    matchLabels: { app: fluentd }
  template:
    metadata:
      labels: { app: fluentd }
    spec:
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1-elasticsearch
          env:
            - name: FLUENT_ELASTICSEARCH_HOST
              value: "elasticsearch.logging.svc.cluster.local"
            - name: FLUENT_ELASTICSEARCH_PORT
              value: "9200"
          volumeMounts:
            - name: varlog
              mountPath: /var/log            # the node's log dir, read-only
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

Once it can see the files, Fluentd does three things:

1. Tail, it follows every /var/log/containers/*.log file and picks up new lines as they get written (and it remembers its position, so that a restart does not resend everything again).
2. Enrich, remember the filename contains pod/namespace/container? Fluentd reads that and attaches Kubernetes metadata like pod_name, namespace, labels, node_name. So in Kibana you can filter by namespace: payments or pod: checkout-7f9c.
3. Buffer and forward, it batches the records into a buffer and ships them ahead. If Elasticsearch is slow or down, the buffer holds the logs and retries instead of dropping them.

A trimmed down Fluentd config for this flow looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 1. TAIL — read every container log file on this node
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kube.*
  <parse>
    @type json
  </parse>
</source>

# 2. ENRICH — add pod/namespace/labels from the K8s API
<filter kube.**>
  @type kubernetes_metadata
</filter>

# 3. FORWARD — batch and ship to Elasticsearch, buffering on failure
<match kube.**>
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
  logstash_format true          # daily indices: logstash-2026.07.18
  <buffer>
    flush_interval 5s
    retry_max_times 10
  </buffer>
</match>

Hop 4: Elasticsearch stores it and Kibana shows it

Fluentd pushes every enriched record to Elasticsearch over HTTP. Elasticsearch indexes it, usually into a daily index like logstash-2026.07.18 (that is what the logstash_format true line above does). Daily indices make retention very easy, to delete last month’s logs you just drop those indices.

Then Kibana connects to Elasticsearch and gives you the search UI. So now that console.log("payment failed") from Hop 1 is:

• searchable by full text (msg: "payment failed")
• filterable by the metadata that Fluentd added (namespace: payments AND level: error)
• and you can correlate it across every pod and node in the cluster, all in one place

The full journey, end to end:

1
2
3
4
5
6
7
8
9
console.log(...)                        ← your code, Hop 1
   ↓ stdout
/var/log/containers/checkout-*.log      ← container runtime, Hop 2
   ↓ tailed by
fluentd DaemonSet pod on that node      ← Hop 3
   ↓ enrich + buffer + HTTP POST
Elasticsearch  (index: logstash-2026.07.18)
   ↓
Kibana  ← you, searching at 2am during an incident

The “L” in ELK is not always Logstash

One naming confusion that is worth clearing up. ELK stands for Elasticsearch, Logstash, Kibana. But in Kubernetes you will rarely see Logstash sitting on the nodes. Logstash is heavy (it runs on the JVM) and it was built as a central processing pipeline, not as a per node agent.

So the modern per node collector is almost always Fluentd or Fluent Bit:

A very common production setup is to run Fluent Bit as the tiny per node DaemonSet and have it forward to a central Fluentd aggregator that does the heavy parsing before Elasticsearch. Same pipeline shape, just split into two tiers. (When people say “EFK stack”, that F is exactly this Fluentd/Fluent Bit swap.)

Some gotchas I have hit (so you don’t have to)

• Do not write logs to a file inside the container. If your app writes to /app/logs/app.log, the collector (which only watches stdout) will never see it, and that file dies with the pod. Just print to stdout.
• Multiline stack traces get split. Each line of a Java or Node stack trace is a separate stdout write, so the collector treats each line as a separate log record. You need a multiline parser to stitch them back into one event.
• Log rotation is real. The kubelet rotates the container log files (default around 10 MB). A good collector follows the rotation using its position file, a naive one can double send or miss lines around the rotation boundary.
• Backpressure will bite you. If Elasticsearch cannot keep up, the collector’s buffer fills up. Size your buffers and set retry limits, otherwise a slow Elasticsearch can OOM your logging pods or silently drop logs.
• kubectl logs is not your log store. It reads node local files that vanish on reschedule. It is good for a quick debug, but never for history or auditing. That is Elasticsearch’s job.

Wrapping up

The ELK stack feels like a black box until you trace the one path that actually matters:

Your app prints to stdout, the container runtime writes it to a file on the node, a Fluentd DaemonSet (one per node) tails that file, enriches it with pod metadata and ships it, Elasticsearch stores it, and Kibana lets you search it.

The nice part here is the separation of concerns. Your application does the simplest possible thing, it just prints. The DaemonSet makes sure there is a collector on every node with zero per app wiring. And swapping any piece of this pipeline never touches your code.

That is the whole trick really. No magic, just stdout, a file, and a collector that runs everywhere.

The strange part is that every one of these slow requests returned a normal HTTP 200. Nothing was actually failing. It was just slow, once in a while, and nobody could explain why.

This post is the story of chasing that 8 seconds all the way down to its root, and the surprisingly small fix at the end.

Clue #1: The spikes were suspiciously identical

The first thing I did was pull the raw durations of the slow requests, instead of trusting the aggregated p95. This is roughly what I saw:

1
2
3
4
5
6
7
8
8419ms
8299ms
8261ms
8260ms
8258ms
8249ms
8249ms
...

Look at how tightly these numbers are clustered. All of them are within around 200ms of 8.2 seconds.

That is a big hint. Real load is noisy. If the slowness was coming from a busy database, or a slow network hop, or CPU contention, you would see a spread like 2s, 5s, 11s, 800ms. When the latency clusters this tightly around a single constant, it is usually not load. It is a fixed delay sitting somewhere in the code path, like a timeout, a sleep, or a retry schedule.

So the question changed from “why is the database slow?” to “what in my code takes exactly 8.2 seconds?”

Clue #2: The time was not where I expected

I took one slow request and pulled every log line for it in order, with per statement timing. The actual job of the endpoint, its main query and the response, took only about 30 milliseconds. The rest of the 8.2 seconds was spent before that, inside what looked like a routine “find or create this record” step in an authentication middleware.

Here is the trace, cleaned up a bit:

1
2
3
4
5
6
7
8
9
10
12.839  START TRANSACTION
12.841  SELECT ... WHERE unique_col = X   -> no row found
12.855  INSERT ...                         (14ms)
14.044  INSERT ...                         (6ms)   <- ~1.19s later
15.545  INSERT ...                         (1ms)   <- ~1.50s later
17.797  INSERT ...                         (1ms)   <- ~2.25s later
21.175  INSERT ...                         (1ms)   <- ~3.38s later
21.177  SELECT ... WHERE unique_col = X   -> found it
21.178  COMMIT
21.215  Done — 8419ms

Two things jumped out at me:

1. There are five INSERT statements for what should have been a single insert.
2. Each INSERT itself finishes in 1 to 14ms. Nothing is blocked waiting on a lock. All the time is in the gaps between the inserts, and those gaps keep growing: around 1.2s, then 1.5s, then 2.25s, then 3.4s.

Growing gaps between the same operation is a classic sign of exponential backoff. Something was retrying that INSERT five times, sleeping a little longer each time, before finally giving up and reading the row instead.

The code: an innocent looking findOrCreate

The middleware creates a local record for a user the first time it sees them. In Sequelize (a popular Node.js ORM), that is a one liner:

1
2
3
4
const [user] = await User.findOrCreate({
  where: { external_id: id },
  defaults: { name, email, external_id: id },
});

findOrCreate does exactly what the name says. It runs a SELECT using the where clause, and if nothing is found, it does an INSERT with the defaults. Simple enough. So why five inserts?

The root cause: a race, and a retry policy that could not tell the difference

Here is what was actually happening.

When a brand new user arrives for the first time, the client fires several requests at once (a typical bootstrap, fetch status, fetch a token, fetch some config), and all of them carry the same new user id. Each of these requests independently reaches that findOrCreate.

And now the classic race plays out:

1. Two requests run the SELECT at almost the same time. Both of them see no row.
2. Both go ahead and INSERT.
3. One transaction commits first and wins, so the row now exists.
4. The other request’s INSERT violates the unique constraint on the column, and the database rejects it with a duplicate key error (ER_DUP_ENTRY in MySQL, which Sequelize surfaces as a UniqueConstraintError).

This race is completely normal and expected. It is why the unique constraint is there in the first place. And findOrCreate already handles it correctly. On a unique constraint error it catches the exception and does a second SELECT to return the row that the winner just created.

But there was a retry policy sitting underneath all of this. Sequelize lets you configure automatic query retries (through the retry option, powered by retry-as-promised), usually added to survive transient failures like deadlocks or dropped connections. Sequelize even ships a sensible, conservative default, retry on exactly one known transient error:

1
2
// Sequelize's built-in default
retry: { max: 5, match: ["SQLITE_BUSY: database is locked"] }

So how did a duplicate key error, which is not in that list, end up getting retried? This is the part that surprised me, and it comes down to two behaviours combining together.

First, the config had overridden retry to tune the backoff, but without specifying match:

1
2
3
4
5
6
7
// somewhere in the DB config
retry: {
  max: 5,
  backoffBase: 1000,
  backoffExponent: 1.5,
  // note: no `match` key
}

Sequelize merges these options with a shallow spread, so this custom block does not extend the default, it replaces it completely. The curated match: ["SQLITE_BUSY"] is silently gone. You can actually watch it happen:

1
2
3
4
5
new Sequelize(db, { dialect: "mysql" }).options.retry;
//=> { max: 5, match: ["SQLITE_BUSY: database is locked"] }

new Sequelize(db, { dialect: "mysql", retry: { max: 5, backoffBase: 1000 } }).options.retry;
//=> { max: 5, backoffBase: 1000 }   // <-- no `match` anymore

Second, and this is the real trap, an empty or missing match does not mean “retry nothing”. In retry-as-promised it means the exact opposite:

1
2
// retry-as-promised
shouldRetry = options.match.length === 0 || options.match.some(m => matches(m, err));

match.length === 0 short circuits to true. So no filter actually means retry on every error. Leaving match off is the most aggressive setting, not the safest one.

Put it all together. The backoff override wiped out the default allowlist, the now empty match meant “retry everything”, and so the losing INSERT’s duplicate key error was treated as retryable. The retry layer re-ran the identical INSERT, with backoff, five times, even though nobody ever listed that error as retryable.

And here is the thing that makes this a real bug and not just slowness. A duplicate key error is permanent. The winning row is already committed. Re-running the exact same INSERT will fail in exactly the same way, every single time, forever. There is no version of “try again” that can ever succeed. So the retry policy was spending 8 seconds sleeping between attempts that were guaranteed to fail, and only after exhausting all five did the error finally reach findOrCreate’s catch block, which then did the one thing that actually works, read the row.

The delays match the config almost exactly:

And there is the 8.2 seconds.

An overlooked side effect: connection starvation

The latency was the visible symptom, but there is a nastier problem hiding underneath it.

That whole 8.2s happens inside an open transaction, which holds a connection checked out from the pool the entire time. Connection pools are small (5, 10, maybe 20 per instance). If a burst of new users arrives together, several connections can each get pinned for 8 seconds doing nothing except sleeping between doomed retries. Once the pool is exhausted, even unrelated requests start queuing for a connection. So one user’s harmless race can quietly degrade the latency for everyone on that instance.

Non-blocking I/O saves the event loop here (the backoff is a setTimeout, not a busy wait, so the CPU stays free), but the connection is still held the whole time. “It’s async so it’s fine” does not cover the resources you are holding across the await.

The fix

The correct recovery for a findOrCreate race is not “retry the insert”, it is “read the row that someone else just created”. That logic already exists in findOrCreate’s catch block. All I had to do was stop the retry layer from getting in its way:

1
2
3
4
5
6
7
8
const [user] = await User.findOrCreate({
  where: { external_id: id },
  defaults: { name, email, external_id: id },
  // A duplicate-key error here is the *expected* outcome of a race, not a
  // transient fault. Don't retry it — fall straight through to the built-in
  // findOne fallback instead of burning the backoff budget on a doomed INSERT.
  retry: { max: 0 },
});

I verified this with a tiny standalone script using the same libraries. I stubbed a create that always throws a UniqueConstraintError, and then timed it under the old policy versus the fix:

1
2
BEFORE (retry max:5)   attempts=5   elapsed=8130.6ms
AFTER  (retry max:0)   attempts=1   elapsed=0.1ms

So it went from around 8,130ms to around 0.1ms per losing request. Same outcome (the error still propagates to the read fallback), just without all the pointless sleeping.

Should you keep one retry?

Tempting, but no. A single retry (max: 1) still costs a full ~1s backoff sleep and still fails, because the duplicate is permanent. One retry of a non retryable error is pure waste. The real value of retries lives entirely with transient errors, which brings me to the deeper fix.

The deeper fix

Disabling retry at this one call site is the fast, local patch. The real root cause is broader. The retry policy had no match allowlist at all, so it was retrying every error. The proper fix is to give it an explicit match that lists only genuinely retryable failures, like deadlocks, lock wait timeouts and connection resets, and never deterministic ones like unique constraint violations. That fixes every findOrCreate and create in the codebase at once, not just the one I happened to be looking at. (I also reported the silent default wiping behaviour upstream, since “override the backoff, accidentally retry everything” is a sharp edge worth flagging.)

Some lessons worth keeping

1. Tightly clustered latency is a fixed delay, not load. If your slow requests all land within a few percent of the same number, stop staring at load graphs and start looking for a timeout, a sleep, or a retry schedule in the code.

2. Measure where the time goes before theorising about why. Per statement timing turned “the database is slow” into “we sleep for 8 seconds between five inserts” in about two minutes. Once you can say the second sentence, the fix is almost obvious.

3. Not every error is retryable, and check what “no filter” actually means. Retries are meant for transient failures. Retrying a deterministic error like a duplicate key, a validation failure or a 400 can never succeed, it just multiplies the cost of failing. And always know your retry library’s default. In more than one of them, an empty match list means “retry everything”, not “retry nothing”. The safe posture is an explicit allowlist of transient errors, deny by default.

4. A findOrCreate race is normal, handle it by reading, not by rewriting. The unique constraint is doing its job when it rejects the second insert. The right response is to go and fetch the row that the winner created, which most ORMs already do for you.

5. Watch what you hold across an await. The event loop being free does not mean nothing is blocked. A connection, a lock, or a transaction that is pinned for 8 seconds is a scalability bug even when your CPU usage looks perfectly fine.

The final diff was one line. Finding which line took a lot longer, and honestly that is almost always the shape of a good debugging story.

But the reality is simple:

If your code runs on the internet, you are already part of DevOps.

Every software engineer — backend, frontend, or mobile — should understand how their code goes from a Git commit to running in production.

You don’t need to be a Kubernetes expert. But understanding the fundamentals of networking, containers, deployments, and monitoring will make you a significantly better engineer.

In this post, I’ll break down the DevOps concepts every engineer should know — with practical examples.

1. The Real Basics: How Networking Works

Before we talk about deploying apps, we need to understand how they talk to each other. At the foundation is the TCP/IP (Transmission Control Protocol / Internet Protocol) model.

• IP Addresses: Think of this as the street address of a server.
• Ports: If the IP is the street address, the port is the specific apartment number (e.g., Port 80 for HTTP, 443 for HTTPS, 5432 for PostgreSQL).
• TCP: A reliable protocol that ensures data packets arrive correctly and in order. It uses a “three-way handshake” (SYN, SYN-ACK, ACK) to establish a connection before sending data.
• DNS: The phonebook of the internet. It maps human-readable domains (like google.com) to IP addresses.

Understanding these basics will save you hours of debugging when an API call fails with a “Connection Refused” or “CORS” error.

2. Docker Basics

“It works on my machine!” are famous last words in software engineering. Enter Docker. Docker solves the “works on my machine” problem by packaging applications with all their dependencies.

Key Concepts:

• Image: A read-only template containing your code, runtime, libraries, and environment variables. Like a blueprint for a house.
• Container: A running instance of an image. Like the actual built house you can live in.

Sample Dockerfile

Here’s a simple example of containerizing a Node.js application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# 1. Use the official Node.js image as a base
FROM node:18-alpine

# 2. Set the working directory inside the container
WORKDIR /app

# 3. Copy package.json and install dependencies
COPY package*.json ./
RUN npm install

# 4. Copy the rest of the application code
COPY . .

# 5. Build the application (if needed) and Expose the port
EXPOSE 3000

# 6. Command to start the app
CMD ["npm", "start"]

💡 Very High level of docker run command

So, when we run the command docker run my-app-image, Docker performs the following steps:

1. Pull the image (if not available locally)
2. Create a container from the image
3. Start the container –> Here Actually npm start is executed.

So effectively:

Docker Image → becomes → Running Container

3. Deployments Before Kubernetes

Kubernetes is great, but it’s overkill for many projects. Before jumping to K8s, it’s crucial to understand simpler deployment methods: the Single Shared Machine model.

Platforms like EC2 (AWS), Heroku, or Railway allow you to deploy your app onto a virtual machine (VM).

• Heroku / Railway: Abstract away the underlying servers (PaaS). You push code, and they build and run the Docker container for you effortlessly.
• EC2: You get a raw virtual server (IaaS). You SSH into it, install Docker, pull your image, and run it.

This model is perfect for MVPs and small-to-medium side projects before scaling horizontally.

ssh ubuntu@server-ip

sudo apt install docker.io

docker run -d -p 80:3000 myapp

User
  ↓
Internet
  ↓
EC2 Instance
  ↓
Docker Container
  ↓
Application

4. Enter Kubernetes (K8s)

When your application scales past a few VMs, managing them manually becomes a nightmare. Kubernetes is a container orchestration tool that automates deploying, scaling, and managing containerized apps.

Kubernetes Terminology

• Cluster: A cluster is simply a group of machines (i.e nodes) working together.
• Node: A physical or virtual server (e.g., an EC2 instance or a machine) that runs your containers.

  1
  2
  3
  4
  Cluster
   ├── Node 1 (EC2)
   ├── Node 2 (EC2)
   └── Node 3 (EC2)
  

• Pod: The smallest deployable unit in K8s. A Pod usually contains one container (sometimes a few tightly coupled ones).

  In most cases: 1 Pod = 1 Container

  Think of a Pod as:

  A wrapper around your Docker container.

The simplest way to picture it is a chain, where each layer only manages the one below it:

1
Deployment  ── manages ──►  ReplicaSet  ── manages ──►  Pods

• ReplicaSet: its only job is to keep a fixed number of identical Pods running. You say “I want 3”, and if a Pod crashes or gets deleted, the ReplicaSet immediately creates a new one to get back to 3. That is the whole job. A ReplicaSet knows nothing about versions or updates.

• Deployment: this sits one level above, and it is the thing you actually create in practice. Here is the key point that is easy to miss, a Deployment never creates Pods directly. It only watches your manifest. When something in it changes, like a new image version (v1 to v2) or a different replica count, the Deployment creates a new ReplicaSet, and that ReplicaSet is what actually creates the new Pods. During an update it scales the new ReplicaSet up while scaling the old one down (a rolling update, so no downtime), and it keeps the old ReplicaSet around at 0 Pods so you can roll back instantly, it just scales the old one back up.

  So the easiest way to remember the difference:

  • ReplicaSet = keep N identical Pods alive.
  • Deployment = manage ReplicaSets so you can update and roll back safely.

  You almost always create a Deployment, and it creates and manages the ReplicaSet for you. You rarely make a ReplicaSet by hand. Here is what it looks like right after an update from v1 to v2:

  1
  2
  3
  Deployment (my-app, now on v2)
    ├── ReplicaSet v2  →  3 Pods   (current)
    └── ReplicaSet v1  →  0 Pods   (kept, ready for instant rollback)
  

• DaemonSet: Ensures that every Node runs a copy of a specific Pod (often used for logging or monitoring agents). (Will cover this in detail very soon in some blog.)

• Workloads: A general term for applications running on K8s (Deployments, StatefulSets, DaemonSets).

  This can be confusing initially because Workload is not an actual Kubernetes resource.

  It is simply a category used by Kubernetes to describe objects that run applications, such as Deployments, StatefulSets, and DaemonSets.

  So, when someone says “workloads”, they mean Deployments, StatefulSets, DaemonSets, etc.

  1
  2
  3
  4
   Workloads
   ├── Deployment
   ├── StatefulSet
   └── DaemonSet
  

• ConfigMap / Secret: Ways to pass environment variables and sensitive data to your Pods without hardcoding them in the image.
• Service: A stable network endpoint that exposes Pods. Since Pods are ephemeral (ip changes every time they are recreated) and can be recreated at any time, Services provide a consistent way to access them.

Official Kubernetes Architecture Diagram

A Sample K8s Deployment Manifest

K8s uses YAML manifests to declare the desired state of the system:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: apps/v1
# Kind in k8s YAML is like the type of resource
kind: Deployment  # This is a workload resource
metadata:
  name: frontend-deployment
spec:
  replicas: 3  # This is the ReplicaSet
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend-container
        image: my-repo/frontend:v1.0.0
        ports:
        - containerPort: 80

1
2
3
4
5
6
7
8
9
10
11
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  selector:
    app: frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Often Confusing Differences

    Cluster  
    ├── Node 1 (EC2 instance)  
    ├── Node 2 (EC2 instance)  
    └── Node 3 (EC2 instance)

Deployment → manages → ReplicaSet → manages → Pods

5. Which Tools to Use for K8s / EKS?

Managing K8s via CLI (kubectl) can be overwhelming to visualize. There are some incredible tools to help engineers manage clusters effectively:

• Lens: Often called the “Kubernetes IDE”. It provides a beautiful, native desktop UI to view Pods, logs, configurations, and cluster metrics instantly. (Tip: If you’re using Lens, check out its intuitive features for port-forwarding and inspecting secrets.)

  

• k9s: A terminal-based UI to interact with your clusters. If you prefer to never leave your terminal but want a visual dashboard, k9s is unmatched for speed.

6. Infrastructure as Code: Terraform

Terraform allows you to write code to provision infrastructure (like AWS EKS clusters, databases, and network firewalls) instead of clicking through web consoles. It uses HCL (HashiCorp Configuration Language) to declare what resources you want, and Terraform figures out how to create, update, or delete them to match your code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
provider "aws" {
  region = "ap-south-1"
}

resource "aws_s3_bucket" "app_bucket" {
  bucket = "my-app-storage"
}

resource "aws_db_instance" "app_db" {
  identifier = "app-db"

  engine = "postgres"
  instance_class = "db.t3.micro"

  allocated_storage = 20
  username = "admin"
  password = "password123"

  skip_final_snapshot = true
}

Run:

1
2
3
terraform init
terraform plan # shows what will be created
terraform apply # creates the resources

terraform-modules/
   ├── s3-module
   ├── rds-module
   └── eks-module

module "app_s3" {
  source = "../modules/s3"
  bucket_name = "team-storage"
}

7. Continuous Integration (CI)

Continuous Integration (CI) is the practice of automatically building and testing your code whenever changes are pushed to a repository.

Instead of manually running builds and tests, CI systems automate the process and provide quick feedback to developers.

Typical CI flow:

1
2
3
4
5
6
7
8
9
10
11
Developer pushes code → GitHub
↓
CI pipeline triggered
↓
Install dependencies
↓
Run tests
↓
Build Docker image
↓
Push image to container registry

One of the most widely used CI tools is Jenkins.

Jenkins runs pipelines defined using a Groovy-based DSL called a Jenkinsfile.

This allows you to version your build pipeline alongside your application code.

pipeline {
  agent any
  stages {
    stage('Checkout Code') {
      steps {
        git 'https://github.com/org/project.git'
      }
    }

    stage('Install Dependencies') {
      steps {
        sh 'npm install'
      }
    }

    stage('Run Tests') {
      steps {
        sh 'npm test'
      }
    }

    stage('Build Docker Image') {
      steps {
        sh 'docker build -t myapp:${BUILD_NUMBER} .'
      }
    }

    stage('Push Image') {
      steps {
        sh 'docker push myrepo/myapp:${BUILD_NUMBER}'
      }
    }
  }
}

8. Helm, FluxCD, and GitOps Automation

Once you have K8s and Terraform, how do you manage updates easily for the whole team?

Helm

Helm is the package manager for Kubernetes. Instead of writing dozens of YAML files (like the deployment manifest above), Helm uses Chart templates. You can install an entire database with a single command: helm install my-db bitnami/postgresql.

deployment-dev.yaml
deployment-staging.yaml
deployment-prod.yaml

replicaCount: 3
image: my-app:v1

helm install my-app ./chart

FluxCD & GitOps

In the past, engineers manually ran deployment scripts. Today, we use GitOps. Tools like FluxCD (or ArgoCD) run inside your K8s cluster and constantly monitor your Git repository. When you merge a PR to GitHub, FluxCD instantly updates the K8s cluster to match the new code.

Self-Serve Architecture for Teams

In larger teams, infrastructure should not become a bottleneck where developers constantly depend on DevOps engineers for deployments.

A common approach is to adopt a GitOps-driven self-serve architecture.

Typical repository structure:

1
2
3
4
5
6
7
repo/
├── terraform-resources/
│ ├── s3
│ ├── networking
│ └── rds
└── k8s-resources/
    ├── helm-charts

Infrastructure resources such as clusters, networks, and databases are managed using Terraform.

Application-level deployments are managed using Kubernetes manifests or Helm charts.

Developers typically work with long-lived branches such as:

• staging
• regression
• production

When changes are merged into these branches, FluxCD continuously monitors the repository.

If Flux detects changes in Kubernetes manifests or Helm charts, it automatically synchronizes those changes to the Kubernetes cluster.

This ensures:

• Git becomes the single source of truth
• Deployments are automated
• Infrastructure changes remain auditable

9. Observability: How the ELK Stack Works

Deploying code is only half the battle. Knowing what it’s doing in production is the other half. The ELK Stack is the industry standard for centralized logging:

1. E - Elasticsearch: A powerful search engine that stores your logs.
2. L - Logstash (or fluentd/fluentbit): The data pipeline that collects logs from your K8s Pods, parses them, and sends them to Elasticsearch.
3. K - Kibana: The UI dashboard where you can filter, search, and visualize your application logs.

When your app throws error 500s, you go to Kibana to read the exact stack trace.

10. Metrics: Prometheus, Grafana & Time Series Databases

While ELK handles logs (text), Prometheus and Grafana handle metrics (numbers over time).

• Time Series Database (TSDB): A database optimized for storing data timestamp by timestamp. Perfect for “CPU usage per second”.
• Prometheus: It scrapes metrics from your applications and stores them in its TSDB.
• Grafana: Plugs into Prometheus and visualizes the data via beautiful charts and gauges.

How to create dashboards: In Grafana, you write PromQL (Prometheus Query Language) queries (e.g., rate(http_requests_total[5m])) to pull data and put it on a graph. You can then set alerts to ping your team’s Slack if CPU usage exceeds 90%.

How Metrics Collection Works

The monitoring flow typically looks like this:

1
2
3
4
5
6
7
8
9
Application 
   ↓
/metrics --> Prometheus scrapes metrics 
   ↓
Metrics stored in TSDB 
   ↓
Grafana queries Prometheus 
   ↓
Dashboards visualize data

Step 1 - Expose Metrics

Applications expose a /metrics endpoint.

Example output:

1
2
3
http_requests_total 10234
http_request_duration_seconds 0.34
cpu_usage_percent 45

Step 2 - Prometheus Scrapes Metrics

1
2
3
4
5
6
7
8
Prometheus periodically scrapes (pulls) metrics from the `/metrics` endpoint.

Prometheus configuration:

scrape_configs:
  - job_name: 'myapp'
    static_configs:
      - targets: ['myapp:8080']

Prometheus stores:

• recent data in memory
• long-term data in a TSDB (Time Series Database)

Example TSDB systems

Common time series storage systems include:

• Prometheus TSDB
• VictoriaMetrics
• Thanos
• Cortex

These systems allow long-term storage and horizontal scaling of metrics.

Step 3 - Grafana Visualizes Data

1
2
3
4
5
6
7
8
9
10
Grafana queries Prometheus using PromQL:

rate(http_requests_total[5m])

Creates dashboards with:

- Line charts
- Gauges
- Heatmaps
- Alerting rules

11. Advanced Deployments: Canary vs. Blue/Green

“Push and pray” is dangerous. Modern deployment strategies minimize downtime and risk:

Blue/Green Deployment

Blue-Green deployment uses two identical environments to release new versions of an application safely.

• Blue → the currently live production environment
• Green → the new environment where the updated version is deployed

Both environments run the same infrastructure, but only one receives user traffic at a time.

Deployment Sequence

1
2
3
4
5
6
7
8
Step 1 — Current state

Users
  ↓
Blue (v1 - live)

Green (idle)
Step 2 — Deploy new version

1
2
3
4
5
Users
  ↓
Blue (v1 - live)

Green (v2 deployed but not receiving traffic)

1
2
3
4
Step 3 — Test Green environment

Run smoke tests, health checks, and integration tests
against the Green environment to ensure it is stable.

1
2
3
4
5
6
7
Step 4 — Switch traffic

Users
  ↓
Green (v2 - now live)

Blue (v1 - standby)

Once the new version is verified, the load balancer is switched so all traffic moves from Blue → Green.

Rollback

If any issue is detected after deployment, rollback is immediate.

1
2
3
4
5
Users
  ↓
Blue (v1 restored)

Green (v2 disabled)

Canary Deployment

• You route a small percentage of traffic (e.g., 5%) to the new version (the “Canary”).
• The remaining 95% stay on the stable version.
• You monitor the error rates and metrics on the Canary. If stable, you gradually increase traffic (10%, 25%, 50%, 100%).
• Benefit: Lowest risk. Bugs only impact a small subset of users before being caught.

12. CDN and Edge Infrastructure

When users access your application, every request doesn’t need to hit your origin server.

This is where a CDN (Content Delivery Network) comes into play.

A CDN is a globally distributed network of servers that caches and delivers content closer to users.

Instead of:

User → Origin Server

It becomes:

User → CDN Edge Server → Origin Server

If the content is cached at the edge, the request never reaches your origin server.

Benefits of using a CDN:

• Faster content delivery
• Reduced load on your servers
• Built-in DDoS protection
• Edge caching of static assets
• TLS termination
• Web Application Firewall (WAF)

One of the most popular CDN providers is Cloudflare.

Conclusion

DevOps isn’t a single person’s job—it’s a culture and a set of practices. By understanding these concepts—networking, containerization, orchestration, and observability—you bridge the gap between “code completing” and “code delivering value to users reliably.”

Next time you see a Dockerfile or a .yaml manifest in your repository, you’ll know exactly what’s going on!

This can lead to:

• Corrupted API responses
• Sensitive information leaking into server logs
• Even other users data or secrets accidentally being exposed
• Hours of debugging headaches

Let’s break this down with an example and then see how to fix it.

🚨 The Danger of Buffer.allocUnsafe

Node.js uses Buffer.allocUnsafe(size) internally in places like zlib.gunzip.
This is fast, but it does not clear memory — meaning it can reuse old memory blocks.

👉 If you don’t overwrite the full buffer, stale data remains and can “leak” into your output.

Demo: Stale Memory Leak

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// ❌ Demo: Showing stale memory leak
// Step 1: Write a large distinctive pattern
const secret = Buffer.from("🔥🔥🔥SUPER_SECRET🔥🔥🔥".repeat(100));
console.log("Original Secret:", secret.toString().slice(0, 50) + "...");

// Step 2: Force memory pressure
let garbage = [];
for (let i = 0; i < 100000; i++) {
  garbage.push(Buffer.from("garbage"));
}
garbage = null;

// Step 3: Unsafe allocation (uninitialized memory)
const leaky = Buffer.allocUnsafe(secret.length);
console.log("Leaky Buffer:", leaky.toString().slice(0, 50) + "...");

Output:

1
2
Buffer 1: SECRET_DATA
Buffer 2 (dirty): 0�c0▒▒▒

That gibberish itself is proof that Node.js did not zero out the memory — it’s showing raw, uninitialized bytes left over from previous allocations.

Even though we allocated a new buffer, Node.js reused the same memory block, causing old data (or arbitrary garbage) to resurface.

This isn’t a problem if you fully overwrite buf2. But if you only partially fill it (for example, due to incorrect stream handling), stale data will remain and may leak into your output.

🌐 Real-World Problem: Handling API Response Streams

Many third-party APIs return data as a stream (for example, https.get, AWS SDKs, Google APIs).

If you mishandle this stream:

• Not consuming all chunks
• Not awaiting asynchronous reads
• Concatenating buffers incorrectly

… you risk ending up with partial data + old memory content.

❌ Wrong Way (Naive Handling)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import https from "https";

function fetchData(url) {
  return new Promise((resolve, reject) => {
    https.get(url, (res) => {
      let data = "";
      res.on("data", (chunk) => {
        data += chunk; // ❌ mixing Buffers with strings!
      });
      res.on("end", () => resolve(data));
      res.on("error", reject);
    });
  });
}

This approach is unsafe because:

• Buffers are being coerced into strings
• Multibyte characters may get split across chunks
• Partial old data can sneak in when decoding

✅ Correct Way: Consume the Stream with for await...of

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import https from "https";

async function fetchData(url) {
  return new Promise((resolve, reject) => {
    https.get(url, async (res) => {
      try {
        const chunks = [];
        for await (const chunk of res) {
          chunks.push(chunk); // always handle Buffers
        }
        const buffer = Buffer.concat(chunks);
        resolve(buffer.toString("utf8")); // safe decoding
      } catch (err) {
        reject(err);
      }
    });
  });
}

Here we:

• Use for await...of to consume the entire stream
• Store raw Buffers in an array
• Combine them with Buffer.concat(), which correctly allocates fresh memory

🎭 Streams with Compression (zlib / gunzip)

The risk is higher with compressed streams, since zlib.gunzip uses Buffer.allocUnsafe.

Safe Usage

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import { createGunzip } from "zlib";
import { pipeline } from "stream/promises";
import https from "https";

async function fetchAndUnzip(url) {
  return new Promise((resolve, reject) => {
    https.get(url, async (res) => {
      try {
        const chunks = [];
        const gunzip = createGunzip();

        await pipeline(res, gunzip);

        for await (const chunk of gunzip) {
          chunks.push(chunk);
        }

        const buffer = Buffer.concat(chunks);
        resolve(buffer.toString("utf8"));
      } catch (err) {
        reject(err);
      }
    });
  });
}

Here, we pipe the response through gunzip, and again collect all buffers safely.

✅ Best Practices to Avoid Buffer Leaks

• Always consume the stream fully (for await...of or pipeline)
• Never mix Buffers with strings mid-stream
• Use Buffer.concat(chunks) — don’t manually slice or reuse old buffers
• For sensitive data, prefer Buffer.alloc() (zero-filled) instead of Buffer.allocUnsafe()
• Be cautious when decompressing — partial/incomplete streams can expose old memory contents

📝 Final Thoughts

Streams in Node.js are powerful, but subtle mistakes can cause old memory contents to reappear in your outputs or logs.

• Always handle streams properly
• Be careful with compression/decompression
• Remember that Buffer.allocUnsafe is unsafe for a reason

💸 Previous Setup: Heroku + PostgreSQL

Until recently, I was using Heroku’s hobby plan setup:

• $7/month: Heroku Dyno (limited RAM, not ideal for traffic spikes)
• $5/month: Heroku Postgres Hobby Dev

Total: $12/month

While Heroku is beginner-friendly, the hobby dynos provide low memory and have limitations under heavy traffic. Performance wasn’t consistent during peak hours, and there’s not much clarity about how compute resources are allocated behind the scenes.

🚀 Step 1: Migrated PostgreSQL to Supabase

Supabase offers a generous free tier for PostgreSQL databases, and since calcont.in only uses the DB for:

• User sign-ups (~10/month)
• Contact form submissions (~50 in total)

It made perfect sense to migrate to Supabase. Minimal usage, no cost, and easy setup.

🔄 Step 2: Deployment Experiment - Google Cloud Serverless

Next, I explored Google Cloud’s serverless options by estimating CPU and request costs. While it looked affordable on paper, real-world deployment revealed unexpectedly high costs.

I quickly moved on.

💡 Step 3: Found the Sweet Spot with Railway

I discovered Railway and tried their $5/month hobby plan, which includes:

• 8 GB RAM
• 8 vCPUs
• Supports Dockerfile-based deployment
• Pay-as-you-go pricing model

Actual cost for my usage? Just $2.50–$3/month!
That’s ~$10/month saved, and the platform offers:

• Better resource allocation
• Fewer crashes or slowdowns
• Predictable pricing with usage caps to avoid surprise bills

Note: While testing Railway, I kept my Heroku setup running in parallel as a fallback. This gave me confidence to monitor traffic, analyze real-world pricing, and make the switch only when I was sure it worked smoothly.

🔐 Bonus: Securing the Admin Panel with Cloudflare One

To protect sensitive parts of my app (like /admin), I set up Cloudflare One Access, which:

• Prevents public exposure of admin endpoints
• Offers Zero Trust protection for free (up to 50 users)

No need to write middleware or complex auth layers!

🧠 Optimization: Removed External Cache Layer

In the Heroku setup, I was using memcachier for caching some operation. But in the new setup:

• As, only one feature uses caching. So, Maintaining an external cache service didn’t make sense
• So I switched to Django’s in-memory RAM caching, which is enough and free

📊 Final Comparison

📦 GitHub Reference

You can check the actual PR here that contains the migration and deployment updates:
🔗 calcont.in/pull/115

🧾 TL;DR

• Migrated DB to Supabase (free)
• Moved hosting to Railway ($2.5–$3/month)
• Secured admin route with Cloudflare One
• Removed unnecessary caching layer
• Saved $120/year while improving reliability

If you’re hosting a personal project and not doing heavy DB operations, you can save a lot by evaluating these alternatives. Don’t pay for what you don’t need.

Have questions or want to share your own infra cost-cutting tips? Drop a comment or reach out via GitHub!