Why the simple approaches don’t work

There are two brute-force ways to take a snapshot, and neither really works for us:

1. Hold a global lock while writing to disk. Every client write waits until the snapshot is done, which could take seconds.
2. Deep-copy the whole map in memory, and let a background thread write the copy to disk. Now we’re using double the memory. And usually only a small fraction of the keys change while the snapshot is being written, so most of that copying is wasted.

So what ClickHouse Keeper needs is a data structure with three properties:

1. Constant-time lookups, for the clients reading the data.
2. No locking, or if that’s not possible, locking only for a very short time.
3. Copy-on-write only for the keys that change. That way, the extra memory we use is proportional to the number of writes during the snapshot.

Why not a plain hash map?

The obvious question is: why can’t we do this with a regular hash map, like std::unordered_map in C++?

The problem is that as writes keep arriving during a snapshot, the hash map will eventually need to grow and rehash. Rehashing moves keys into different buckets, so the order of the keys gets scrambled. If the snapshot thread is walking through the buckets while that happens, it can miss some keys and write others twice.

We need a structure that gives us constant-time lookups, but also a stable, predictable order to walk through.

A hash map and a doubly linked list

To get both, we can combine two data structures we already know:

1. A hash map for constant-time lookups.
2. A doubly linked list that keeps a stable insertion order.

The key-value data actually lives in the nodes of the linked list. The hash map stores the keys, and each key’s value is a pointer to that key’s node in the list.

Let’s look at how this behaves in two cases: during normal operation, when no snapshot is running, and while a snapshot is being taken.

In the source code

ClickHouse is an open-source OLAP database. It has its own implementation of ZooKeeper, called ClickHouse Keeper, and that’s what we’ve been looking at. The code below is from the v26.2.1.1139-stable release.

using List = std::list<ListElem>;
using Mapped = typename List::iterator;
using IndexMap = HashMap<std::string_view, Mapped>;

List list;
IndexMap map;
bool snapshot_mode{false};
/// Allows to avoid additional copies in updateValue function
size_t current_version{0};
size_t snapshot_up_to_version{0};

struct
{
    uint64_t active_in_map : 1;
    uint64_t free_key : 1;
    uint64_t version : 62;
} node_metadata{false, false, 0};

{ /// lock storage for a short period time to turn on "snapshot mode". After that we can read consistent storage state without locking.
    LockGuardWithStats<false> lock(storage_mutex);
    snapshot_task.snapshot = std::make_shared<KeeperStorageSnapshot<Storage>>(storage.get(), snapshot_meta_copy, getClusterConfig());
}

auto [size, ver] = storage->container.snapshotSizeWithVersion();
snapshot_container_size = size;          // this is N
storage->enableSnapshotMode(ver);        // the flag, and the cut-off version
begin = storage->getSnapshotIteratorBegin();

if (snapshot_mode)
{
    /// We in snapshot mode but updating some node which is already more
    /// fresh than snapshot distance. So it will not participate in
    /// snapshot and we don't need to copy it.
    if (list_itr->getVersion() <= snapshot_up_to_version)
    {
        auto elem_copy = list_itr->copyFromSnapshotNode();
        ...
        list_itr->setInactiveInMap();
        snapshot_invalid_iters.push_back(list_itr);
        updater(elem_copy.value);

        elem_copy.setVersion(current_version);
        auto itr = list.insert(list.end(), std::move(elem_copy));
        it->getMapped() = itr;
        ...
    }
    else
    {
        updater(list_itr->value);
        ...
    }
}

for (auto it = snapshot.begin; counter < snapshot.snapshot_container_size; ++counter)

/// Destroy snapshot with lock
LockGuardWithStats<false> lock(storage_mutex);
...
/// Turn off "snapshot mode" and clear outdate part of storage state
storage->clearGarbageAfterSnapshot();
...
snapshot.reset();

Summary

We wanted to snapshot a large in-memory hash map to disk without blocking writes. ClickHouse Keeper does it with two simple data structures: a hash map for constant-time lookups, and a doubly linked list where the data actually lives.

When a snapshot starts, it marks a cut-off line in the list. After that, new and updated values go after the line, and deleted or replaced nodes stay in the list, marked inactive. The snapshot thread writes the first N nodes without blocking anyone, and a quick cleanup at the end removes the inactive nodes.

Credits and Resources

For this post, I have heavily used Claude Code. Along with that, I have relied on the following resources:

1. ClickHouse Keeper, ClickHouse docs
2. Redis persistence, for how Redis uses fork() for snapshots
3. ClickHouse source code, v26.2.1.1139-stable

How the WAL is laid out

A write-ahead log is an infinite, append-only stream of records. For the tables we normally use, every change is recorded in the WAL: an insert, an update, a change to an index. Each of them produces a WAL record. And once a record is logged, it is immutable.

But we can’t write an infinite stream to disk as a single file. So Postgres breaks the stream into chunks, and each chunk is a file on disk. These chunks are called segments, and each segment is 16MB by default.

Each segment is again split into pages of 8KB each. Every page starts with some headers ( things like the timeline ID and some flags ), followed by the actual WAL records.

Two small details here. The first page of a segment has a few more headers than the rest: the system identifier, the segment size, and the WAL page size. And a WAL record can cross a page boundary. So every page header has a flag that tells Postgres whether the start of this page is a continuation of a record from the previous page.

Each record also gets a Log Sequence Number ( LSN ). It’s a 64-bit number that keeps increasing and is unique for every record. We don’t need to worry about how Postgres generates it. What matters for us is that the LSN tells us where a record lives. If we know the LSN of an INSERT record, we can find that record and read its details. Without the LSN, finding it is possible, but quite hard.

So let’s find one.

Prerequisites

I’m using the same postgres 17.6 container as the earlier posts. The Dockerfile is on GitHub.

docker build -t postgres-17.6:v1 .
docker network create pgnet
docker run -d --name postgres --network pgnet -e POSTGRES_PASSWORD=secret postgres-17.6:v1

I keep two terminals open: one running psql, and one with a shell inside the container for looking at the files.

Finding our INSERT in the WAL

This is a freshly created container, so nothing has happened in it yet. But Postgres has already set up its WAL and has a current LSN. We can get it with pg_current_wal_lsn():

SELECT pg_current_wal_lsn() AS lsn_before_any_activity \gset
\echo LSN before any activity: :lsn_before_any_activity

LSN before any activity: 0/195A318

\gset stores the result in a psql variable so we can reuse it later. It also hides the output, which is why there’s an \echo after it.

Now let’s create a table and save the LSN again. Then insert one row and save it once more:

CREATE TABLE users (id integer, name text, email text);
SELECT pg_current_wal_lsn() AS lsn_after_creating_table \gset
\echo LSN after creating table: :lsn_after_creating_table

INSERT INTO users VALUES (1, 'Alice', 'alice@example.com');
SELECT pg_current_wal_lsn() AS lsn_after_inserting_a_tuple \gset
\echo LSN after INSERT: :lsn_after_inserting_a_tuple

LSN after creating table: 0/19773B0
LSN after INSERT: 0/1977468

So we now have three LSNs: one before the CREATE TABLE, one between the CREATE TABLE and the INSERT, and one after the INSERT. The records we’re interested in are somewhere between them.

Everything Postgres stores lives in its data directory:

SHOW data_directory;

      data_directory
--------------------------
 /var/lib/postgresql/data

In the other terminal, the WAL is in the pg_wal directory inside it:

docker exec -it postgres bash
cd /var/lib/postgresql/data/pg_wal
ls

000000010000000000000001  archive_status  summaries

There’s only one segment file, since this is a fresh instance. A busy system will have many more. To find out which segment an LSN belongs to, and where inside that file it is, Postgres has pg_walfile_name_offset:

SELECT pg_walfile_name_offset(:'lsn_before_any_activity');

       pg_walfile_name_offset
------------------------------------
 (000000010000000000000001,9806616)

It gives us the segment file’s name and the byte offset inside it. Doing the same for the other two:

SELECT pg_walfile_name_offset(:'lsn_after_creating_table');
SELECT pg_walfile_name_offset(:'lsn_after_inserting_a_tuple');

 (000000010000000000000001,9925552)
 (000000010000000000000001,9925736)

How does it compute this? The LSN itself carries this information. With 16MB segments, the upper bits of the LSN give the segment number, and the lower 24 bits are the byte offset inside that segment. For example, 0/19773B0 is segment 1, offset 0x9773B0, which is 9925552.

The two offsets around the CREATE TABLE are far apart, so that range has a lot of data. The two around our INSERT are only 184 bytes apart. Let’s dump those bytes with xxd:

xxd -s 9925552 -l "$((9925736 - 9925552))" ./000000010000000000000001

009773b0: 3200 0000 0000 0000 486f 9701 0000 0000  2.......Ho......
009773c0: 1008 0000 9780 5a82 ff18 0000 0000 0000  ......Z.........
009773d0: 0000 00b9 3901 f102 0000 f102 0000 f002  ....9...........
009773e0: 0000 0000 0000 0000 5300 0000 f102 0000  ........S.......
009773f0: b073 9701 0000 0000 800a 0000 6f26 a7d3  .s..........o&..
00977400: 0060 2200 7f06 0000 0500 0000 0640 0000  .`"..........@..
00977410: 0000 0000 ff03 0300 0208 1800 0100 0000  ................
00977420: 0d41 6c69 6365 2561 6c69 6365 4065 7861  .Alice%alice@exa
00977430: 6d70 6c65 2e63 6f6d 0100 0000 0000 0000  mple.com........
00977440: 2200 0000 f102 0000 e873 9701 0000 0000  "........s......
00977450: 0001 0000 6352 4a9d ff08 2f2b f9c2 59fe  ....cRJ.../+..Y.
00977460: 0200 0000 0000 0000                      ........

We can already see Alice and alice@example.com in there. We could try to read the rest byte by byte, like we did for the table pages in the earlier post. But Postgres ships a tool that does it for us, pg_waldump. It takes LSNs, not byte offsets:

pg_waldump -s 0/19773B0 -e 0/1977468

rmgr: Standby     len (rec/tot):     50/    50, tx:          0, lsn: 0/019773B0, prev 0/01976F48, desc: RUNNING_XACTS nextXid 753 latestCompletedXid 752 oldestRunningXid 753
rmgr: Heap        len (rec/tot):     83/    83, tx:        753, lsn: 0/019773E8, prev 0/019773B0, desc: INSERT+INIT off: 1, flags: 0x00, blkref #0: rel 1663/5/16390 blk 0
rmgr: Transaction len (rec/tot):     34/    34, tx:        753, lsn: 0/01977440, prev 0/019773E8, desc: COMMIT 2026-09-13 10:43:50.075183 UTC

One SQL statement produced three WAL records. A more complicated statement could produce many more.

• The first one belongs to the Standby resource manager. It describes the transactions running at that moment. We don’t need it here.
• The second one is our insert. It comes from the Heap resource manager. INSERT+INIT means Postgres inserted the tuple and also initialized a new heap page in the same record, since this is the first row in the table.
• The third one is the Transaction resource manager’s COMMIT, because the insert ran inside a transaction.

If we did the same for the CREATE TABLE, we’d see many more records, mostly for the system catalogs it updates.

So these are the WAL records for the operations we ran. But what is a “resource manager”, and how do we interpret these records?

Resource managers

Let’s pretend for a moment that we’re writing the WAL code for Postgres. Different operations need to log different things. An INSERT record needs enough information to rebuild the tuple and the heap page it went into. A CREATE TABLE record, on the other hand, needs to know about the physical file that was created for the table.

So every record has the same fixed header, with things like the transaction ID and the LSN of the previous record. What comes after the header, let’s call it the payload, is different for every operation.

How would we design this? We’d keep a registry, say an array. In it, we’d register a “resource manager” for each kind of operation. Each resource manager has a name and implements a set of functions: how to replay its records during recovery, how to describe them, and so on. In Rust we’d do this with trait objects. In Java, with interfaces. In C, with function pointers.

And that’s what Postgres does. The registry is an array called RmgrTable, and each resource manager in it is a struct called RmgrData.

In the source code

All links below are to the REL_17_6 tag.

The fixed header is the XLogRecord struct in xlogrecord.h. This is the 24-byte header every WAL record starts with:

typedef struct XLogRecord
{
    uint32      xl_tot_len;     /* total len of entire record */
    TransactionId xl_xid;       /* xact id */
    XLogRecPtr  xl_prev;        /* ptr to previous record in log */
    uint8       xl_info;        /* flag bits, see below */
    RmgrId      xl_rmid;        /* resource manager for this record */
    /* 2 bytes of padding here, initialize to zero */
    pg_crc32c   xl_crc;         /* CRC for this record */

    /* XLogRecordBlockHeaders and XLogRecordDataHeader follow, no padding */

} XLogRecord;

The comment above it describes the full layout of a record: the fixed header, then a header for each block ( page ) the record touches, then the data for those blocks, and finally the “main data” that belongs to the resource manager.

The registry, RmgrTable, is in rmgr.c:

#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,decode) \
    { name, redo, desc, identify, startup, cleanup, mask, decode },

RmgrData    RmgrTable[RM_MAX_ID + 1] = {
#include "access/rmgrlist.h"
};

Its entries come from rmgrlist.h. That’s where all the resource managers are listed, including the Heap, Standby and Transaction ones we saw in pg_waldump. The Heap entry looks like this:

PG_RMGR(RM_HEAP_ID, "Heap", heap_redo, heap_desc, heap_identify, NULL, NULL, heap_mask, heap_decode)

Each entry fills in an RmgrData, which is mostly function pointers:

typedef struct RmgrData
{
    const char *rm_name;
    void        (*rm_redo) (XLogReaderState *record);
    void        (*rm_desc) (StringInfo buf, XLogReaderState *record);
    const char *(*rm_identify) (uint8 info);
    void        (*rm_startup) (void);
    void        (*rm_cleanup) (void);
    void        (*rm_mask) (char *pagedata, BlockNumber blkno);
    void        (*rm_decode) (struct LogicalDecodingContext *ctx,
                              struct XLogRecordBuffer *buf);
} RmgrData;

Every resource manager implements its own versions of these functions, and the WAL machinery calls them through these pointers. That’s roughly the design we had in mind.

if (ItemPointerGetOffsetNumber(&(heaptup->t_self)) == FirstOffsetNumber &&
    PageGetMaxOffsetNumber(page) == FirstOffsetNumber)
{
    info |= XLOG_HEAP_INIT_PAGE;
    bufflags |= REGBUF_WILL_INIT;
}

XLogBeginInsert();
XLogRegisterData((char *) &xlrec, SizeOfHeapInsert);
...
XLogRegisterBuffer(0, buffer, REGBUF_STANDARD | bufflags);
XLogRegisterBufData(0, (char *) &xlhdr, SizeOfHeapHeader);
XLogRegisterBufData(0,
                    (char *) heaptup->t_data + SizeofHeapTupleHeader,
                    heaptup->t_len - SizeofHeapTupleHeader);
...
recptr = XLogInsert(RM_HEAP_ID, info);

switch (info & XLOG_HEAP_OPMASK)
{
    case XLOG_HEAP_INSERT:
        heap_xlog_insert(record);
        break;
    case XLOG_HEAP_DELETE:
        heap_xlog_delete(record);
        break;
    case XLOG_HEAP_UPDATE:
        heap_xlog_update(record, false);
        break;
    ...
}

if (XLogRecGetInfo(record) & XLOG_HEAP_INIT_PAGE)
{
    buffer = XLogInitBufferForRedo(record, 0);
    page = BufferGetPage(buffer);
    PageInit(page, BufferGetPageSize(buffer), 0);
    action = BLK_NEEDS_REDO;
}
else
    action = XLogReadBufferForRedo(record, 0, &buffer);

Summary

• The WAL is an infinite stream of records, split into 16MB segment files, which are further split into 8KB pages.
• Every record can be located by its LSN. Each record has a fixed header, followed by a payload specific to that operation.
• Resource managers define what goes in the payload, how to replay it, and how to decode it. They are registered in RmgrTable.
• Tools like pg_waldump let us read the WAL in a human-readable form.

What I left out

• Full-page images. After every checkpoint, the first change to a page writes the entire 8KB page into the WAL, to protect against torn pages.
• Redo is safe to repeat. Replaying a record on a page that already has the change does nothing. The page’s LSN tells Postgres whether the change is already there.
• wal_level changes what gets logged. Under wal_level = logical, the INSERT record gets an extra flag ( 0x08 ) and always keeps the tuple data, so logical decoding can read it.
• Indexes write their own records. An insert into a table with an index also produces records from the index’s resource manager ( for example, Btree ), separate from the Heap record.

Credits and Resources

1. WAL Internals, PostgreSQL docs
2. pg_waldump, PostgreSQL docs
3. The transam README in the postgres source
4. Postgres Source Code

The wishlist

Before looking at what postgres does, let’s write down what we want from this data structure.

1. Search and update should be fast. Before inserting a row, we search for a page with enough free space. After the insert, we update that page’s free space. Both of these happen on every insert, so both should be something like O(log n).

2. Finding the maximum free space should be instant. If even the page with the maximum free space can’t accommodate our row, there’s no point searching through the other pages. We should immediately give up and add a new, empty page to the table. So checking the maximum free space size has to be O(1).

3. Inserts should be spread across pages. This one is a bit nuanced. Say we did the simplest thing: go through the pages one by one and pick the first one with enough room. If page 3 has a lot of free space, the first insert goes to page 3. So does the second, and the third.

Every insert adds onto the same page. That’s bad for performance, because all those inserts end up fighting over the same page. If pages 3, 4 and 5 all have room, we’d rather send one insert to each of them.

So the wishlist is:

1. Search and update in O(log n).
2. Find the maximum free space in O(1).
3. Spread the inserts evenly.

A Max-Heap

There is one data structure that already gives us the first two: a max-heap.

The heap nodes store the free space of each page, one heap node per page. Every internal node stores the maximum of its two children. So the root always holds the largest free space across all pages ( that’s our O(1) maximum ), and searching or updating means walking one path from the root to a leaf or back up ( that’s O(log n) ).

So how do we now spread the inserts evenly? For this, postgres implements a round-robin search:

• The search starts at a leaf, not at the root.
• Postgres remembers where the last search ended, in a variable called next_slot. The next search starts from there, and only moves to the right ( wrapping around at the end ).

The search itself has two phases:

1. Climb. Starting from the next_slot leaf, check if the current node has enough free space. If not, move one node to the right, then climb up to its parent. Repeat until you reach a node with enough space.
2. Descend. Treat that node as the root and walk down like a normal heap search, preferring the left child whenever it has enough space, until you reach a leaf.

Moving right-then-up means we only ever look to the right of where we started, and the area we cover doubles with each step. So we still find a page in O(log n), but each search starts where the last one stopped, and the inserts spread out across pages.

Let’s do a dry run on this tree. The leaves are the free spaces of 8 pages, numbered 0 to 7:

Search(8), with next_slot = 0:

• Start at leaf 0, which has 5. Not enough.
• Move right to leaf 1 and climb to its parent, which has 10. That’s enough, so the climb stops.
• Descend from 10: the left child has 5 ( not enough ), the right child has 10. That’s leaf 1.
• Return page 1, and set next_slot = 2.

Search(25), with next_slot = 2:

• Start at leaf 2, which has 7. Not enough.
• Move right to leaf 3 and climb to its parent, which has 20. Still not enough.
• Move right to the node with 35 and climb to its parent, which has 40. That’s enough.
• Descend from 40: the left child has 35, which is enough, so go left. From 35: the left child has 5, the right child has 35. That’s leaf 5.
• Return page 5, and set next_slot = 6.

Notice that the second search never looked at pages 0 or 1. It started where the first one stopped.

From one page to millions

There is one more problem. This max-heap has to live on disk, and everything postgres stores on disk lives in 8KB pages. One 8KB page can hold the max-heap for about 4,000 data pages. That’s nowhere near the millions of pages we want to handle.

So postgres splits the heap across many pages and arranges those pages as a tree too. Each FSM page holds its own small max-heap. The leaves of a top-level FSM page don’t describe data pages. Each leaf holds the root value of a child FSM page one level below. Only the leaves of the bottom-level FSM pages hold the free space of the actual data pages.

So we have a tree of FSM pages, and inside each FSM page there is a max-heap. Put together, it’s still one big max-heap.

The search changes a little:

1. Start at the root FSM page and go down level by level.
2. Inside each FSM page, run the same climb-and-descend search. It tells us which child page to read next. At the bottom level, it tells us which data page to use.

For every search, postgres reads exactly three FSM pages: the root, one page from the middle level, and one from the bottom level.

How far does this scale? With 8KB pages, each FSM page has room for 4,069 leaves. Three levels give us 4,069 × 4,069 × 4,069, about 67 billion pages. That’s more than the roughly 4 billion pages a single postgres table can have anyway. So reading 3 FSM pages is enough to find room for a row, even in the biggest table postgres supports.

To summarise: when we insert a row, postgres checks the root of the FSM. If even the biggest free space isn’t enough, it adds a new page to the table. Otherwise, it walks down the three levels of the FSM, doing a round-robin search inside each FSM page, and ends up at a data page with enough room.

Prerequisites

I’m using the same postgres container as the earlier posts. The Dockerfile is on GitHub. The pg_freespacemap extension we’ll use is included in the official postgres image.

docker build -t postgres-17.6:v1 .
docker network create pgnet
docker run -d --name postgres --network pgnet -e POSTGRES_PASSWORD=secret postgres-17.6:v1
docker exec -it postgres bash

I keep two terminals open inside the container: one for psql -U postgres and one for looking at the files.

The FSM in the terminal

Let’s switch to a test database and create a table:

CREATE DATABASE dummy;
\c dummy

CREATE TABLE items (id serial, data text);

SELECT pg_relation_filepath('items');

 pg_relation_filepath
----------------------
 base/16386/16388

The FSM is stored per table, right next to the table’s data file. In the other terminal:

cd $PGDATA
ls -l base/16386 | grep 16388

-rw------- 1 postgres postgres     0 ... 16388

There’s just the data file, and it’s empty: zero bytes, because we haven’t inserted anything yet. There is no FSM file. Postgres is lazy about the FSM. It won’t create it until it needs it.

Let’s insert enough rows to fill a few pages:

INSERT INTO items (data)
SELECT repeat('x', 200) FROM generate_series(1, 200);

ls -l base/16386 | grep 16388

-rw------- 1 postgres postgres 49152 ... 16388
-rw------- 1 postgres postgres 24576 ... 16388_fsm

The table now has 6 pages ( 49152 / 8192 ), and a new file, 16388_fsm, has appeared. It’s 24KB, which is 3 pages. That’s what we’d expect: the root FSM page, one page at the middle level, and one page at the bottom level. These pages are stored in the file depth-first. So if the table grows enough to need a second bottom-level page, it goes right after the first one.

We won’t decode the raw bytes this time. Instead, postgres has an extension that reads the FSM for us:

CREATE EXTENSION IF NOT EXISTS pg_freespacemap;

SELECT * FROM pg_freespace('items');

 blkno | avail
-------+-------
     0 |   128
     1 |   128
     2 |   128
     3 |   128
     4 |   128
     5 |     0

Each row is 200 x’s plus some headers, so 34 of them fit in a page. The first five pages are nearly full, with only 128 bytes left. The last page says 0, but it isn’t actually full: it only has 30 rows. Postgres records a page’s free space in the FSM when it moves on from that page, and we never moved on from the last one.

Now let’s delete half the rows:

DELETE FROM items WHERE id % 2 = 0;

SELECT * FROM pg_freespace('items');

 blkno | avail
-------+-------
     0 |   128
     1 |   128
     2 |   128
     3 |   128
     4 |   128
     5 |     0

Nothing changed. Postgres is lazy here too. A DELETE only marks the rows as deleted. The space is reclaimed later by VACUUM, which also updates the FSM. We can run it ourselves:

VACUUM items;

SELECT * FROM pg_freespace('items');

 blkno | avail
-------+-------
     0 |  4064
     1 |  4064
     2 |  4064
     3 |  4064
     4 |  4064
     5 |  4544

Now every page is about half empty, which matches the half of the rows we deleted. If you list the files again, you’ll also see a 16388_vm file. That’s the visibility map, which VACUUM creates. It isn’t in the scope of this post.

In the source code

The FSM code lives in src/backend/storage/freespace. The README there is a very readable description of the whole design.

The search inside one FSM page is fsm_search_avail in fsmpage.c. The climb is just a few lines ( L227–L238 ):

nodeno = target;
while (nodeno > 0)
{
    if (fsmpage->fp_nodes[nodeno] >= minvalue)
        break;

    /*
     * Move to the right, wrapping around on same level if necessary, then
     * climb up.
     */
    nodeno = parentof(rightneighbor(nodeno));
}

After the descend, it updates the start point for the next search ( L303 ):

fsmpage->fp_next_slot = slot + (advancenext ? 1 : 0);

Going down the three levels happens in fsm_search in freespace.c. It reads one FSM page per level and calls fsm_search_avail on it. One small detail there: advancenext is only true at the bottom level ( L697 ). So it’s the bottom-level pages, the ones pointing at real data pages, that move their next_slot forward and spread the inserts.

What I left out

• Quantization. Postgres doesn’t store the exact free space of a page. It stores one byte per page: 256 categories of 32 bytes each ( FSM_CATEGORIES and FSM_CAT_STEP in freespace.c ). The exact free space would need 13 bits. Postgres rounds the available space down and the requested space up, so it never sends a row to a page that can’t fit it. That’s why we saw neat numbers like 128 and 4064 above.
• The FSM is not WAL-logged. It’s treated as a hint. If it’s wrong after a crash, that’s fine: searches detect and fix broken FSM pages, and VACUUM rewrites the correct values.
• The FSM can be stale. It’s mostly updated by VACUUM, and by inserts that find a page full and move on. So a page can have more ( or less ) room than the FSM says. If an insert lands on a page that’s actually full, it records the real value and searches again.
• The tree depth. The three levels are a compile-time constant ( FSM_TREE_DEPTH ). With very small page sizes, postgres would use four levels instead.

Credits and Resources

For this post, I have heavily used Claude Code. Along with that, I have relied on the following resources:

1. Free Space Map, PostgreSQL docs
2. The FSM README in the postgres source
3. Postgres Source Code

How postgres organizes storage

Before we go inserting and searching for data, let’s first build a picture of how postgres organizes data. That way, when we trace the bytes for our data, we’ll know what we’re looking at.

In Postgres, every table is stored in one or more files. We call these files segment files. A segment file can grow up to 1GB. If a table grows beyond 1GB, postgres creates another segment file for the next 1GB, then another one, and so on. For a table whose file is named 16388, the segments would be named 16388, 16388.1, 16388.2, and so on. ( The 1GB limit can be changed at the time of setup )

Each segment file is made of pages ( also called blocks ). A page is 8KB by default. This too can only be changed at compile time. The page is the smallest unit of IO in postgres: when postgres reads from or writes to a table file, it reads or writes a whole 8KB page. Any change to a row, however small, happens by changing the page that contains it.

Now let’s zoom into a page. Every page has the same layout:

• A page header of 24 bytes at the start.
• An array of line pointers right after the header. Each line pointer is 4 bytes and points to one row in the page. This array grows from the top of the page downwards.
• The tuples ( rows ) themselves, which are stored from the bottom of the page upwards.
• The free space in between. New rows go here. The line pointers grow into it from the top, the tuples grow into it from the bottom.
• There’s also some special space at the very end. Tables don’t use it, but some indexes use it. This is beyond our scope for this post.

Each tuple also has a small header of its own, with things like the transaction id that created it, followed by the actual column values.

Prerequisites

For this post, I’m assuming you know what docker containers are and how to build and run them.

I’m using the same postgres container as the previous post, with a few extra tools installed for looking at binary files ( like xxd ). The Dockerfile is on GitHub.

docker build -t postgres-17.6:v1 .

# To let the python app talk to postgres, we create a docker network
docker network create pgnet
docker run -d --name postgres --network pgnet -e POSTGRES_PASSWORD=secret postgres-17.6:v1

# Open a shell inside the container
docker exec -it postgres bash

I’ll keep two terminals open inside the container: one for psql and one for looking at the files.

echo $PGDATA
# /var/lib/postgresql/data

cd $PGDATA && ls -l

total 128
drwx------ 6 postgres postgres  4096 Mar 14 13:05 base
drwx------ 2 postgres postgres  4096 Mar 14 13:05 global
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_commit_ts
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_dynshmem
-rw------- 1 postgres postgres  5743 Mar 14 13:05 pg_hba.conf
-rw------- 1 postgres postgres  2640 Mar 14 13:05 pg_ident.conf
drwx------ 4 postgres postgres  4096 Mar 14 13:05 pg_logical
drwx------ 4 postgres postgres  4096 Mar 14 13:05 pg_multixact
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_notify
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_replslot
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_serial
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_snapshots
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_stat
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_stat_tmp
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_subtrans
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_tblspc
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_twophase
-rw------- 1 postgres postgres     3 Mar 14 13:05 PG_VERSION
drwx------ 4 postgres postgres  4096 Mar 14 13:05 pg_wal
drwx------ 2 postgres postgres  4096 Mar 14 13:05 pg_xact
-rw------- 1 postgres postgres    88 Mar 14 13:05 postgresql.auto.conf
-rw------- 1 postgres postgres 30777 Mar 14 13:05 postgresql.conf
-rw------- 1 postgres postgres    36 Mar 14 13:05 postmaster.opts
-rw------- 1 postgres postgres    94 Mar 14 13:05 postmaster.pid

ls base/
# 1  16385  4  5

CREATE DATABASE dummy;

SELECT oid, datname FROM pg_database;

  oid  |  datname
-------+------------
     5 | postgres
 16385 | mydatabase
     1 | template1
     4 | template0
 16386 | dummy

ls base/
# 1  16385  16386  4  5

ls base/16386 | wc -l
# 298

CREATE TABLE users (
    id   serial PRIMARY KEY,
    name text,
    age  int
);

INSERT INTO users (name, age) VALUES ('Bob', 25);

SELECT pg_relation_filepath('users');

 pg_relation_filepath
----------------------
 base/16386/16388

ls -l base/16386/16388
# -rw------- 1 postgres postgres 8192 ... base/16386/16388

xxd -a base/16386/16388

00000000: 0000 0000 a8ac d401 0000 0000 1c00 d81f  ................
00000010: 0020 0420 0000 0000 d89f 4800 0000 0000  . . ......H.....
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
00001fd0: 0000 0000 0000 0000 de02 0000 0000 0000  ................
00001fe0: 0000 0000 0000 0000 0100 0300 0208 1800  ................
00001ff0: 0100 0000 0942 6f62 1900 0000 0000 0000  .....Bob........

typedef struct PageHeaderData
{
    PageXLogRecPtr pd_lsn;
    uint16      pd_checksum;
    uint16      pd_flags;
    LocationIndex pd_lower;     /* offset to start of free space */
    LocationIndex pd_upper;     /* offset to end of free space */
    LocationIndex pd_special;   /* offset to start of special space */
    uint16      pd_pagesize_version;
    TransactionId pd_prune_xid;
    ItemIdData  pd_linp[FLEXIBLE_ARRAY_MEMBER]; /* line pointer array */
} PageHeaderData;

xxd -l 24 base/16386/16388

00000000: 0000 0000 a8ac d401 0000 0000 1c00 d81f  ................
00000010: 0020 0420 0000 0000                      . . ....

typedef struct ItemIdData
{
    unsigned    lp_off:15,      /* offset to tuple (from start of page) */
                lp_flags:2,     /* state of line pointer, see below */
                lp_len:15;      /* byte length of tuple */
} ItemIdData;

xxd -s 24 -l 4 -b base/16386/16388

00000018: 11011000 10011111 01001000 00000000                    ..H.

00000000 01001000 10011111 11011000

 lp_len (15 bits)   lp_flags (2)   lp_off (15 bits)
 000000000100100       01        001111111011000

xxd -s 8152 -l 36 base/16386/16388

00001fd8: de02 0000 0000 0000 0000 0000 0000 0000  ................
00001fe8: 0100 0300 0208 1800 0100 0000 0942 6f62  .............Bob
00001ff8: 1900 0000                                ....

xxd -s 8176 -l 12 base/16386/16388

00001ff0: 0100 0000 0942 6f62 1900 0000            .....Bob....

CREATE EXTENSION pageinspect;

SELECT lower, upper, special, pagesize, version
FROM page_header(get_raw_page('users', 0));

 lower | upper | special | pagesize | version
-------+-------+---------+----------+---------
    28 |  8152 |    8192 |     8192 |       4

SELECT lp, lp_off, lp_flags, lp_len, t_xmin, t_ctid, t_hoff, t_data
FROM heap_page_items(get_raw_page('users', 0));

 lp | lp_off | lp_flags | lp_len | t_xmin | t_ctid | t_hoff |           t_data
----+--------+----------+--------+--------+--------+--------+----------------------------
  1 |   8152 |        1 |     36 |    734 | (0,1)  |     24 | \x0100000009426f6219000000

Prerequisites

For this post, I’m assuming you:

• Know what Docker containers are and how to build and run them.
• Are familiar with processes and threads.

We’ll use two containers: one running Postgres, and one with a small Python script that connects to it. Create a folder with these three files.

FROM postgres:17.6

USER root

# procps gives us `ps` inside the container
RUN apt-get update && apt-get install -y procps \
    && rm -rf /var/lib/apt/lists/*

# A user and database for the Python client to connect to
RUN echo "CREATE USER myuser WITH PASSWORD 'mypassword'; CREATE DATABASE mydatabase OWNER myuser;" \
    > /docker-entrypoint-initdb.d/init.sql

EXPOSE 5432

USER postgres

FROM python:3.12-slim

RUN pip install psycopg2-binary

COPY connect.py /app/connect.py

ENTRYPOINT ["python", "/app/connect.py"]

import sys
import os
import time
import psycopg2

conn_params = {
    "host":     os.getenv("PG_HOST", "postgres"),
    "port":     os.getenv("PG_PORT", "5432"),
    "dbname":   os.getenv("PG_DB",   "mydatabase"),
    "user":     os.getenv("PG_USER", "myuser"),
    "password": os.getenv("PG_PASSWORD", "mypassword"),
}

cmd = sys.argv[1] if len(sys.argv) > 1 else ""

if cmd == "connect":
    # Open a connection, hold it for 30 seconds, then close it
    conn = psycopg2.connect(**conn_params)
    print("Connected to postgres successfully.")
    time.sleep(30)
    conn.close()

elif cmd == "select":
    # Run a query that takes 30 seconds
    conn = psycopg2.connect(**conn_params)
    cur = conn.cursor()
    print("Running SELECT pg_sleep(30) ...")
    cur.execute("SELECT pg_sleep(30)")
    print("Done.")
    cur.close()
    conn.close()

else:
    print("Usage: python connect.py [connect|select]")
    sys.exit(1)

All the code for this post is also on GitHub.

Then build the images and start Postgres:

# Build the Postgres and Python client images
docker build -t postgres-17.6:v1 .
docker build -f Dockerfile.client -t pg-client:v1 .

# Create a network so the client container can reach Postgres by name
docker network create pgnet

# Start Postgres
docker run -d --name postgres --network pgnet -e POSTGRES_PASSWORD=secret postgres-17.6:v1

Walkthrough

Now that we have the docker containers built, let’s begin. Let’s exec into the postgres container:

docker exec -it postgres bash

Once inside, we can list all the processes running in our container by doing ps aux. We get the following output:

postgres@0f41997a8b1b:/$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
postgres       1  0.0  0.1 217536 28556 ?        Ss   Mar12   0:00 postgres
postgres      49  0.0  0.0 217668 10088 ?        Ss   Mar12   0:00 postgres: checkpointer 
postgres      50  0.0  0.0 217692  7436 ?        Ss   Mar12   0:00 postgres: background writer 
postgres      52  0.0  0.0 217536 10264 ?        Ss   Mar12   0:00 postgres: walwriter 
postgres      53  0.0  0.0 219112  9136 ?        Ss   Mar12   0:00 postgres: autovacuum launcher 
postgres      54  0.0  0.0 219120  8276 ?        Ss   Mar12   0:00 postgres: logical replication launcher 
postgres     399  1.3  0.0   7352  4096 pts/0    Ss   06:51   0:00 bash
postgres     406  0.0  0.0   9392  4116 pts/0    R+   06:51   0:00 ps aux

Already a lot seems to be going on, even though we have just freshly started this container. There are several processes already running. We’ll get to the other processes later in the series, but for now the most important process for us is the one with PID 1. This is the postmaster. When we start postgres, the postmaster process is started. It then spawns the other postgres processes that we see listed above.

The postmaster process has a key responsibility: it runs an infinite loop, waiting for new connections, and “handles” the incoming connections. Of course, there are other things as well that this postmaster process does, but for the purposes of this blog, we’ll focus on connection handling.

Now, let’s try to connect to postgres using the python application. In a second terminal (outside the container), run:

docker run --rm --network pgnet pg-client:v1 connect

This is going to run the python application that will try to connect to the postgres, hold the connection open for 30 seconds and then close it. From the postgres server terminal, if we again run ps aux, we get a new entry now, the process with PID 432:

postgres@0f41997a8b1b:/$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
postgres       1  0.0  0.1 217536 28556 ?        Ss   Mar12   0:00 postgres
postgres      49  0.0  0.0 217668 10088 ?        Ss   Mar12   0:00 postgres: checkpointer 
postgres      50  0.0  0.0 217692  7436 ?        Ss   Mar12   0:00 postgres: background writer 
postgres      52  0.0  0.0 217536 10264 ?        Ss   Mar12   0:00 postgres: walwriter 
postgres      53  0.0  0.0 219112  9136 ?        Ss   Mar12   0:00 postgres: autovacuum launcher 
postgres      54  0.0  0.0 219120  8276 ?        Ss   Mar12   0:00 postgres: logical replication launcher 
postgres     426  0.6  0.0   7352  4060 pts/0    Ss   06:57   0:00 bash
postgres     432  0.0  0.0 219648 12960 ?        Ss   06:57   0:00 postgres: myuser mydatabase 172.18.0.3(48278) idle
postgres     433  0.0  0.0   9392  4136 pts/0    R+   06:57   0:00 ps aux

When our python application tried to connect to postgres server, the postmaster process jumped to action and created a new process to handle that connection request. From the name of the process, we can interpret a few things:

• postgres: - this is the prefix attached to all the processes spawned by postmaster
• myuser - the name of the user that connected to postgres
• mydatabase - the name of the database to which the user connected
• 172.18.0.3(48278) - the host and port of the python application i.e. the client
• idle - the state of the connection. Currently, the connection is sitting idle, without any work to do.

We can see the same thing from inside Postgres. There is a view in Postgres called pg_stat_activity, which has one row per backend process:

psql -U myuser -d mydatabase -c "SELECT pid, usename, datname, client_addr, state FROM pg_stat_activity WHERE backend_type = 'client backend';"

The pid column matches the process ID from ps aux. You’ll also see a second row for the psql session we just opened, because it’s a connection too, with its own backend process.

If we try to simulate a select query that runs for 30 seconds using:

docker run --rm --network pgnet pg-client:v1 select

we see the state of the connection change from idle to SELECT. This is postgres indicating to us that the handler process is currently executing a SELECT request sent by the client.

If we open another terminal and again try to connect to postgres server, we will see another process is being spawned. This is a key point: for every connection, the postgres server is going to have a process handling that connection. This process has its own stack, memory space, and the context switching overhead.

At the same time, this process is independent of the other connections. If it exits or runs into an error, the postgres server continues to function as before. ( Note that a real crash, like a segfault or the Linux OOM killer killing the process, is different: postgres then stops all the processes and restarts. )

Postgres runs on a process-per-connection model. For every new connection, postgres is going to spawn a new process to handle it. When the connection is terminated, the process exits. Now, many modern databases do not follow multi-process model, but rather the multi-threaded model. For each new connection, they’re gonna spawn (reuse from the threadpool) some threads. This is lighter than the process based approach but if not carefully designed, can cause the database to crash.

The multi-process model of postgres has important implications on how we scale postgres.

If we have an application with thousands of users who want to connect to the database, the postgres server is going to end up spawning thousands of processes. This is going to hamper the performance of our database. We simply cannot have a case where we are spawning thousands of processes just to handle connections.

To get around this, we can do several things. The OpenAI blog post lists a bunch of things they did to scale postgres to serve 800 million users. We’ll focus on a couple of them:

1. Setting max_connections parameter: at the most basic level, we can restrict the number of connections postgres will accept. By default, this is set to 100. After this limit is reached, the clients trying to connect will get an error. Out of these 100, a few connections (3 by default, set by superuser_reserved_connections) are kept aside for superusers, so that an admin can still log in and fix things even when all the other connections are used up. This safeguards the postgres server but doesn’t serve the clients well. So, we look at the next option.

2. Using a connection pooler like PgBouncer. In such a setup, the clients try to connect to the pooler and the pooler maintains a small pool of connections with postgres. As a result, from our application’s point of view, there could be thousands of connections but from the point of view of the database, there are only a handful of connections.

Source Code Walkthrough

Let’s see where this happens in the postgres source code. I’m using the REL_17_6 branch here, so the line numbers in the links match postgres 17.6.

The postmaster runs an infinite loop called ServerLoop in postmaster.c. When a new client connects, the loop accepts the connection and calls BackendStartup. This function is called once for every new connection. It does some bookkeeping for the new backend, like generating a cancel key, and then calls postmaster_child_launch.

postmaster_child_launch lives in launch_backend.c, and this is where the fork actually happens:

pid = fork_process();
if (pid == 0)               /* child */
{
    /* Close the postmaster's sockets */
    ClosePostmasterPorts(child_type == B_LOGGER);
    ...

fork_process() is a small wrapper around the fork() system call. After the fork, there are two processes running the same code. In the child (pid == 0), the first thing it does is close the postmaster’s listening sockets, since it doesn’t need them. But it keeps the socket for its own client. This is the “copy of the socket” I mentioned earlier: fork() copies all the open file descriptors of the parent, so the child gets the client’s connection for free.

That’s it- one fork() per connection. Every backend process we saw in ps aux was created here.

Credits and Resources

For this post, I have heavily used Claude Code. Along with that, I have relied on the following resources:

1. Intro to Database Systems by CMU
2. Postgres Source Code