Under The Hood, Off The Hot Path: How D1 Scales To Millions Of Databases

Welcome back to Self-Hosted Tech! If you’ve been running a homelab for any amount of time, you already know that databases are the beating heart of almost every application we deploy. Whether you’re spinning up Nextcloud, tracking your smart home metrics in Home Assistant, or building a custom wrapper for your local LLMs, you need somewhere to store that data.

For most of us, that means spinning up a Docker container for PostgreSQL or MariaDB. But what if I told you there’s a way to run millions of databases across a global network without dedicating a single virtual machine to any of them?

Today, we’re taking a deep dive into Cloudflare’s D1. D1 is a serverless SQL database, but under the hood, it’s powered by something near and dear to our self-hosting hearts: SQLite. But Cloudflare didn’t just slap SQLite on a server and call it a day. They completely re-architected how it works to scale to millions of databases.

The secret lies in a concept called moving storage “off the hot path.” Let’s look under the hood to see how D1 works, why it’s a massive shift in database architecture, and how you can apply these same concepts to your own homelab and local AI projects.

The SQLite Dilemma: Fast, But Local

To understand why D1 is so impressive, we have to talk about vanilla SQLite. SQLite is arguably the most widely deployed database engine in the world. It’s fast, incredibly reliable, and doesn’t require a background daemon. It’s just a file on a disk.

In a homelab, SQLite is perfect for small apps. But it has a fundamental limitation: it’s designed to live on a local disk. If you try to put a SQLite file on a network share (like an NFS mount or a SMB share) and have multiple nodes read and write to it, you’re going to have a bad time. Network file systems don’t handle SQLite’s file locking mechanisms well, leading to corruption and database locks.

Because SQLite is tied to a single disk, scaling it is hard. If you want to scale a traditional database, you usually provision a bigger server with more RAM and CPU, or you set up a complex primary-replica clustering topology.

Cloudflare wanted to offer databases to millions of developers without managing millions of VMs. They needed a way to run SQLite globally, durably, and cheaply. Their solution was to completely separate the compute (the database engine) from the storage (the actual data on disk).

What Does “Off The Hot Path” Mean?

In software architecture, the “hot path” is the sequence of operations that happens most frequently and requires the lowest possible latency. If you’re querying a database, the hot path is: receive query -> execute query -> return result.

In a traditional database, reading and writing to the disk is on the hot path. The database engine talks directly to the local storage controller. If the disk is slow, your query is slow.

Moving storage “off the hot path” means the database engine doesn’t wait on the primary storage medium to do its job. Instead, it relies on fast, local caches for the hot path, and asynchronously syncs data to the durable, long-term storage in the background.

In D1’s architecture, the “hot path” is served by local RAM and NVMe caches on Cloudflare’s edge servers. The “cold path”—the actual durable storage of the database—is pushed to Cloudflare’s R2 object storage.

Under the Hood: The D1 Architecture

So, how does D1 actually run SQLite globally without corrupting the file? It uses a three-tier architecture: Compute, Durable Objects, and Object Storage.

1. Stateless Compute

When you send a query to a D1 database, it hits a Cloudflare Worker. The Worker is completely stateless. It contains the SQLite engine compiled to WebAssembly (WASM). Because it’s stateless, Cloudflare can spin up and tear down these Worker instances anywhere in the world in milliseconds.

2. Durable Objects (The Coordinator)

If the compute is stateless, how does it know where the data is, and how do we prevent two Workers from writing to the database at the same time?

This is where Cloudflare Durable Objects come in. Every D1 database has a corresponding Durable Object that acts as the single source of truth for that specific database. When a stateless Worker wants to read or write data, it sends a request to the Durable Object.

The Durable Object handles all the locking. Because only one Durable Object exists for a given database at any time, there’s no risk of concurrent writes corrupting the file. The Durable Object keeps a transactional log in memory and on local NVMe drives (the hot path).

3. Object Storage (The Cold Path)

The actual SQLite file isn’t stored on a traditional database server. It’s chunked up and stored in R2, Cloudflare’s S3-compatible object storage.

When a Durable Object receives a write, it writes it to its local NVMe cache and acknowledges the write to the Worker. Then, in the background, it periodically takes those changes, packages them up, and ships them off to R2.

If the server holding the Durable Object catches fire, no data is lost. The system simply spins up a new Durable Object in another data center, pulls the latest chunks from R2, replays any missing transaction logs, and continues serving traffic.

How This Scales to Millions

Why is this architecture such a game-changer for scale?

Because of packing density.

In a traditional setup, if you have 10,000 databases, you need enough servers to hold the active memory and compute for all 10,000, even if 9,000 of them haven’t been queried in a week.

With D1, databases that aren’t being used are essentially just files sitting in object storage. They consume zero CPU and zero RAM. Cloudflare can pack thousands of inactive databases into a single R2 bucket. When a query finally comes in for one of those dormant databases, a Durable Object is spun up, the relevant chunks are pulled from R2 into the local NVMe cache, and the query is served.

This “wake on request” model is exactly how D1 can handle millions of databases without breaking the bank. You only pay for the compute and storage you actually use.

Practical Walkthrough: Using D1 for a Local AI Chat Log

While D1 is a managed Cloudflare service, it’s incredibly useful for self-hosters who want to deploy apps to the edge or need a zero-maintenance database for a side project. Let’s walk through setting up a D1 database to store chat history for a local AI project.

Note: You’ll need Node.js and a Cloudflare account for this.

First, install the Wrangler CLI, which is Cloudflare’s command-line tool for managing Workers and D1:

1
npm install -g wrangler

Log into your Cloudflare account:

1
wrangler login

Now, let’s create a D1 database for our AI chat history:

1
wrangler d1 create ai-chat-history

The CLI will output some configuration details. Take note of the database_id. You’ll need to add this to your project’s wrangler.toml file:

1
2
3
4
5
6
7
8
name = "ai-chat-wrapper"
main = "src/index.js"
compatibility_date = "2023-10-30"

[[d1_databases]]
binding = "DB"
database_name = "ai-chat-history"
database_id = "YOUR_DATABASE_ID_HERE"

Next, let’s create a schema for our chat logs. Create a file named schema.sql:

1
2
3
4
5
6
7
CREATE TABLE IF NOT EXISTS conversations (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    user_prompt TEXT NOT NULL,
    ai_response TEXT NOT NULL,
    model TEXT NOT NULL,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

Apply this schema to your D1 database directly from your terminal:

1
wrangler d1 execute ai-chat-history --file=./schema.sql

Now, let’s write a simple Cloudflare Worker that acts as an API endpoint. When it receives a POST request with a user prompt and an AI response, it logs it to the D1 database.

Create src/index.js:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
export default {
  async fetch(request, env) {
    if (request.method === "POST") {
      try {
        const { user_prompt, ai_response, model } = await request.json();

        // Insert the chat log into D1
        const result = await env.DB.prepare(
          "INSERT INTO conversations (user_prompt, ai_response, model) VALUES (?, ?, ?)"
        ).bind(user_prompt, ai_response, model).run();

        return new Response(JSON.stringify({ success: true, id: result.meta.last_row_id }), {
          headers: { "Content-Type": "application/json" },
        });
      } catch (e) {
        return new Response(JSON.stringify({ error: e.message }), { status: 500 });
      }
    }

    // Fetch the last 10 chat logs
    if (request.method === "GET") {
      const { results } = await env.DB.prepare(
        "SELECT * FROM conversations ORDER BY created_at DESC LIMIT 10"
      ).all();

      return new Response(JSON.stringify(results), {
        headers: { "Content-Type": "application/json" },
      });
    }

    return new Response("Method not allowed", { status: 405 });
  },
};

Deploy your worker:

1
wrangler deploy

Just like that, you have a globally distributed, zero-maintenance SQLite database backing your AI application. You can query it locally using the CLI anytime:

1
wrangler d1 execute ai-chat-history --command="SELECT * FROM conversations;"

Stealing D1’s Design for Your Homelab

You might be thinking, “That’s cool, but I prefer to keep my data on my own metal.” I totally get it. That’s the self-hosted way.

The good news is, you can replicate D1’s “off the hot path” architecture in your own homelab.

If you’re writing an app and want to use SQLite for its speed and simplicity, but you’re worried about backups or running it on a network-attached drive, you can use a tool called Litestream.

Litestream is an open-source tool that runs alongside your SQLite database. It works very similarly to D1’s storage layer:

Your application reads and writes to a local SQLite file on your fast NVMe drive (the hot path).
Litestream monitors the SQLite Write-Ahead Log (WAL) in real-time.
As changes happen, Litestream asynchronously streams those changes to S3-compatible storage (the cold path).

In your homelab, you can run your apps on a high-performance node, and use MinIO or TrueNAS to act as your S3-compatible object storage. If your primary node dies, you still have your local SQLite file, but if the disk fails, Litestream can automatically restore the database from your MinIO bucket onto a new server.

This gives you the exact same benefits as D1: blazing fast local compute, with durable, off-the-hot-path storage.

Summary

Cloudflare’s D1 is a masterclass in rethinking database architecture. By separating the SQLite compute engine from the storage layer, pushing durable storage “off the hot path” into object storage, and using Durable Objects to coordinate state, they’ve managed to scale a local file-based database to millions of instances globally.

For homelabbers and self-hosters, the lessons here are invaluable. Whether you choose to use D1 for your next edge-deployed AI wrapper, or you implement Litestream to back up your local SQLite databases to your NAS, moving storage off the hot path is a pattern that leads to faster apps and more resilient data.

What do you think? Are you using SQLite in your homelab, or are you strictly a Postgres/MariaDB person? Let us know in the comments below how you handle database backups and scaling in your local setup!

If you enjoyed this deep dive, make sure to hit that subscribe button on the Self-Hosted Tech YouTube channel and drop a comment with what architecture topics you want us to cover next. Happy self-hosting!

Under The Hood, Off The Hot Path: How D1 Scales To Millions Of Databases#

The SQLite Dilemma: Fast, But Local#

What Does “Off The Hot Path” Mean?#

Under the Hood: The D1 Architecture#

1. Stateless Compute#

2. Durable Objects (The Coordinator)#

3. Object Storage (The Cold Path)#

How This Scales to Millions#

Practical Walkthrough: Using D1 for a Local AI Chat Log#

Stealing D1’s Design for Your Homelab#

Summary#