Inside Hyperdrive: How We Solved A Rare Hyperdrive MySQL Caching Bug

If you’ve been following the self-hosting scene for a while, you know that the dream is simple: run your own services, own your data, and keep latency low. Cloudflare’s Hyperdrive has been a game-changer for a lot of us in that regard. It pools database connections and caches query results at the edge, which means your self-hosted MySQL database sitting in a closet at home can suddenly feel like it’s running in a data center across the street.

But a few weeks ago, we hit a bug that nearly made us rip Hyperdrive out of our stack entirely. It was subtle, intermittent, and — as it turns out — rare enough that almost nobody was talking about it online. So we’re writing this up in case you ever run into the same wall.

Let’s walk through what happened, how we debugged it, and the fix that got everything running smoothly again.

The Setup: What We Were Running

Before we get into the bug, here’s the context. Our homelab stack looked like this:

Database server: A Dell PowerEdge R720 running Debian 12, with MySQL 8.0.36
App layer: A Node.js API (Express) deployed on Cloudflare Workers
Connection layer: Cloudflare Hyperdrive, configured to connect to our home MySQL instance over a Cloudflare Tunnel
Cache: Hyperdrive’s built-in query caching, enabled with a 60-second TTL

The app was a personal finance dashboard. It pulled transaction data, category breakdowns, and budget summaries from MySQL. Most queries were read-heavy aggregations — exactly the kind of workload Hyperdrive caching was designed for.

Everything worked great for about three weeks. Then the weird stuff started.

The Symptoms: Ghost Data And Phantom Updates

It began with a user report (okay, it was my spouse, but she counts). She logged a new transaction in the app, refreshed the dashboard, and… it wasn’t there. She refreshed again. Still nothing. Then, about two minutes later, it appeared.

At first, I assumed it was a frontend rendering issue. Maybe the API response was stale on the client side. But when I checked the API directly with curl, the response was missing the new transaction too.

Here’s what made this genuinely confusing:

The transaction was in MySQL. I could see it with a direct SELECT from the database server.
The Node.js API, when run locally against the database directly (bypassing Hyperdrive), returned the correct data immediately.
The same API running on Workers through Hyperdrive returned stale data for roughly 60–120 seconds.
It didn’t happen on every insert. Only about 30% of the time.

That last point is what made this a nightmare. It was intermittent.

Step 1: Ruling Out The Obvious

The first thing we did was check whether this was just normal cache behavior. Hyperdrive caches SELECT queries, and the default TTL is 60 seconds. If you write data and then immediately read it, you might get a cached result that’s up to 60 seconds old.

But we had already accounted for that. Our write-then-read flow looked like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
// 1. Insert the transaction
await db.execute(
  'INSERT INTO transactions (amount, category, date) VALUES (?, ?, ?)',
  [amount, category, date]
);

// 2. Immediately re-query the dashboard summary
const summary = await db.execute(
  'SELECT category, SUM(amount) as total FROM transactions GROUP BY category'
);

Hyperdrive is supposed to invalidate cached results when it detects a write operation (INSERT, UPDATE, DELETE) on the same connection. The cache key is based on the query text and parameters, and a write should bust any cached SELECT results that might be affected.

So in theory, step 2 should never return stale data after step 1. But it was.

Step 2: Enabling Hyperdrive Debug Logging

We turned on verbose logging in our Worker to see exactly what Hyperdrive was doing. Cloudflare Workers lets you add structured logs, so we instrumented the code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
export default {
  async fetch(request, env) {
    const db = env.DB.withSession(); // Use session mode for consistency

    console.log('Inserting transaction...');
    const insertResult = await db.execute(
      'INSERT INTO transactions (amount, category, date) VALUES (?, ?, ?)',
      [amount, category, date]
    );
    console.log('Insert result:', JSON.stringify(insertResult));

    console.log('Querying summary...');
    const summary = await db.execute(
      'SELECT category, SUM(amount) as total FROM transactions GROUP BY category'
    );
    console.log('Summary result:', JSON.stringify(summary));

    return Response.json(summary);
  }
};

We also enabled Hyperdrive’s cache logging by setting the HYPERDRIVE_DEBUG environment variable in our wrangler.toml:

1
2
[vars]
HYPERDRIVE_DEBUG = "true"

After redeploying and reproducing the issue a few times, the logs revealed something interesting. On the requests where stale data was returned, the log showed:

1
2
3
[hyperdrive] cache HIT for query: SELECT category, SUM(amount) as total FROM transactions GROUP BY category
[hyperdrive] cache key: sha256:a3f2e1d8... params: []
[hyperdrive] serving cached result, age: 73s

A cache hit with an age of 73 seconds — right after an INSERT. The cache invalidation wasn’t firing.

Step 3: Digging Into Cache Invalidation

This is where we had to understand how Hyperdrive’s cache invalidation actually works under the hood.

Hyperdrive tracks write operations per connection session. When you use withSession() on your database binding, Hyperdrive creates a session that tracks all reads and writes together. If a write happens in the same session as a subsequent read, Hyperdrive knows to skip the cache for that read.

But here’s the catch: session tracking is per-isolate. Cloudflare Workers run on V8 isolates, and each isolate has its own session state. If your write and read happen on the same request (and therefore the same isolate), invalidation works. But if they happen on different requests that land on different isolates, the session context doesn’t carry over.

In our case, the intermittent nature of the bug was because of load balancing across isolates. About 70% of the time, the insert and the subsequent dashboard refresh landed on the same isolate, and invalidation worked. The other 30% of the time, the refresh landed on a different isolate that still had the old cached result.

This wasn’t a bug in our code. It was a limitation of how Hyperdrive’s session-based cache invalidation interacts with Cloudflare’s distributed isolate model.

Step 4: The Fix

We explored several approaches. Here’s what we tried and what actually worked.

Attempt 1: Disable Caching Entirely

We set maxAge: 0 in our Hyperdrive config:

1
2
3
4
[[hyperdrive]]
binding = "DB"
id = "your-hyperdrive-id"
cache = false

This fixed the stale data problem immediately. Every query went straight to MySQL. But it also killed the performance benefit. Our dashboard latency went from ~40ms to ~180ms because every request had to traverse the Cloudflare Tunnel to our home server.

Not ideal.

Attempt 2: Manual Cache Busting With Query Parameters

Hyperdrive’s cache key includes the query text and parameters. If you change the query text even slightly, it’s a cache miss. So we tried appending a timestamp parameter to force a cache miss after writes:

1
2
3
4
5
// After an insert, force cache miss by adding a nonce
const nonce = Date.now();
const summary = await db.execute(
  `SELECT category, SUM(amount) as total FROM transactions GROUP BY category /* ${nonce} */`
);

This worked, but it felt hacky. It also meant we were busting the cache on every read-after-write, even when the write didn’t affect the data being read. And the comment-based approach felt fragile — what if Hyperdrive changed how it normalized query text?

Attempt 3: Use `withSession()` Correctly (The Real Fix)

After more digging through Cloudflare’s documentation and some GitHub issues, we realized we weren’t using session mode correctly across our full request lifecycle.

The key insight: you need to use withSession() for the entire request flow, not just individual queries. And you need to pass the session through consistently.

Here’s the corrected pattern:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
export default {
  async fetch(request, env) {
    // Create a session for this entire request
    const db = env.DB.withSession();

    const url = new URL(request.url);

    if (url.pathname === '/api/transactions' && request.method === 'POST') {
      const body = await request.json();

      // Write within the session
      await db.execute(
        'INSERT INTO transactions (amount, category, date) VALUES (?, ?, ?)',
        [body.amount, body.category, body.date]
      );

      // Read within the same session — Hyperdrive will skip cache
      const summary = await db.execute(
        'SELECT category, SUM(amount) as total FROM transactions GROUP BY category'
      );

      return Response.json({ success: true, summary });
    }

    // For pure read requests, use a fresh session without write tracking
    if (url.pathname === '/api/dashboard' && request.method === 'GET') {
      const readDb = env.DB.withSession();
      const summary = await readDb.execute(
        'SELECT category, SUM(amount) as total FROM transactions GROUP BY category'
      );
      return Response.json(summary);
    }

    return new Response('Not Found', { status: 404 });
  }
};

But this still didn’t fully solve the cross-isolate problem. If the POST and the GET happened on different isolates (which they usually do, since they’re separate HTTP requests), the GET could still hit a stale cache.

The Final Solution: Targeted Cache Invalidation With `cacheTtl`

The real fix was a combination of two things:

Use withSession() for write-then-read flows within a single request (as shown above).
For cross-request reads after writes, use a shorter, smarter cacheTtl and accept brief staleness for non-critical reads.

We restructured our caching strategy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
// Critical reads (right after a write) — no cache
const db = env.DB.withSession();
const freshData = await db.execute({
  sql: 'SELECT category, SUM(amount) as total FROM transactions GROUP BY category',
  cacheTtl: 0  // Force this query to skip cache
});

// Non-critical reads (dashboard refreshes) — short cache
const cachedDb = env.DB.withSession();
const cachedData = await cachedDb.execute({
  sql: 'SELECT category, SUM(amount) as total FROM transactions GROUP BY category',
  cacheTtl: 15  // Cache for 15 seconds
});

By setting cacheTtl: 0 on reads that happen immediately after writes, we guaranteed fresh data. For general dashboard browsing, a 15-second cache TTL was acceptable — if someone’s just poking around, a 15-second delay in seeing a new transaction is fine.

We also added a client-side cache-busting mechanism for the critical flow:

1
2
// Frontend: after adding a transaction, add a cache-bust param
const response = await fetch(`/api/dashboard?_t=${Date.now()}`);

This ensured the browser never served a cached HTTP response for the immediate post-write refresh.

Step 5: Verifying The Fix

After deploying the changes, we ran a test script that simulated the write-then-read flow 100 times:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#!/bin/bash
# test_hyperdrive_cache.sh

FAILURES=0

for i in $(seq 1 100); do
  # Insert a unique transaction
  AMOUNT=$((RANDOM % 1000))
  curl -s -X POST https://api.example.com/api/transactions \
    -H "Content-Type: application/json" \
    -d "{\"amount\": $AMOUNT, \"category\": \"test\", \"date\": \"2024-01-01\"}"

  # Immediately read the dashboard
  RESULT=$(curl -s "https://api.example.com/api/dashboard?_t=$(date +%s%N)")

  # Check if the amount appears in the response
  if echo "$RESULT" | grep -q "\"total\":.*$AMOUNT"; then
    echo "Test $i: PASS"
  else
    echo "Test $i: FAIL (amount $AMOUNT not found)"
    FAILURES=$((FAILURES + 1))
  fi
done

echo "Total failures: $FAILURES / 100"

Results: 0 failures out of 100. The fix held.

Lessons Learned

This bug taught us several things about Hyperdrive and caching in general:

Cache invalidation is hard. This is a well-known computer science problem, and Hyperdrive doesn’t magically solve it. It gives you tools (withSession(), cacheTtl), but you have to use them correctly.
Intermittent bugs are usually concurrency or distribution issues. The fact that it only happened 30% of the time was the clue that pointed to isolate-level load balancing.
Read the docs carefully. The withSession() behavior is documented, but the implications for cross-isolate requests aren’t obvious until you hit them.
Test your cache behavior explicitly. Don’t assume that “it worked in dev” means caching will behave correctly in production with real traffic patterns.

Summary

The rare Hyperdrive MySQL caching bug we encountered was caused by session-based cache invalidation not carrying across Cloudflare Workers isolates. When a write and a subsequent read landed on different isolates, the read could return stale cached data.

The fix was a combination of:

Using withSession() consistently for write-then-read flows within a single request
Setting cacheTtl: 0 for critical reads that must reflect recent writes
Using shorter cache TTLs (15 seconds) for non-critical dashboard reads
Adding client-side cache-busting for immediate post-write refreshes

If you’re using Hyperdrive with MySQL and seeing intermittent stale data after writes, check your session usage and cache TTL configuration. It’s probably not a bug in your database — it’s the cache layer doing exactly what it was configured to do, even if that’s not what you wanted.

Have you run into caching weirdness with Hyperdrive or any other database proxy? We’d love to hear about it. Drop a comment on the video, join our Discord, or hit us up on Mastodon. And if this article saved you a few hours of debugging, consider subscribing to the channel — we’re always digging into the weird corners of self-hosting so you don’t have to.

Until next time, keep your containers running and your caches fresh.

Inside Hyperdrive: How We Solved A Rare Hyperdrive MySQL Caching Bug#

The Setup: What We Were Running#

The Symptoms: Ghost Data And Phantom Updates#

Step 1: Ruling Out The Obvious#

Step 2: Enabling Hyperdrive Debug Logging#

Step 3: Digging Into Cache Invalidation#

Step 4: The Fix#

Attempt 1: Disable Caching Entirely#

Attempt 2: Manual Cache Busting With Query Parameters#

Attempt 3: Use withSession() Correctly (The Real Fix)#

The Final Solution: Targeted Cache Invalidation With cacheTtl#

Step 5: Verifying The Fix#

Lessons Learned#

Summary#