Hyperdrive MySQL Caching Bug: A Deep Dive into Protocol Mismatches in Distributed Connection Pooling

Understanding Hyperdrive's Architecture

Hyperdrive is a distributed connection pooler for SQL databases that optimizes serverless compute platforms like Cloudflare Workers. It maintains warm SQL connections across Cloudflare's global network, sharing them among many Worker requests. This is crucial for performance and resource management, as establishing new SQL connections is inherently expensive. Beyond connection pooling, Hyperdrive also incorporates query caching using Cloudflare's global CDN cache to further reduce latency and origin database load.

Split Node Architecture and Caching

Hyperdrive employs a split architecture comprising two main nodes: the Client node and the Endpoint node. The Client node co-locates with the user's Worker, handling local caching and forwarding queries. The Endpoint node, positioned close to the origin database, manages the actual connection pool. Both nodes leverage the local Cloudflare CDN cache. A key architectural decision was allowing these nodes to share a cache when co-located in the same data center, which generally improves performance by allowing either node to serve cached responses directly.

The Caching Bug: Protocol Capability Mismatch

The core of the bug lay in a faulty assumption within Hyperdrive Client's caching logic: that it could cache the exact bytes written back to the Worker. While valid for Postgres, this assumption broke down with MySQL due to differing protocol capabilities. Specifically, the `CLIENT_DEPRECATE_EOF` capability, which alters how result sets are terminated, was often not supported by Worker's MySQL drivers. Hyperdrive's Client node would translate origin responses to match the Worker's expected capabilities (e.g., adding EOF packets), and then cache these *translated* bytes. However, the Endpoint node always operates with Hyperdrive's own fixed set of capabilities. This meant a response cached by one node could be in an incompatible format for the other, leading to parsing errors and empty results when the shared cache was utilized.

Failure Modes

• Endpoint caches, Client reads: Endpoint caches a response encoded with Hyperdrive's capabilities. A Client then reads this from the shared cache and returns it directly to the Worker *without* translating it to the Worker's expected capabilities.
• Client caches, Endpoint reads: Client caches a response that has already been translated to the Worker's capabilities. An Endpoint then reads this and, expecting Hyperdrive-encoded bytes, passes it to another Client, which attempts further translation, leading to errors or hangs.

The fix involved ensuring that all query result bytes stored in the shared cache are *always* in a canonical version, encoded using Hyperdrive's capabilities, regardless of which node initiated the cache write. This eliminated the capability mismatch when reading from the shared cache, resolving the empty row issue and improving reliability.