One Cache to Rule Them All: Handling Responses and In-Flight Requests with Durable Objects

"In-flight work and completed responses can be treated as two states of the same cache entry, eliminating duplicate computations during cache misses. Per-key singleton routing, shared in-memory state, and serialized execution allow a single owner to safely coordinate both in-flight promises and cached results. This pattern helps reduce thundering-herd effects on cache misses, simplifies system design by avoiding distributed locks or polling, and preserves correctness under horizontal scaling."

"When multiple clients request the same resource at roughly the same time, a cache miss can trigger several identical computations in parallel. In a single-process JavaScript application, this is commonly mitigated by storing the in-flight Promise in memory so that subsequent callers can await the same result. In other languages and runtimes, similar effects may be achieved through different concurrency primitives, but the underlying assumption is the same: shared memory and a single execution context."

Completed responses and in-flight work can be unified as two states of a single cache entry to avoid duplicate computation on cache misses. Per-key singleton routing with shared in-memory state and serialized execution lets one owner coordinate promises and cached results safely. Serializing requests for very hot keys prevents thundering-herd effects and preserves correctness under horizontal scaling. The pattern maps well to actor-like runtimes such as Cloudflare Durable Objects, Akka, or Orleans and is difficult to reproduce with only stateless functions and eventually-consistent key-value stores. Production implementations require timeouts, retries, eviction policies, error handling, and optional persistence of completed responses.

Read at InfoQ