Things that I Only Learned After Scaling: Non-Obvious Lessons from Production

"Retries can cause a system to collapse under its own load. Without backoff and jitter, retries create thundering herds that turn a partial outage into a full-blown incident. Use exponential backoff with jitter. Distinguish between transient and fatal errors. Cap retries."

"Verbose logging at scale leads to I/O congestion and skyrocketing costs. Even structured logs can introduce latency. Log at the right level. Sample where appropriate. Avoid logs in hot paths."

"In-memory caches, sticky sessions, or local temp files often sneak in. These are invisible until you scale. Test node kill and failover scenarios. Make session state truly external."

"It's rare that a whole system goes down. One shard, AZ, or dependency fails-and it's just as painful. Monitor per-shard/AZ health. Design for degraded modes."

Scaling production systems exposes non-obvious failures absent from best-practice documentation. Uncontrolled retries create thundering herds that amplify outages; exponential backoff with jitter is essential. Verbose logging causes I/O congestion and latency at scale, requiring strategic sampling. Supposedly stateless services often contain hidden in-memory caches or sticky sessions that break during failover. Load balancer connection draining prevents request loss during deploys. Partial failures affecting single shards or availability zones are common and require per-component health monitoring. Shared resources like databases become bottlenecks as services scale independently. Metrics provide misleading signals without proper context, necessitating comprehensive observability beyond dashboard indicators.

Read at Medium