Saturation at GitHub: the saga continues – Surfing Complexity

Open original

GitHub's September 13 incident stemmed from a background data-cleanup job that overwhelmed a shared database cluster because the safeguard only monitored replica lag, not the primary's connection limit. Long request timeouts and retry loops made recovery harder and exposed how offline jobs can silently tank online systems.

  • The safeguard watched the wrong metric (replica lag stayed healthy while the primary hit its connection limit), showing that health signals must track the actual bottleneck, not just a proxy.
  • Long timeouts are invisible failures: they don't surface during normal operation like short timeouts do, but instead accumulate in-flight requests that starve the system of resources.
  • Offline jobs against shared online databases need rate-limiting and load-shedding controls built in advance—you can't improvise this during an incident when you're already saturated.
  • The fixes (rate-limits, better monitoring, request timeouts, database sharding) all add complexity, which is the eternal tradeoff of preventing incidents.

GitHub's September 13 incident started when a data-cleanup job began writing to a shared database cluster that handles permission checks on nearly every authenticated request. The safeguard monitoring this job only watched replica lag, which stayed low the whole time—but the primary database itself was quietly filling up with connections. When it hit max_connections, new requests couldn't connect and didn't fail fast because the timeouts were too long. These blocked requests piled up in web servers, consuming resources until the whole system degraded. To make it worse, a token creation retry loop kept hammering the already-saturated primary, preventing recovery. It took from 7:33 AM until 8:50 AM just to declare an incident, then another two hours to recover by shedding load and pausing the job. The author highlights three patterns worth understanding. First, cleanup jobs are inherently risky—you can cause outages by cleaning up data that's still in use, but you also risk incidents by not cleaning up unused data. It's pure tradeoff. Second, long timeouts are a hidden danger. Short timeouts announce themselves through excessive failures during normal operation, so you fix them. Long timeouts silently accumulate in-flight requests until they saturate your service, and you usually don't see the problem until an incident happens. Third, when systems saturate, the immediate priority is reducing load, not finding root cause. GitHub's responders couldn't quickly identify what was overloading the database, so they had to block queries and pause jobs manually. GitHub's fixes address these gaps directly: rate-limiting background jobs against shared databases, monitoring primary server load instead of just replica lag, showing running background work on dashboards so responders can pause it instantly, bounding retry loops, adding request-level timeouts, and breaking apart the database cluster so one problem doesn't take down everything. The author's point is blunt—your systems will eventually overload. The time to build the tooling and safeguards to handle that gracefully is before you're in the middle of an incident.

Ask about this article