
A capacity problem, not a coding mistake
GitHub’s own account is direct about what didn’t happen: this wasn’t triggered by a bad code push or a misconfigured setting. Traffic hit a new peak, and a critical infrastructure component in the company’s Central US data center failed to scale with it, and the resulting pressure spread outward into authentication and other services. The company frames both this outage and an earlier one on August 6 as capacity failures — cases where growth outran the systems built to absorb it. Monthly commits on the platform have roughly doubled since April, climbing from 1.4 billion to 2.9 billion, and that kind of growth curve is exactly the sort of thing that turns "comfortable headroom" into "just enough" almost overnight.
But "not enough capacity" is a description, not an explanation. The more useful question is where, specifically, the platform ran out of room — and why the shortage in one place turned into a slowdown almost everywhere else.
Following the chain: sidecar, gateway, authentication
According to a detailed technical account of the incident, the trouble started with small helper processes called sidecars, which the Istio service mesh runs alongside GitHub’s core services to manage traffic between them. Service meshes exist to give large platforms more control over how internal requests are routed and observed — but that control comes at the cost of an extra layer that itself has to survive under load. In this case, the sidecars hit their own concurrency limits, yet the autoscaling system didn’t add capacity for them, because it was watching metrics on the main service, not on the sidecar layer itself. Autoscaling only works if it is watching the part of the system that’s actually running out of room; when the real bottleneck sits somewhere the monitoring doesn’t reach, the system can look fine on the dashboard while quietly choking underneath.
From there, the strain moved one layer up. Four HAProxy nodes — load balancers that route traffic to the right backend — exhausted their connection limits, which degraded the gateway path used for authentication and produced widespread sign-in failures. That detail matters because authentication is a gatekeeper: pull requests, Actions runs, API calls, and Copilot sessions all need to confirm who’s asking before anything else can happen. When the gatekeeper slows down, everything behind it backs up too, even features that had nothing to do with the original failure.
Here is the sequence in simplified form:
| Stage | What failed | Why it mattered |
|---|---|---|
| Sidecar autoscaling | Istio sidecars hit concurrency limits unnoticed by scaling policy | Bottleneck formed where monitoring wasn’t looking |
| Load balancer saturation | Four HAProxy nodes exhausted flow limits | Degraded the gateway path used for sign-in |
| Authentication failures | Sign-in and token requests slowed or failed | Blocked dependent features platform-wide |
| Retry amplification | Failed requests were automatically resent | Added load on top of an already strained system |
| Staged recovery | Core services restored in phases through the day | Some components, notably Copilot, lagged behind |
Why retries can turn a slowdown into a storm
Modern software is full of automatic retry logic, and for good reason: a brief network hiccup shouldn’t require a human to intervene. But retries assume the system asking for a redo has some spare capacity to handle it. When that assumption fails, retries stop being a safety net and start being fuel.
flowchart TD A[Requests fail under load] --> B[Client retries automatically] B --> C[Retry traffic adds to existing load] C --> D[System gets busier, not calmer] D --> A
That loop is what engineers call a retry storm, and it’s a big part of why this outage dragged on so long. GitHub reported that optimistic retry logic piled extra load onto internal load balancers exactly while those balancers were already struggling. Pausing the affected HAProxy nodes reportedly brought a broad, near-immediate improvement — evidence that the retries themselves, not just the original shortage, were prolonging the pain.
The Copilot tail: an old bug meets a new bottleneck
Most GitHub services recovered in stages through the afternoon, but Copilot lagged noticeably behind. GitHub’s own postmortem says errors in Copilot-related services triggered a client-side retry loop that increased traffic during recovery, and that this behavior had to be mitigated before traffic could be safely restored. The more detailed account fills in the mechanism: when some traffic was shifted to another data center during recovery, delayed replies from an internal service exposed a longstanding retry bug in the VS Code client, and that bug amplified Copilot token traffic roughly tenfold, pushing request rates from a normal few thousand per second into the tens of thousands. In other words, the assistant’s slow return wasn’t a separate AI failure — it was the authentication and retry problem finding one more place to hide.
That distinction is worth sitting with, because it illustrates a broader truth about recovery: the status page turning green for one component doesn’t mean the whole platform is healthy. One lingering bottleneck — in this case, a feedback loop between a struggling backend and an over-eager client — can outlast the first wave of "resolved" signals for hours. Independent tracking of the incident timeline noted exactly this pattern: core functionality largely stabilized while authentication issues tied to Copilot persisted well past that point, with the incident status remaining unresolved even as most services looked fine.
It’s worth being cautious here about how far the numbers travel. Reported error rates — around 20% for web and API traffic, and closer to 50% for archive downloads and raw file retrieval — described specific paths at specific moments, not a blanket failure of half of all Git operations. Lumping distinct measurements together overstates the incident’s scope in ways the underlying data doesn’t support.
The real lesson: design has to match demand
It’s tempting to read this as a story about needing more servers, and GitHub has indeed added millions of CPU cores and moved a growing share of its platform load onto Azure infrastructure. But raw capacity was only part of the picture. The sidecar layer had headroom problems invisible to its own autoscaler; the retry logic meant to help recovery instead worked against it; and a chain of dependencies meant a bottleneck in one obscure layer could surface as an authentication outage for everyone. GitHub says it is now applying consistent retry limits and timeouts across internal services and reviewing alerts that could miss the next spike — sensible steps, though the company itself treats this as ongoing work rather than a finished fix.
The takeaway for anyone trying to make sense of modern developer platforms isn’t that GitHub is uniquely fragile. It’s that a service built from authentication, meshes, load balancers, automation, and AI features is only as resilient as its least-watched layer — and that watching the right layer, not just adding more machines, is what determines whether growth breaks a system or simply gets absorbed by it.


