Inside GitHub’s Seven-Hour Outage: How a Hidden Bottleneck Cascaded Into a Global Slowdown

On August 17, developers around the world watched pull requests stall, sign-ins fail, and Copilot suggestions time out — not because anyone shipped broken code, but because a piece of infrastructure quietly ran out of room. The incident lasted 7 hours and 47 minutes, and by the time GitHub's status page finally turned green, it had touched authentication, Actions, APIs, issues, pull requests, and its AI coding assistant. For a platform that hosts the daily work of millions of engineers, that is not a minor hiccup. It is a reminder that "the cloud" is really a chain of interdependent machinery, and chains break at their weakest link.

A server room view illustrating a GitHub outage caused by a hidden bottleneck in cloud infrastructure

A capacity problem, not a coding mistake

GitHub’s own account is direct about what didn’t happen: this wasn’t triggered by a bad code push or a misconfigured setting. Traffic hit a new peak, and a critical infrastructure component in the company’s Central US data center failed to scale with it, and the resulting pressure spread outward into authentication and other services. The company frames both this outage and an earlier one on August 6 as capacity failures — cases where growth outran the systems built to absorb it. Monthly commits on the platform have roughly doubled since April, climbing from 1.4 billion to 2.9 billion, and that kind of growth curve is exactly the sort of thing that turns "comfortable headroom" into "just enough" almost overnight.

But "not enough capacity" is a description, not an explanation. The more useful question is where, specifically, the platform ran out of room — and why the shortage in one place turned into a slowdown almost everywhere else.

Following the chain: sidecar, gateway, authentication

According to a detailed technical account of the incident, the trouble started with small helper processes called sidecars, which the Istio service mesh runs alongside GitHub’s core services to manage traffic between them. Service meshes exist to give large platforms more control over how internal requests are routed and observed — but that control comes at the cost of an extra layer that itself has to survive under load. In this case, the sidecars hit their own concurrency limits, yet the autoscaling system didn’t add capacity for them, because it was watching metrics on the main service, not on the sidecar layer itself. Autoscaling only works if it is watching the part of the system that’s actually running out of room; when the real bottleneck sits somewhere the monitoring doesn’t reach, the system can look fine on the dashboard while quietly choking underneath.

From there, the strain moved one layer up. Four HAProxy nodes — load balancers that route traffic to the right backend — exhausted their connection limits, which degraded the gateway path used for authentication and produced widespread sign-in failures. That detail matters because authentication is a gatekeeper: pull requests, Actions runs, API calls, and Copilot sessions all need to confirm who’s asking before anything else can happen. When the gatekeeper slows down, everything behind it backs up too, even features that had nothing to do with the original failure.

Here is the sequence in simplified form:

Stage What failed Why it mattered
Sidecar autoscaling Istio sidecars hit concurrency limits unnoticed by scaling policy Bottleneck formed where monitoring wasn’t looking
Load balancer saturation Four HAProxy nodes exhausted flow limits Degraded the gateway path used for sign-in
Authentication failures Sign-in and token requests slowed or failed Blocked dependent features platform-wide
Retry amplification Failed requests were automatically resent Added load on top of an already strained system
Staged recovery Core services restored in phases through the day Some components, notably Copilot, lagged behind

Why retries can turn a slowdown into a storm

Modern software is full of automatic retry logic, and for good reason: a brief network hiccup shouldn’t require a human to intervene. But retries assume the system asking for a redo has some spare capacity to handle it. When that assumption fails, retries stop being a safety net and start being fuel.

flowchart TD
 A[Requests fail under load] --> B[Client retries automatically]
 B --> C[Retry traffic adds to existing load]
 C --> D[System gets busier, not calmer]
 D --> A

That loop is what engineers call a retry storm, and it’s a big part of why this outage dragged on so long. GitHub reported that optimistic retry logic piled extra load onto internal load balancers exactly while those balancers were already struggling. Pausing the affected HAProxy nodes reportedly brought a broad, near-immediate improvement — evidence that the retries themselves, not just the original shortage, were prolonging the pain.

The Copilot tail: an old bug meets a new bottleneck

Most GitHub services recovered in stages through the afternoon, but Copilot lagged noticeably behind. GitHub’s own postmortem says errors in Copilot-related services triggered a client-side retry loop that increased traffic during recovery, and that this behavior had to be mitigated before traffic could be safely restored. The more detailed account fills in the mechanism: when some traffic was shifted to another data center during recovery, delayed replies from an internal service exposed a longstanding retry bug in the VS Code client, and that bug amplified Copilot token traffic roughly tenfold, pushing request rates from a normal few thousand per second into the tens of thousands. In other words, the assistant’s slow return wasn’t a separate AI failure — it was the authentication and retry problem finding one more place to hide.

That distinction is worth sitting with, because it illustrates a broader truth about recovery: the status page turning green for one component doesn’t mean the whole platform is healthy. One lingering bottleneck — in this case, a feedback loop between a struggling backend and an over-eager client — can outlast the first wave of "resolved" signals for hours. Independent tracking of the incident timeline noted exactly this pattern: core functionality largely stabilized while authentication issues tied to Copilot persisted well past that point, with the incident status remaining unresolved even as most services looked fine.

It’s worth being cautious here about how far the numbers travel. Reported error rates — around 20% for web and API traffic, and closer to 50% for archive downloads and raw file retrieval — described specific paths at specific moments, not a blanket failure of half of all Git operations. Lumping distinct measurements together overstates the incident’s scope in ways the underlying data doesn’t support.

The real lesson: design has to match demand

It’s tempting to read this as a story about needing more servers, and GitHub has indeed added millions of CPU cores and moved a growing share of its platform load onto Azure infrastructure. But raw capacity was only part of the picture. The sidecar layer had headroom problems invisible to its own autoscaler; the retry logic meant to help recovery instead worked against it; and a chain of dependencies meant a bottleneck in one obscure layer could surface as an authentication outage for everyone. GitHub says it is now applying consistent retry limits and timeouts across internal services and reviewing alerts that could miss the next spike — sensible steps, though the company itself treats this as ongoing work rather than a finished fix.

The takeaway for anyone trying to make sense of modern developer platforms isn’t that GitHub is uniquely fragile. It’s that a service built from authentication, meshes, load balancers, automation, and AI features is only as resilient as its least-watched layer — and that watching the right layer, not just adding more machines, is what determines whether growth breaks a system or simply gets absorbed by it.

Sources

  1. The August 17 outage, and the work ahead
  2. GitHub’s 7-Hour 47-Minute Outage Traced to a Capacity Shortfall, With Retry Storms Delaying Copilot Recovery | claypier
Scroll to Top