Capacity Is Architecture: What GitHub’s Outage Actually Exposed
🎯 Hook
GitHub’s August 17 outage lasted 7 hours and 47 minutes. It affected github.com, authentication, Actions, APIs, pull requests, issues, and Copilot.
The stated trigger was not a bad deploy. A new traffic peak hit a critical component in Central US that did not scale with demand. During recovery, Copilot client retries created more traffic and had to be mitigated before GitHub could safely restore service.
That makes this more interesting than a postmortem about one broken machine. It is a question about architecture: when volume doubles, can your system absorb it, isolate it, and recover without its own clients making things worse?
🔥 Hot Take
💬 My take: We should not say that “GitHub’s architecture failed” as if the public incident report proved a single bad design choice. GitHub has not named the failed component. But the incident does show an architectural gap: a critical path could run out of capacity, the pressure could cascade into authentication and unrelated workflows, and retries could amplify the recovery load. That is not a hardware problem alone. It is capacity planning, dependency design, and overload control failing together.
The hard lesson is that scale is not something an architecture gains by adding more nodes after demand arrives. The architecture must already have headroom, independent failure domains, and a defined degraded mode.
📰 What we know about GitHub’s architecture
GitHub’s public engineering posts describe a platform in transition, not a static system.
- GitHub.com is historically a large Ruby on Rails monolith backed by MySQL. Core data such as users, repositories, issues, and pull requests once shared a major database cluster. GitHub has spent years partitioning those data domains to reduce shared failure risk.
- In 2026 it has been moving workload to Azure for elastic capacity while extracting services from the monolith. By July, more than half of monolith read traffic and 47% of Git traffic were running in Azure Central US; 29% of repositories had a second replica there.
- GitHub was also separating users, authentication, and authorization from the oldest shared database path. By July, the dedicated users service was offloading more than one million queries per second at peak.
- The August post says Azure was carrying roughly 58% of platform load and half of Git operations. A second Azure region was still described as future work for regional resilience.
This matters because a migration can improve the destination while temporarily increasing the operational burden: capacity must be proven at every ramp step, traffic must be steerable, and old and new dependencies must not share the same blast radius.
❌ What failed — and what we cannot claim
1. Capacity headroom was insufficient
GitHub says traffic reached a new peak and a critical Central US component failed to scale. Monthly commits had grown from 1.4 billion in April to 2.9 billion by August.
The failure is not that the forecast missed an exact number. Forecasts always miss. The failure is having insufficient tested headroom and no safe overload posture when the forecast is wrong.
2. A critical dependency still had too much blast radius
Authentication failures disrupted multiple services. That is exactly the class of shared dependency GitHub had already been working to split apart.
The public evidence does not identify the failed component as MySQL, the Rails monolith, Azure, or a load balancer. Naming one would be speculation. What is supported is the outcome: one capacity failure propagated beyond its local boundary.
3. Recovery behavior was not bounded
Copilot errors triggered a client-side retry loop that increased traffic during recovery. A retry without a budget is just distributed load amplification.
The service can be healthy enough to start recovering and still fail again because every client retries at once. That is an architecture problem at the interaction boundary, not a Copilot-only bug.
4. The platform was not yet region-resilient
GitHub’s July report explicitly frames a second Azure region as the next foundation for withstanding the loss of a region. Moving traffic to one Azure region improves elastic capacity, but it is not the same as proving that a regional failure, or a regional capacity bottleneck, can be absorbed elsewhere.
🛠️ What a better design would do
There is no one magic architecture. The highest-leverage options are layered:
- Capacity as a release gate. Model peak and burst load per critical dependency, retain explicit headroom, and prove it with production-like load tests before every traffic ramp. GitHub’s own per-turnup stability gates are the correct direction.
- Isolate critical domains. Keep identity, authorization, repository metadata, Git content, and asynchronous work queues independently scalable where possible. A busy PR workflow should not consume the same last-resort capacity as sign-in.
- Graceful degradation and load shedding. Preserve write paths, authentication, and core Git operations first. Shed low-priority reads, expensive queries, and optional features before queues and databases collapse.
- Bound retries everywhere. Use exponential backoff with jitter, retry limits, deadlines, circuit breakers, and per-service retry budgets. Retries should consume a controlled slice of capacity, never all remaining capacity.
- Multi-region failover that is exercised, not merely diagrammed. Maintain independent capacity in another region, automate traffic steering, and run failure drills. Replicas and failover plans only count after they survive a real test.
- Measure customer workflows, not just host health. CPU and memory can look acceptable while pull requests or authentication are failing. Alert on the SLOs users feel: login success, Git fetch/push success, PR read/write latency, and Actions queue delay.
💡 Dev Tip of the Week
Write retry policy as an API contract, not as a client implementation detail.
// Retriable calls need a deadline, capped attempts, exponential backoff,
// jitter, and a circuit breaker / retry budget owned by the service boundary.
// "Retry until it works" transfers failure into the next dependency.
Why it matters: A client retry can turn a partial outage into a full one. If a dependency is unhealthy, the safest request is usually the one you do not send.
🤔 Community Question
❓ When traffic grows faster than expected, what do you protect first in your system: correctness, availability of core writes, or every feature equally? And have you tested that choice under failure rather than only documented it?