Retry Budgets Have Been the Fix for a Decade. GitHub's 8-Hour Outage Shows Why Teams Still Skip Them.

Two capacity outages, eleven days apart

On August 6, 2026, GitHub Actions began delaying workflow runs across the platform. GitHub's postmortem traced it to an internal Actions service that ran out of memory under load, delaying roughly two percent of workflows before engineers restored capacity.

Eleven days later, on August 17, a far larger failure took down Issues, Pull Requests, the API, Actions and Copilot for seven hours and forty-seven minutes. What stretched a capacity problem into an outage that long was, in large part, a missing retry budget: the reliability pattern that caps how much retry traffic a struggling service is allowed to absorb before it fails fast instead. Peak error rates reached close to twenty percent for web and API traffic, and around fifty percent for archive and raw repository downloads.

GitHub’s own account of the incident reads like a different story than the August 6 delay if you only skim the headlines. Read both postmortems back to back and they turn out to be the same failure, twice. A service ran out of headroom, and the systems built to handle that gracefully made it worse instead. That repetition, eleven days apart, is the actual story. Not that a platform this size had an outage, but that the fix engineers reach for by instinct wasn't the fix that would have prevented either one.

What broke on August 17: a concurrency limit nobody was watching

The immediate cause was network saturation on load balancers in GitHub's Central US facility, triggered when a service mesh sidecar hit its configured concurrency limit. The monitoring policy in place watched the host service, not the sidecar sitting in front of it, so there was no alert when the limit was reached, only a growing wave of failed requests once it had been.

The limit wasn't reached by accident. Monthly commit volume had grown from 1.4 billion in April to 2.9 billion by August, a near-doubling in four months. That's a genuine capacity problem, and on its own it's a fairly ordinary one. Provisioning lagged demand, and something eventually gave. The August 6 incident was a different mechanism entirely, a service running out of memory rather than a sidecar hitting a connection ceiling, but it belongs to the same category of failure: a limit that existed but wasn't instrumented, reached under load nobody had modelled for.

What made August 17 a multi-hour outage rather than a brief blip is what happened next. Once the sidecar started rejecting connections, legitimate retries from browsers, IDEs and CI runners kicked in, exactly as they were built to. Copilot's client, hitting the same saturated path, entered a retry loop of its own. GitHub's team described the result as roughly a tenfold surge in traffic during the recovery window, generated almost entirely by clients trying to recover from the outage and landing on top of the outage itself.

The retry storm: backoff changes timing, not volume

Exponential backoff with jitter exists to solve a specific coordination problem. Without it, every client hit by the same failure retries on roughly the same schedule, so the retries arrive as synchronised waves that repeatedly overwhelm a service just as it starts to recover. Backoff spreads the waiting time out; jitter randomises it further so clients stop resynchronising with each other. Both are cheap to add, a few lines around an HTTP call, and they solve the coordination problem well.

What they don't solve is volume. A client configured with backoff and jitter still retries. It just does so a little later, and a little less predictably, than a client without them. If ten thousand clients are each configured to retry up to five times, backoff changes when those fifty thousand retries arrive. It does not change that fifty thousand retries are coming. Against a dependency that's failing because it's out of capacity, arrival timing is the wrong lever to pull.

GitHub's team went as far as disabling authentication retries mid-incident, because the retries were adding load to the exact system the recovery depended on. That's a manual, mid-outage override standing in for a control that should have been automatic.

Why every client library ships backoff, and almost none ship a budget

The retry budget pattern caps the ratio of retry traffic to original request traffic. A service that allows a twenty percent budget can spend at most one retry for every five original requests before further failures fail fast instead of retrying again. The idea has been documented in reliability engineering practice for roughly a decade. Implementing it is a different matter, because it requires tracking that ratio across a rolling window, which means shared state across calls, more code than adding a single sleep-and-retry line.

Backoff is a per-request decision: a client checks only its own last call and decides whether to wait and try again. A budget is a population decision: it requires knowing what share of all recent traffic was already retries, which means either a shared counter, a central rate limiter, or a client library that tracks its own history across calls. That's a meaningfully larger lift, and the gap it leaves doesn't show up in a load test. It only shows up under exactly the conditions GitHub hit on August 17, a downstream dependency in trouble, and every client independently deciding to keep trying.

There's an organisational reason this gap persists, not just a technical one. Backoff lives entirely inside one client, so any engineer can add it to a single HTTP wrapper in an afternoon without asking anyone. A retry budget only works if it's consistent across every client hitting a given dependency, which means someone has to own the shared library, the rolling counters, and the alerting when a budget starts running dry. That's a platform team's job, not an individual contributor's, and platform teams have a long backlog of things that look more urgent than a control that only matters once every few years.

The popular general-purpose HTTP libraries make the split visible if you go and check. Python's urllib3 ships a Retry class with attempt counts and a backoff factor built in, out of the box. Node's axios-retry does the same for axios calls, with a configurable delay function. Neither ships anything resembling a shared budget, because a budget needs state that outlives a single request object, and general-purpose libraries are deliberately built to avoid owning that kind of shared state. Teams get budgets, when they get them at all, from a service mesh layer like Envoy or Istio, or from an RPC framework like gRPC that was built with the whole fleet in mind rather than one call at a time. If a system talks to its dependencies over plain HTTP through a general-purpose client, the odds that a budget exists anywhere in that path are low, regardless of how careful the backoff configuration looks.

The retry budget pattern, concretely

The core of it is small enough to sketch in a few dozen lines. A budget tracks two rolling counts, original requests and retries, over a fixed window, and only permits a retry while the retry count stays under a set ratio of the original count:

import time

class RetryBudget:
    def __init__(self, ratio=0.2, window_seconds=10):
        self.ratio = ratio
        self.window_seconds = window_seconds
        self.originals = []
        self.retries = []

    def _prune(self, bucket):
        cutoff = time.time() - self.window_seconds
        while bucket and bucket[0] < cutoff:
            bucket.pop(0)

    def record_original(self):
        self.originals.append(time.time())

    def can_retry(self):
        self._prune(self.originals)
        self._prune(self.retries)
        allowance = len(self.originals) * self.ratio
        return len(self.retries) < allowance

    def record_retry(self):
        self.retries.append(time.time())


def call_with_budget(fn, budget, max_attempts=3):
    budget.record_original()
    for attempt in range(max_attempts):
        try:
            return fn()
        except TransientError:
            if attempt + 1 >= max_attempts or not budget.can_retry():
                raise
            budget.record_retry()
            time.sleep(backoff_delay(attempt))

Production systems track this ratio per downstream dependency rather than globally, and most teams reach for an existing implementation, gRPC's retry throttling, Envoy's retry budget filter, Finagle's failure accrual, rather than hand-rolling it. The point isn't the code above; it's that can_retry() is a check most retry wrappers simply don't have. They check whether a single call has retried too many times, which is a per-call limit. They don't check whether the service on the other end has already absorbed too many retries from everyone, which is a population limit. Both checks matter, and only the first one is standard practice.

Testing for this before it becomes a postmortem

A load test proves a service can handle expected traffic. It rarely proves anything about retry behaviour, because nothing in a load test is actually failing. The useful test is different: hold traffic roughly constant, and inject a fixed rate of errors or added latency into one downstream dependency, then watch what happens to total request volume against that dependency, not just to the error rate individual clients report.

Four questions are worth answering before the next incident does it for you.

• Does the retry wrapper enforce a ceiling as a share of traffic, or only a count of attempts per call?
• Is any budget that exists scoped per dependency, or shared globally across everything a service calls?
• When the budget is exhausted, does the client fail fast, queue, or silently keep retrying anyway?
• Is anyone paged when a budget is running low, since that is a leading indicator of an incident already in progress?

Most teams can answer the first question by reading one file. Very few can answer the fourth without checking, and that's usually the honest measure of how prepared a system is for a dependency that's degraded rather than fully down.

What GitHub's fix list tells you to check this week

GitHub's own remediation list includes consistent retry limits, retry budgets, and variable timeouts across service-to-service calls, a tacit admission that these weren't consistently in place beforehand. The table below is the gap between what most teams already have and what would actually have capped the damage.

Most incident reviews stop at 'we added a circuit breaker', because breakers are simple to reason about: the dependency is down, so stop calling it. Retry budgets are less visible in roughly the way seatbelts are less visible than airbags. Nobody notices them working, because the alternative would have been a much longer paragraph in the postmortem.

Growth outpacing capacity isn't the interesting part

Every platform at this scale eventually has a quarter where usage growth outruns provisioned capacity. GitHub's near-doubling of monthly commit volume since April is not, on its own, a story. Capacity misses are common and usually survivable. What turns a capacity miss into a seven-hour outage is what the surrounding systems do once the miss happens, and on August 17 the answer was: retry, then retry again, at every layer, until the retries were doing more damage than the original failure.

If a client's retry logic has never been tested against a dependency that's actually struggling, not fully down, not fully up, just slow and shedding load, there's no way to know yet whether it behaves like backoff or like a budget. The cheaper way to find out is to read the retry code before the incident, not during the postmortem.