Coinbase's rollback hit a circular dependency: the tooling needed the gateway that had just failed

Coinbase's fix for its 14 July 2026 outage was a two-line rollback. It still took fifty minutes, because the standard path to deploying that fix ran straight into a circular dependency: the tooling engineers needed to reach was itself only reachable through the gateway that had just gone down.

The fix was simple. Reaching it wasn’t.

At 12:34pm ET on 14 July, Coinbase pushed what its own postmortem describes as a routine configuration change to a shared production Kubernetes cluster, part of an ongoing migration to a new service-deployment model. The change collided with resource names tied to the cluster's Istio ingress gateway, an undetected naming collision that altered configuration meant for something else entirely. Within minutes, transfers, card transactions and onchain services degraded across Coinbase's retail, institutional and developer platforms. No customer funds were ever at risk, but in-flight transactions stalled and debit-card authorisations started failing.

The actual fix, once engineers found it, wasn't hard: redeploy the ingress gateway to its last known-good version. Most teams could execute that kind of rollback in minutes on a normal day. This wasn't a normal day, because the network path to the deployment tooling that would push the rollback ran through the same ingress gateway that had just broken.

That's a circular dependency in the strict sense, not the loose one people use to mean “everything depends on everything.” Step one of the rollback required the gateway to be reachable. The gateway being unreachable was the entire incident. Engineers broke the loop by requesting just-in-time privileged access and invoking the rollback workflow directly through the cloud provider's console, sidestepping the broken path rather than fixing it in place. That detour, not the diagnosis and not the fix itself, is most of where the fifty minutes went.

It's worth noting when this loop was introduced: not through years of neglect, but through the migration itself. The ingress gateway was part of a new deployment model Coinbase was actively rolling out. New reachability paths get built and tested for the traffic they carry. They rarely get tested for whether they're also, quietly, the only path back if they fail. That's a detail worth sitting with, because it means the loop is most likely to appear exactly when a team is modernising, not when a team is coasting on old infrastructure.

The same circular dependency broke AWS’s control plane nine months earlier

Coinbase's incident looks like a one-off until you line it up against AWS's 20 October 2025 outage. That incident started as a DNS-resolution failure affecting the DynamoDB API endpoint in us-east-1, a regional problem on paper. In practice, us-east-1 also hosts a meaningful share of AWS's global control-plane services, IAM and STS among them. A DNS fault in one region cascaded into account-wide authentication failures for customers who had no workloads running in us-east-1 at all, because signing in, or minting a fresh API token, routed back through the affected region regardless of where the actual application lived.

Different company. Different stack: Kubernetes and Istio at Coinbase, DNS and IAM at AWS. Same shape of failure, nine months apart: the thing an operator, or in AWS’s case millions of downstream customers, needed to reach in order to respond to the outage sat downstream of the outage itself.

Redundancy doesn’t fix a dependency-direction problem

The instinctive response to either incident is “add more redundancy”: more nodes, more availability zones, more regions. That instinct misreads what actually failed. Coinbase’s cluster wasn’t a single point of compute failure. AWS’s IAM and STS are among the most heavily replicated systems on the internet. Neither company was short on redundancy. What both were short on was a recovery path that pointed somewhere other than the object that had just broken.

Power-grid engineers have a name for this problem, and a decades-old answer to it. After a total blackout, you cannot use grid power to restart the grid, because there isn’t any left to draw on. The standard fix is a black-start generator: a small diesel or hydro unit, deliberately kept independent of the grid it exists to revive, capable of starting cold with no external power at all. The independence is the entire point of the design; a black-start generator that quietly drew its ignition current from the grid would defeat its own purpose. Software incident response rarely gets built with that discipline. Rollback tooling tends to be engineered to be reachable, full stop, not to be reachable independently of the exact infrastructure it targets.

Three places this loop hides in a typical stack

Neither Coinbase nor AWS is unusual here; the pattern shows up in most organisations that haven’t specifically audited for it. Three places it hides most often:

• Deployment tooling network access. VPNs, bastion hosts and CI runners are commonly reachable only through the same load balancer, service mesh or VPC routing that fronts production. If that layer is what broke, the team’s own path to fix it just broke with it, exactly as it did at Coinbase. A CI runner that can only reach its deploy target through the production ingress is a smaller version of the same design.
• Authentication and DNS. Single sign-on, internal DNS resolvers and, worst of all, break-glass credentials are frequently hosted on the same identity provider and the same infrastructure as the production systems they’re meant to rescue. AWS’s outage showed the account-wide version of this: control-plane services concentrated in one region became a dependency for customers everywhere, including customers who had never deployed anything to that region.
• Status pages and incident communication. This one is old enough to have a well-known folk fix, which is precisely why dedicated status-page vendors exist that deliberately run on infrastructure separate from the product they report on. Teams that skip that separation and host their status page on the same cloud account, region or CDN as their product routinely find that the outage takes down the page telling customers about the outage. Same loop, aimed at communication instead of remediation.

Why postmortems rarely catch it

Five-whys analysis is good at finding proximate causes and bad at finding structural ones, because it stops as soon as it reaches an answer that's specific enough to act on. “a resource-name collision altered the ingress gateway config” is specific enough to act on: add a validation check, ship it, close the ticket. “our recovery path shares infrastructure with everything it recovers” is a much harder finding to act on, because the fix isn't a single line item, it's a design property that has to be verified across every tool in the incident response chain.

There is also a timing problem. The person best placed to notice the loop is whoever is stuck inside it during the incident, and that person is, by definition, busy trying to end the incident, not documenting the shape of the obstacle in their way. By the time the postmortem is written, the workaround already worked, the outage is over, and the natural focus shifts to the thing that started the chain of events rather than the thing that made the chain so long to unwind.

Auditing your own rollback path for the loop

This is auditable in an afternoon, and the audit is more useful than most tabletop exercises, because it asks a narrower, more falsifiable question than “what could go wrong”: for each documented step in your incident and rollback runbooks, what does executing this step require to be true, and could the incident you’re responding to make that false?

• Trace the network path your deployment tooling actually takes, not the path in the architecture diagram. Does it resolve through the same load balancer, mesh, or DNS zone as the production traffic it manages?
• Confirm break-glass credentials authenticate independently of your primary identity provider, on a separate account, with no SSO passthrough and no dependency on internal DNS.
• Confirm your status page and incident-communication tooling run on infrastructure separate from your product, ideally a different cloud provider and account entirely.
• Test it. Pick a chaos-engineering exercise where the “normal” path is deliberately cut, and have on-call execute the documented rollback exactly as written, no manual shortcuts allowed. If the runbook fails at step one, that’s the finding. It’s not a reason to write a better runbook next time; it’s a reason to fix the dependency before there is a next time.

The runbook works right up until it doesn’t

Most postmortems fix the proximate cause: Coinbase will presumably add validation against resource-name collisions, and AWS has already made changes to how DNS is managed for DynamoDB’s endpoints. Those fixes are necessary and almost never sufficient, because they answer “what broke” without asking whether the path back from broken survives the same failure. That second question doesn’t show up on an architecture diagram, and it rarely shows up in an on-call rotation until the day it’s the only question that matters. Auditing for it now costs an afternoon. Finding it live costs fifty minutes, if you’re fast.