Clerk’s Failover Didn’t Fire Because Postgres Was “Technically Still Online.” That’s a Design Category, Not a Fluke.

What the On-Call Team Found, in the Order They Found It

On February 19, 2026, Clerk’s Postgres database spent roughly ninety minutes in a state its own postmortem describes as “technically still online, just degraded.” That phrase carries the whole story. It is the exact gap between the failure Clerk’s automatic failover was built to catch and the failure that actually happened.

The trigger was mundane. A routine Postgres auto-ANALYZE ran against a production table at 8:11 AM PST and flipped the execution plan for one frequently-run query. Database performance dropped immediately. Request handlers queued behind the now-slow query, and within minutes more than 95% of Clerk’s traffic was being shed with a 429.

The remaining slice of traffic, the requests that did reach the database, came back with a 200. Just an extremely slow one. Binary uptime checks had nothing to complain about: the database was accepting connections, executing queries, and returning successful responses the entire time.

Clerk’s published recovery timeline shows what that cost in diagnosis time. The incident was escalated at 16:15 UTC. For the next forty minutes the team chased a red herring: a customer with an unusually aggressive retry pattern, blocked manually at 16:35 UTC in a fix that did nothing, because the retries were a symptom of the slowdown, not its cause. Only at 16:55 UTC did the team find the structural problem: the automatic failover built specifically to route around a Postgres failure had never triggered, because Postgres had never technically failed.

The Number Behind It: 99.9996% Rounded Up to 100%

The mechanism is worth walking through, because it is almost comically small for the size of the outage it caused. Postgres’ query planner decides how to execute a query using statistics about the data it touches, statistics that go stale as rows are inserted and deleted. An auto-ANALYZE periodically refreshes them by sampling a subset of rows rather than reading the whole table.

For one column on the affected table, the true share of NULL values was 99.9996%. The sample the planner drew that morning happened to contain no non-null values at all, so it concluded the column was 100% NULL. Acting on that number, the planner assumed a specific query would return zero non-null rows for that filter, and planned accordingly. In production, the query returned more than 17,000 rows. The plan built for zero rows was catastrophically wrong for seventeen thousand, and the query ran frequently enough that the mismatch alone consumed most of the database’s available resources.

Clerk’s postmortem calls this a "query plan flip," and it is a well-known Postgres failure class. Statistics-based planning is a bet, and a bad sample makes the bet wrong in a way that looks nothing like a hardware failure or a crash. Nothing broke. The database simply started making a worse decision, correctly, every time the query ran.

Why the Failover Built for This Exact Scenario Didn’t Fire

Clerk already had a documented Session API failover, designed to activate during Postgres outages. On paper, this incident is exactly the scenario it existed for: the primary path to session data became unusable. In practice, the failover’s trigger condition was "is Postgres down," and Postgres was never down. It kept accepting connections and returning 200s throughout, just on a timescale measured in seconds instead of milliseconds. The condition that was supposed to fire the safety net never evaluated to true.

There is a second layer worth sitting with. Clerk had already built a newer, broader failover, one designed to trigger on any failure at the origin rather than specifically a Postgres outage, and it had been tested over the preceding months. It simply had not been wired up with automatic triggers in production yet, and the on-call team had not been trained to activate it manually under pressure. It worked once engineers enabled it by hand at 17:08 UTC, and the incident started resolving from there. The fix already existed. The gap was not engineering capability. It was that the trigger everyone had built their instincts around was still calibrated to the failure mode they already knew how to detect, not the one that actually showed up.

Even that manual fix was partial. Enabling the new failover restored access to customer applications for most users within two minutes, but sign-in and account-management endpoints, the paths that still needed a direct round trip to the degraded database, stayed unusable until the root cause was found and reversed twenty minutes later. A failover that routes around a dependency only protects the parts of the system that do not still, somewhere downstream, depend on it.

“Technically Still Online” Is a Design Category, Not a Fluke

It is tempting to read this as one unlucky statistics bug. The more useful reading is structural: any system whose failure detection reduces a dependency’s state to a boolean, up or down, reachable or not, has a blind spot shaped exactly like "degraded." A database that answers every ping and every connection attempt correctly, while taking minutes instead of milliseconds to run a query, will pass every check built around reachability and fail every expectation built around usability.

Clerk’s own history makes the contrast sharper. Three weeks earlier, on March 10, a failed live migration of the underlying database VM caused a separate outage, and that one was caught immediately: monitoring flagged elevated latency and error rates within moments, because the team was watching a continuous signal, not a binary one. The difference between the two incidents was not tooling maturity. It was which question each check had been built to answer.

The Same Blind Spot Shows Up Past the Database

Postgres is the specific dependency in this incident, but the design mistake is not specific to databases. A load balancer’s upstream health check, an API gateway’s backend probe, a CDN’s origin check: almost all of them, by default, ask a version of the same question Clerk’s failover asked. Did the upstream respond, yes or no, within some generous timeout. A backend that answers every request correctly but three times slower than usual will keep passing that check indefinitely, quietly consuming more of every caller’s request budget while every dashboard built around pass or fail stays green.

This is also why the fix cannot just be "add more health checks." A synchronous dependency ping added to a public-facing check has its own failure mode: run it too often against a struggling backend and the checking traffic itself becomes part of the load problem, arriving at exactly the moment the backend can least absorb it. The signal that actually catches degradation has to come from something already watching real traffic, request latency and error rate observed as it happens, rather than a separate probe manufacturing its own request to ask the question.

The Missing Piece in Every Liveness and Readiness Guide

Separate your liveness checks from your readiness checks, and never let a readiness probe restart a fleet, is correct advice, and it is genuinely well established at this point across Kubernetes’ own documentation and a decade of postmortems. It solves a real problem: an orchestrator that treats "my dependency is unreachable" as "kill this process" turns one blip into a fleet-wide restart storm.

What that advice does not cover is the axis Clerk’s incident actually ran into. Readiness, as usually implemented, is still a boolean: did the check complete within its timeout, yes or no. That is an improvement over checking nothing, but it treats an answer that comes back in 40 milliseconds and one that comes back in 4 seconds identically, as long as both land under the timeout. Nothing in the standard pattern asks how close to acceptable the answer actually was.

A failover or circuit breaker built to survive Clerk’s failure mode needs a third kind of signal alongside liveness and readiness: a rolling measure of latency or error rate against an explicit threshold, evaluated continuously rather than sampled once per probe interval.

// what Clerk's original failover watched — a boolean
const postgresIsDown = !(await postgres.ping({ timeoutMs: 2000 }))
if (postgresIsDown) triggerFailover()

// a threshold-based trigger that would have caught the actual failure
const window = latencyTracker.rollingWindow('session_lookup', { seconds: 30 })
const isDegraded =
  window.sampleCount >= MIN_SAMPLES &&
  window.p95Ms > FAILOVER_LATENCY_THRESHOLD_MS

if (postgresIsDown || isDegraded) triggerFailover()

Two details keep a check like this from becoming its own source of noise. The threshold should sit on a tail percentile, p95 or p99, rather than an average, since an average can hide a meaningful chunk of slow requests behind a majority of fast ones. And entering the degraded state should require a higher bar than leaving it, a small amount of hysteresis, so the trigger does not flip back and forth every time latency wobbles near the line. Without that gap, a marginal, recovering dependency can end up failing over and recovering every few seconds, which is its own kind of outage.

What Clerk Changed, and What’s Worth Checking in Your Own Stack

Four remediations came out of the postmortem, and each maps onto a generalisable gap. Clerk added dedicated alerting for query-plan flips specifically, rather than continuing to infer them from downstream symptoms like elevated 429s; detecting the cause directly is faster than reconstructing it from its effects, which is exactly what cost the team its first forty minutes. They broadened the newer failover’s trigger to cover any failure at the origin, not just an outright Postgres outage, closing the specific gap this incident exposed. They increased the query planner’s statistics sample size and refactored the vulnerable query for deterministic planning, addressing the immediate cause. And they formalised incident communication, committing to a dedicated communications lead and a regular status-page update cadence, after acknowledging that customers were told too little, too late, and with a severity label that undersold the impact.

That last point is easy to skip past, but it is the same lesson applied to people instead of code. A status page that updates rarely is its own boolean, quietly meaning either "fine" or "we have not gotten to it yet," and a reader has no way to tell which.

A Short Audit for Your Own Failover Triggers

Four questions worth running against whatever failover or circuit breaker sits in front of your own primary database.

• Does the trigger condition check reachability, or does it check whether responses are arriving fast enough to be useful?
• If a dependency can be "degraded but technically online," is anything watching for that state, or only for an outright outage?
• Has the newer, better-designed failover in your stack actually been wired into production triggers, or does it work in testing while the on-call runbook still points at the old one?
• When did anyone last deliberately make a dependency slow, not just unavailable, in a test environment, and watch what your monitoring actually did about it?

Most systems have a documented answer for what happens when the database goes down. Fewer have tested what happens when it does not, and simply stops being fast enough to matter.