Microsoft 365 Went Down With No Traffic Spike. Load Balancer Health Checks Need a Removal Limit.
Health checks pulled servers out, the rest overloaded, and restarted servers failed on arrival. Most load balancers have a setting for this.
The Microsoft 365 outage that began on 12 November ran for 36 hours, and three hours in Microsoft said something unusual: traffic had not gone up. Load on Exchange Online was high because a large number of servers had been taken out of rotation, and the ones still in rotation were carrying everyone. Those servers had been removed by automation. They were failing load balancer health checks, so the system did what it was built to do and stopped sending them requests.
Microsoft has not published its root cause yet, and nothing here guesses at it. The sequence in its public status updates is enough to raise a question you can answer about your own stack this week: if half your backends failed their health checks in the same minute, how many would your load balancer agree to remove?
What Microsoft has said so far
Everything below comes from Microsoft's status updates for incidents EX1550543 and MO1550821, as posted to the admin centre and the Microsoft 365 Status account. Times are UTC.
| Time (UTC) | What Microsoft reported |
|---|---|
| 12 Nov, 16:17 | Impact begins. Users cannot reliably reach Exchange Online mailboxes. |
| 19:06 | All deployments and configuration changes paused. |
| 19:18 | Elevated load is not a traffic increase. Many servers are unexpectedly unavailable and the rest are strained. |
| 19:43 | Affected servers are failing automated health checks and being removed from service. |
| 19:57 | No recent change yet found that lines up with the start. Steps taken to stop further removals. |
| 22:00 | Actions so far have not brought the expected relief. A core process on the servers is overwhelmed. |
| 23:59 | Servers returned to service can become unresponsive again once they receive traffic. |
| 13 Nov, 01:06 | The overwhelmed process is the one that establishes secure connections. Revert of a network configuration change begins. |
| 04:10 | Traffic to the affected infrastructure is limited and partly blocked. Recovery is partial. |
| 22:00 | Sustained recovery reported. Extended monitoring begins. |
| 14 Nov, 04:17 | Status set to restored. Preliminary cause: an issue during an earlier network capacity upgrade. |
Two caveats. The cause in the last row is preliminary; the post-incident report was still pending when this was written, and the root cause field on the incident read "N/A". And Microsoft has not said that health checks caused the outage. It said servers failed them and were removed, and that it then acted to stop more being removed. Why they failed in the first place is the part the report will have to explain.
Press reports citing Downdetector put user reports for Outlook above 129,000 at the peak. Teams chat and calls mostly held up, while anything that touched a mailbox, including calendars and meeting invites, did not.
The arithmetic of removing servers
A health check answers one question about one server: should this machine receive requests? Removing a single bad server is nearly free, because its share of traffic is spread thinly over many healthy ones. The trouble is that the decision is made per server, and nothing in the basic mechanism asks what the fleet looks like after the removal.
Take 100 servers sized to run at 70% of capacity at peak. Remove 20 and the other 80 run at 87.5%. Remove 30 and they run at 100%. Remove 31 and every remaining server is being asked for more than it can do. A fleet running at a fraction u of its capacity can lose a fraction of 1 minus u before the survivors saturate. Run hotter, at 85%, and the margin is 15 servers.
Past that point the failure feeds itself. Saturated servers answer slowly. Health checks have timeouts, so slow servers start failing them. Each one removed pushes its load onto fewer machines, which slows them further. The automation is now removing servers because of the load its earlier removals created. Nobody changed anything and no traffic arrived from outside, which is the pattern Microsoft described at 19:18.
Why restarting them did not help
Microsoft's 23:59 update is the most instructive line in the incident. Servers brought back into rotation went unresponsive again as soon as they took traffic. Researchers have a name for a system that stays broken after the original fault is gone: a metastable failure, described by Nathan Bronson and co-authors in a 2021 HotOS paper. Their point is that the trigger and the thing keeping you down are separate. A feedback loop sustains the bad state, and removing the trigger does not break the loop. Microsoft has not used that term, and the fit is our reading of its updates.
The usual sustaining loop is expensive work that happens at connection time. A server in steady state holds thousands of long-lived connections and sets up a few new ones per second. A server coming back from a restart has none, and every client that was cut off is retrying. Each new TLS connection costs a full handshake with public-key operations, far more CPU than a request on an existing connection. The returning server is asked to do hours of connection setup in seconds, falls behind, times out, fails its health check and is removed again. The clients it dropped go back into the retry pool.
Microsoft said the overwhelmed component was the process "responsible for establishing secure connections". That is consistent with a reconnect storm, though only the report can confirm it. Either way, the lesson for the rest of us does not depend on the detail: adding capacity back one server at a time into a full-strength retry storm does not work, because each server is destroyed on arrival. What worked for Microsoft, by its own account, was restricting which traffic could reach the affected servers and restoring it gradually.
How many backends your load balancer health checks may remove
Most load balancers have a setting for this, and the defaults differ more than people expect. Some fail open: when too many backends look unhealthy, they stop believing the health checks and send traffic to everything. Some fail closed and return errors. A few leave the floor at a single healthy server, which for a large fleet is no floor at all.
| System | Default behaviour | Setting to review |
|---|---|---|
| AWS Application Load Balancer | Fails open only when every target is unhealthy. Until then, one healthy target is treated as enough. | unhealthy_state_routing.minimum_healthy_targets.percentage (off by default) |
| Envoy active health checks | Panic mode: below 50% healthy hosts, traffic goes to all hosts regardless of health. | healthy_panic_threshold (0 disables it) |
| Envoy outlier detection | Ejects at most 10% of a cluster. | max_ejection_percent |
| Google Cloud Application Load Balancers | Fail closed. HTTP 503 when all backends are unhealthy (502 on the classic type). | No fail-open option; passthrough Network Load Balancers do spread traffic over unhealthy backends as a last resort |
| Kubernetes Service | Fails closed. A pod that fails readiness leaves the endpoints, with no minimum. | What the readiness probe depends on; publishNotReadyAddresses for special cases |
| HAProxy | Fails closed. 503 once no server in the backend is up, unless backup servers are defined. | backup servers, slowstart, check timeouts |
The AWS row deserves a second look, because it is the most common setup. The target group documentation says a target group counts as healthy while it has at least one healthy target. With 100 targets, the load balancer will remove 99 and route everything to the last one before fail-open applies. Setting the percentage threshold changes that: below it, traffic goes to all targets, healthy or not. Envoy's panic threshold does the same job and is on at 50% by default.
# Application Load Balancer: stop trusting health checks once fewer
# than half the targets pass, and ramp returning targets over 2 minutes.
# The DNS failover threshold must be >= the routing failover threshold.
aws elbv2 modify-target-group-attributes \
--target-group-arn "$TG_ARN" \
--attributes \
Key=target_group_health.dns_failover.minimum_healthy_targets.percentage,Value=50 \
Key=target_group_health.unhealthy_state_routing.minimum_healthy_targets.percentage,Value=50 \
Key=slow_start.duration_seconds,Value=120Fail-open is a bet that servers failing a check can still do useful work. That is usually true when many fail at once for a shared reason, and false when they are down for good. Pick the threshold from the arithmetic above: if your fleet saturates after losing 30%, a floor at 50% healthy arrives too late, and 70% is closer to what you need. On Kubernetes, where there is no floor, the lever is the probe. A readiness probe that calls the database turns a database blip into every pod leaving the Service together. PodDisruptionBudgets do not help here; they limit voluntary evictions and have no say over readiness.
Bring servers back slowly, and shed load in front of them
A removal floor handles the way down. The way up needs two more controls.
- Slow start. A returning backend gets a rising share of traffic over a window, so it can build connections and warm caches. On AWS this is slow_start.duration_seconds, 30 to 900 seconds and disabled by default. Envoy and HAProxy have equivalents.
- Admission control at the expensive step. Cap new connections or handshakes per second per server and reject the excess quickly. A fast refusal costs the server almost nothing. A handshake that is half done when the client gives up costs the full price and serves nobody.
- Client backoff with jitter. If your clients reconnect immediately and in step, the limit above is doing all the work. Spreading retries over time lowers the peak the servers must refuse.
Then look at the health check itself. The AWS Builders' Library article on implementing health checks recommends that load balancer checks test only what is local to the server, and that checks on shared dependencies run elsewhere with thresholds and a human involved. It also describes reserving worker threads for health checks so an overloaded server can still answer them. A check that competes with user traffic for the same saturated resource reports overload as death.
This has happened before
In AWS's October 2025 us-east-1 outage, Network Load Balancer health checks began failing against healthy targets. Capacity cycled in and out of DNS, and connection errors lasted from 5:30 AM to 2:09 PM Pacific. Engineers ended the errors at 9:36 AM by switching automatic health check failover off. The follow-up AWS committed to was a velocity control: a cap on how much capacity one load balancer may remove when health checks fail.
Thirteen months apart, two of the largest operators in the world reached for the same emergency action, which was to stop the automation from removing servers. That step is better taken in advance, as configuration, than at hour three of an incident.
What to check before the post-incident report lands
- For each load balancer, find the setting that limits removals and write down its current value. If the answer is "one healthy target", change it.
- Work out your saturation margin from peak load, and set the floor above it.
- Turn on slow start for anything with costly connection setup or cold caches.
- List what each health check depends on. Anything shared across the fleet does not belong in a check that can remove a server.
- Alert on the rate of health check removals across a fleet. Ten removals in a minute is a different event from ten in a day.
- Test it. Fail the health endpoint on 60% of a staging fleet under load and watch what the load balancer does.
Microsoft's report will say what broke first, and it may well be something no health check setting would have prevented. The second half of the incident, the part where recovery took a day and a half, is the half every operator can prepare for. When the report arrives, the section worth reading closely is the one explaining how long the removal ran before a person stopped it.
Frequently asked questions
Related reading
Daylight Saving Ends 25 October (EU) and 1 November (US). Know What Your 01:30 Scheduled Job Does
Two clock changes arrive in the next two weeks. Here is what daylight saving does to cron jobs, application schedulers and "daily" reports, and how to decide which jobs belong in UTC.
A Server Response Crashed Thousands of iOS Apps at Launch. Your SDK Is a Remote Control.
A malformed server payload crashed thousands of iPhone apps through the Firebase SDK. No app shipped a bug. Here is how to stop a vendor backend from deciding whether your app opens.
Clerk’s Failover Didn’t Fire Because Postgres Was “Technically Still Online.” That’s a Design Category, Not a Fluke.
On February 19, 2026, Clerk’s session failover didn’t trigger because Postgres was “technically still online” — degraded enough that queries returning 200 took minutes instead of milliseconds.