Reddit's zero-downtime migration of 500 Kafka brokers wasn't about Kafka. It was three reusable techniques.

Reddit's Kafka fleet ran over 500 brokers and held more than a petabyte of live data by early 2026, provisioned on Amazon EC2 through a stack of Terraform, Puppet, and custom scripts that had grown slower and more error-prone as the fleet scaled. The team wanted it on Kubernetes instead, which meant a genuine zero-downtime migration rather than a routine infrastructure move: no maintenance window, no data loss, and no client application allowed to change how it connected to Kafka.

Framed as a Kafka story, this is a good read and not much else, since most teams aren't running a petabyte-scale broker fleet. Framed as a zero-downtime migration story, it's a template. Three techniques did essentially all of the work, and none of them are specific to Kafka, Kubernetes, or Reddit's particular stack. They show up whenever a stateful system has to move out from under live traffic without anyone downstream noticing.

The constraints that rule out a normal zero-downtime migration playbook

Three ordinary migration strategies were unavailable from the start. A scheduled cutover, moving everything during an announced maintenance window, needs a window, and a platform serving Reddit's traffic has none worth calling acceptable. Dual-write, having clients write to both the old and new systems during a transition, requires every client to change its configuration, which was explicitly ruled out, and it opens a window where the two systems can silently disagree about state. Replay-based migration, copying a snapshot and then replaying the writes that happened afterward, works for bounded datasets, but it creates its own consistency gap at cutover time, exactly what a live, still-growing, partitioned dataset doesn't tolerate.

What was left: move the physical location of the data while it keeps flowing, and hide the fact that it moved from every client talking to it. That's a different problem than "migrate the data," and it needs a different set of tools.

Technique 1: put both fleets under one control plane

The Kubernetes-hosted broker operator Reddit used, Strimzi, doesn't support joining new Kubernetes brokers to an already-running EC2-hosted cluster out of the box.

According to Reddit's account of the migration, as summarised by ByteByteGo's write-up of the project, the team forked Strimzi and made a small number of targeted changes: plaintext inter-broker listeners, pointing the new brokers' ZooKeeper connection at the existing EC2-hosted ZooKeeper ensemble, and keeping the Cruise Control topic consistent across both broker sets.

The effect of those three changes is that, for the duration of the migration, there's one logical Kafka cluster with two physically separate sets of brokers, coordinated by the same ZooKeeper ensemble. That's the load-bearing idea, not the Strimzi fork itself. Any zero-downtime migration of a stateful system needs the old and new locations to become members of the same logical cluster before a single byte of production data moves. If the old and new sides are two clusters coordinated independently, that's dual-write with extra steps, not a migration.

Technique 2: move data by moving leadership, not by moving bytes

With both broker sets inside one cluster, Reddit used Cruise Control, Kafka's partition rebalancing tool, to incrementally shift partition leadership and replicated data from the EC2 brokers to the Kubernetes brokers. Over roughly a week, leadership on the EC2 side declined steadily while leadership on the Kubernetes side rose in parallel, watched continuously rather than triggered as a single event.

The reason to do it gradually rather than all at once is observability, not caution for its own sake. A partition-by-partition shift creates hundreds of checkpoints where a team can watch consumer lag, in-sync replica counts, and request-latency percentiles, and stop or reverse the process before a regression touches the whole fleet. A single cutover gives exactly one moment to notice a problem, and by the time it's noticed, everything has already moved.

The same shape shows up outside Kafka. A Postgres primary migration using logical replication, followed by a staged shift of read traffic before the final promotion, is the same pattern. An Elasticsearch reindex followed by an alias swap that can be dialled back per shard rather than flipped in one step is the same pattern. A load balancer moving traffic between an old backend pool and a new one through weighted routing, five percent, then twenty, then fifty, is the same pattern too. Even a stateful websocket or gRPC streaming tier can borrow it: drain connections off the old pool gradually, one node at a time, and let clients reconnect to the new pool through their normal retry logic instead of severing everyone at once. In every case the mechanism isn't "move the data", it's "move who's authoritative", one small unit at a time.

# Move a single partition from an EC2 broker (7) to a Kubernetes
# broker (107), throttled so the rebalance can't saturate inter-broker
# bandwidth and starve normal produce/consume traffic.

cat > partition-42-move.json <<'JSON'
{
  "version": 1,
  "partitions": [
    { "topic": "events", "partition": 42, "replicas": [107, 3, 9] }
  ]
}
JSON

kafka-reassign-partitions.sh --bootstrap-server "$BROKERS" \
  --reassignment-json-file partition-42-move.json \
  --throttle 50000000 \
  --execute

# Re-run with --verify before moving the next partition. Leadership
# only transfers to 107 once replication has fully caught up.
kafka-reassign-partitions.sh --bootstrap-server "$BROKERS" \
  --reassignment-json-file partition-42-move.json \
  --verify

Technique 3: the indirection layer that let clients not notice

The third piece is what Reddit calls a DNS facade: client applications connect to infrastructure-controlled DNS records, not directly to broker hostnames. That detail decides whether a migration needs client changes at all. If an application resolves or hardcodes the actual address of a specific broker or instance, the physical location can't move without touching every one of those applications, no matter how careful the rest of the migration is.

This layer has to exist before it's needed. Building the indirection during the migration, under the same deadline as the migration itself, defeats the purpose: that would mean shipping a client-facing change at exactly the moment the plan promised not to. Any team planning a zero-downtime migration of a stateful system should treat "do our clients talk to a stable, infrastructure-controlled name rather than a physical address" as a prerequisite to check months before the move, not a step inside it.

; Clients resolve this name. They never see a broker hostname,
; an EC2 instance ID, or a Kubernetes pod IP directly.
kafka-brokers.internal.example.com.   CNAME   ec2-broker-pool.internal.example.com.

; Once the Kubernetes fleet holds leadership for the majority of
; partitions, the facade repoints here. No client redeploy, no
; config change, no restart on the consuming side.
kafka-brokers.internal.example.com.   CNAME   k8s-broker-pool.internal.example.com.

What to watch while leadership is moving

Cruise Control (or an equivalent for a different system) tells you the rebalance is proceeding. It doesn't tell you whether it's safe to keep going. That judgment comes from three numbers, watched per partition or per shard rather than as fleet-wide averages, since a fleet-wide average hides the one partition that's actually struggling:

• Consumer lag on the partitions being moved, compared against a pre-migration baseline, not against zero
• In-sync replica count, since a replica that falls out of sync mid-move is a data-loss risk if the leader fails before it catches back up
• Request latency at the p99, not the average, because the clients most likely to notice a slow migration are the ones already closest to a timeout

A regression in any of the three is the signal to pause the rebalance, not to push through it faster. Reddit's week-long timeline for the leadership shift wasn't a scheduling choice so much as a consequence of watching these numbers and only proceeding when they stayed flat.

The three techniques, side by side

Where the pattern breaks down

The pattern isn't universal. If clients pin TLS certificates or IP allowlists to specific hosts, a DNS-level indirection layer doesn't fully hide a location change, since the client still validates against the old identity underneath the new address. If the system needs synchronous consistency across both fleets and the network path between the old and new location adds meaningful latency, for instance because the new infrastructure sits in a different region, gradual rebalancing can quietly violate a latency budget even though it never causes an outage. And if there's no equivalent of Cruise Control for the system being moved, building a granular rebalancing tool becomes its own project with its own timeline. Starting the migration clock before that tool exists and has been tested against a non-production fleet is how a zero-downtime plan turns into an incident.

When the juice isn't worth the squeeze

None of this is free. Forking a CNCF operator, building or extending a rebalancing tool, and retrofitting a DNS facade in front of a system that never had one is real engineering effort, measured in weeks even for a team as capable as Reddit's. For a system with a genuinely tolerable maintenance window, a modest dataset, and a client base small enough to coordinate directly, a scheduled cutover with a clear rollback plan is still the cheaper, lower-risk choice. This pattern earns its cost at the point where a maintenance window is either operationally unacceptable or logistically impossible to coordinate across every client, not before.

What to check before promising zero downtime

Four questions are worth answering, in roughly this order, before a team commits to this pattern for its own stateful system:

1. Do clients already resolve a stable, infrastructure-controlled name rather than a physical host or IP address?
2. Can the old and new infrastructure be brought under one control plane, or would doing so require two independently-arbitrated systems to somehow agree?
3. Is there a tool, existing or buildable in advance, that can shift authority in small, reversible units rather than one flip?
4. Has the full sequence been rehearsed against a non-production fleet at a scale close enough to surface the failure modes that only appear under real load?

A "no" to any of the four isn't a reason to abandon the migration. It's a reason to fix that gap before the clock starts, since every one of the three techniques above depends on the others already being in place.

The specifics will age. The pattern won't.

Strimzi, Cruise Control, and ZooKeeper are all particular to Kafka's current era of tooling, and each will look dated within a few years. The three techniques underneath them won't age the same way. Any team that has to keep a stateful system serving traffic while it physically moves is solving the same problem: unify control before moving anything, shift authority in small reversible steps instead of one, and make sure clients were never talking to a physical location in the first place.