Designing a Reconnect-Safe Real-Time System at 10 Million Connections
A WebSocket server with 100,000 connected clients crashes.
At first, that sounds like a capacity problem.
It isn't.
The real problem starts about half a second later.
Those 100,000 clients notice the connection is gone. They reconnect. Each connection performs authentication, restores subscriptions, loads session state, requests missed events, establishes a new socket, and starts receiving traffic again.
Now imagine that your other servers were already operating at 65% capacity.
The surviving fleet receives a sudden burst of 100,000 connection attempts.
Authentication traffic spikes.
Redis gets hammered.
Your event store receives a burst of replay queries.
Connection creation consumes CPU.
Outbound fan-out increases.
Some clients time out.
Those clients reconnect again.
Your system has now entered a feedback loop.
One failed server has turned into a fleet-wide outage.
This is a reconnect storm.
And it exposes a deeper truth about real-time architecture:
A system that can handle 10 million live connections is not necessarily a system that can recover from losing 1 million of them.
This article designs the latter.
We will build a real-time architecture capable of handling millions of persistent connections while specifically protecting against:
- mass reconnects
- rolling deployments
- gateway failures
- regional failures
- replay storms
- slow clients
- duplicate messages
- event gaps
- overloaded authentication
- state rehydration spikes
- cascading connection failures
The goal isn't simply to keep sockets open.
The goal is to make failure boring.
1. The Architecture Everyone Draws First
Let's start with the obvious design.
Suppose we're building a real-time platform for:
- chat
- notifications
- live dashboards
- collaborative applications
- presence
- operational alerts
The first architecture usually looks like this:
┌──────────────┐
│ Clients │
│ Web / Apps │
└──────┬───────┘
│
▼
┌──────────────┐
│Load Balancer │
└──────┬───────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│Gateway │ │Gateway │ │Gateway │
│ A │ │ B │ │ C │
└────┬───┘ └────┬───┘ └────┬───┘
│ │ │
└────────────┼────────────┘
▼
┌─────────────┐
│ Event Bus │
│ Kafka/NATS │
└──────┬──────┘
│
▼
┌─────────────┐
│ Applications │
└─────────────┘
Looks reasonable.
It scales horizontally.
Each gateway holds thousands or tens of thousands of connections.
The event bus distributes messages.
The load balancer spreads new connections across gateways.
But there's a serious flaw.
What happens when Gateway B dies?
Suppose Gateway B holds 100,000 connections.
The architecture above tells us where those connections go during normal operation.
It doesn't tell us what happens during recovery.
That is the part we need to design.
2. A Connection Is Not Just a Connection
This distinction is easy to miss.
Consider a normal HTTP request:
Client │ │ GET /profile ▼ Server │ ▼ Response │ ▼ Done
The server can disappear after the response.
The next request can go to another server.
The server doesn't need to remember much about the previous TCP connection.
WebSockets are different.
A WebSocket connection might contain:
Connection ├── TCP state ├── TLS state ├── authentication state ├── user identity ├── tenant ├── subscriptions ├── room memberships ├── outbound queue ├── heartbeat state ├── last acknowledged event └── connection-specific metadata
The socket is therefore not merely a network pipe.
It is part of the application's runtime state.
That has a major consequence.
Replacing a WebSocket server is also replacing thousands of active client sessions.
This is why deployment and failure recovery become first-class system-design problems.
Modern real-time architecture guidance increasingly calls out this exact distinction, especially the cost of authentication and state rehydration during reconnects.
3. The Reconnect Storm
Let's put numbers on the problem.
Assume:
Total connections: 10,000,000 Gateway servers: 100 Connections per gateway: 100,000
Gateway #37 crashes.
That means:
Lost connections = 100,000
Now imagine every client reconnects immediately.
Your load balancer sees:
100,000 connection attempts
But a connection attempt isn't free.
Suppose each reconnect requires:
1 TLS handshake 1 authentication operation 2 cache reads 1 subscription lookup 1 replay query 1 subscription registration
That's already:
100,000 TLS handshakes 100,000 authentication operations 200,000 cache reads 100,000 replay queries 100,000 subscription registrations
A single machine failure has generated hundreds of thousands of backend operations.
And that's before sending any real-time messages.
This is why the statement:
"Our system can handle 100,000 connections"
is incomplete.
The better question is:
How many connections can we establish per second while simultaneously restoring their state?
That is our first capacity metric.
4. Connection Capacity and Recovery Capacity Are Different
A system might support:
10 million established connections
while only safely creating:
10,000 new connections/sec
This distinction matters enormously.
Let's define:
C = active connections R = safe connection establishment rate F = failed connections that must reconnect
The recovery time is approximately:
Recovery time ≈ F / R
Suppose:
F = 100,000 R = 10,000/sec
Then:
Recovery time ≈ 10 seconds
That sounds fine.
But now imagine your authentication service can only handle 5,000 reconnects/sec.
The actual limit is no longer the WebSocket gateway.
It is authentication.
If replay can handle only 3,000 requests/sec, replay becomes the bottleneck.
If subscription restoration can handle only 2,000/sec, that's the bottleneck.
The recovery path is a pipeline.
Its capacity is determined by its weakest stage.
Reconnect │ ▼ TLS │ ▼ Authentication │ ▼ Session lookup │ ▼ Subscription restore │ ▼ Replay │ ▼ Live stream
The fastest component doesn't matter.
The slowest component controls recovery.
5. The First Rule: Never Let Clients Reconnect Without Control
The naive client implementation looks like this:
function connect() {
const socket = new WebSocket(URL);
socket.onclose = () => {
connect();
};
}
This code is small.
It is also capable of destroying your infrastructure.
Imagine 100,000 clients receiving a disconnect at the same time.
They all execute:
onclose()
↓
connect()
You have created synchronized load.
The first thing we need is jittered exponential backoff.
6. Exponential Backoff Is Necessary, But Not Enough
A better strategy:
Attempt 1: wait ~1 second Attempt 2: wait ~2 seconds Attempt 3: wait ~4 seconds Attempt 4: wait ~8 seconds Attempt 5: wait ~16 seconds Maximum: 30 or 60 seconds
But don't use exactly:
1s 2s 4s 8s
for every client.
Add randomness.
For example:
delay = random(0, min(cap, base × 2^attempt))
If 100,000 clients all need to retry after a failure, the reconnects become distributed across a window instead of arriving as one enormous spike.
Jittered exponential backoff is now standard guidance for large-scale real-time systems.
But there is a deeper problem.
The server still has no control over the aggregate rate.
Every client is making its own decision.
We can do better.
7. Server-Controlled Reconnect Admission
Instead of relying entirely on clients to behave nicely, make the server part of the control loop.
Suppose a gateway has:
Maximum safe connection rate: 5,000/sec
We can enforce an admission limit.
New connection
│
▼
┌────────────────┐
│ Admission │
│ Controller │
└───────┬────────┘
│
┌──────────┴──────────┐
│ │
ACCEPT REJECT
│ │
▼ ▼
WebSocket Retry-After
The server can return a retryable response before doing expensive work.
For example:
HTTP/1.1 503 Service Unavailable Retry-After: 8
Or communicate a reconnect delay through the application protocol.
The idea is simple:
Don't allow a reconnect storm to reach the expensive parts of your system.
This creates a controlled front door.
A 2026 production-oriented realtime design example uses exactly this kind of layered protection, including connection admission limits, per-node connection caps, and graceful draining.
8. Protect the Expensive Path First
A common mistake is putting rate limiting after authentication.
That's backwards during a reconnect storm.
Imagine:
100,000 reconnects
│
▼
JWT verification
│
▼
Redis
│
▼
Database
│
▼
Connection
By the time you reject the request, you've already consumed expensive resources.
Instead:
100,000 reconnects
│
▼
Cheap admission control
│
├──── reject / retry
│
▼
Connection limit
│
▼
Authentication
│
▼
Session restoration
│
▼
Replay
The principle is:
Apply the cheapest protection as early as possible.
This isn't only useful for WebSockets.
It's a general distributed-systems principle.
9. Connection Draining During Deployments
A particularly stupid way to deploy a WebSocket service is:
Deploy ↓ Kill old process ↓ Start new process
Suppose the process owns:
80,000 WebSockets
You just manufactured an 80,000-client reconnect storm.
Instead, use connection draining.
The lifecycle becomes:
RUNNING
│
▼
STOP NEW CONNECTIONS
│
▼
ANNOUNCE SERVER DRAIN
│
▼
ASK CLIENTS TO RECONNECT
│
▼
DRAIN EXISTING WORK
│
▼
CLOSE REMAINING
│
▼
SHUTDOWN
The load balancer should stop sending new WebSocket upgrades to the draining server.
Existing connections remain alive for a controlled period.
Clients receive a server-draining signal.
They reconnect with randomized delays.
Only then does the old process terminate.
Graceful draining is particularly important for long-lived connections because normal HTTP deployment assumptions don't apply to sockets that can live for hours.
10. The Deployment Problem Becomes a Capacity Calculation
Suppose:
Connections on server = 100,000 Target reconnect rate = 5,000/sec
Minimum theoretical redistribution time:
100,000 / 5,000 = 20 seconds
But we shouldn't run at 100% capacity.
Suppose our safe operating rate is:
3,000 reconnects/sec
Then:
100,000 / 3,000 ≈ 33 seconds
Now add:
- TLS overhead
- authentication
- state restoration
- event replay
- network variance
- client retries
- load-balancer behavior
A 60-second drain period might be reasonable.
The important part isn't the exact number.
It's this:
Connection draining should be designed from recovery capacity, not copied from a Kubernetes example.
11. Reconnecting Is Only Half the Problem
Now the client reconnects.
What does it receive?
Suppose the client had processed:
Event 100
Before the connection failed, the server produced:
101 102 103 104 105
The client never received them.
If we simply send the newest state, we may lose information.
If we replay everything from the beginning, that's obviously impossible.
We need a cursor.
12. Every Durable Stream Needs a Position
Give each event a monotonically increasing sequence number.
100 101 102 103 104 105
The client stores:
lastProcessed = 100
When reconnecting:
GET /stream Last-Event-ID: 100
The server can then determine:
events where sequence > 100
and replay:
101 102 103 104 105
Then transition back to live delivery.
Conceptually:
RECONNECT
│
▼
lastSeen = 100
│
▼
Replay 101-105
│
▼
Replay done
│
▼
LIVE STREAM
This is one reason SSE can be attractive for one-way real-time delivery. The protocol has Last-Event-ID and automatic reconnection semantics built into the browser API. WebSockets require you to build equivalent recovery behavior yourself.
For a true bidirectional application, WebSockets may still be the correct choice. The important lesson is to explicitly design the recovery protocol.
13. Don't Confuse "Sent" With "Delivered"
This is one of the most important concepts in real-time systems.
Suppose:
server.send(event)
returns successfully.
Did the user receive it?
No guarantee.
The message may have:
- entered a kernel buffer
- entered a TLS buffer
- entered a proxy buffer
- reached the client
- reached the application
- been parsed
- been persisted
- been processed
These are different states.
Consider:
Server │ │ send(event 103) ▼ TCP buffer │ ▼ Network │ ▼ Client runtime │ ▼ Application │ ▼ Persistent client state
A system that needs recovery semantics must define what "processed" means.
14. The Cursor Should Move Only After Processing
Suppose the client receives:
event 103
but crashes before saving:
lastProcessed = 103
After reconnecting, it reports:
lastProcessed = 102
The server sends 103 again.
That's okay.
We intentionally prefer:
duplicate
over:
lost event
This is the core tradeoff behind at-least-once delivery.
The application needs idempotency.
For example:
event_id = 7c4a...
The client or application can maintain a deduplication record:
processed: 7c4a... 91bd... e321...
If the same event arrives twice:
if already_processed(event_id):
ignore
else:
process
mark_processed
Exactly-once delivery across an unreliable network is not something you get merely by putting WebSockets on top of TCP.
You design the semantics explicitly.
15. Replay Storage Changes the Architecture
Now we need somewhere to obtain missed events.
A naive design might query the primary database:
SELECT * FROM events WHERE user_id = ? AND sequence > ? ORDER BY sequence;
At small scale, fine.
During a reconnect storm:
100,000 users reconnect
Now you have:
100,000 replay queries
That can be disastrous.
This is why the recovery path deserves its own storage strategy.
A common architecture is:
Event Producers
│
▼
Durable Log
┌──────────────┐
│ Kafka │
└──────┬───────┘
│
┌───────────┴──────────┐
▼ ▼
Live Delivery Replay Storage
│ │
▼ ▼
WebSocket Reconnect
Gateways Service
The exact technology isn't the important part.
The important part is separating:
live delivery
from:
recovery
16. The Replay Window
We don't necessarily need infinite history.
Define a replay window.
For example:
Replay retention = 24 hours
If a client reconnects after 20 seconds:
Replay available
If it reconnects after 3 days:
Replay unavailable
At that point, we need a snapshot.
Client │ │ last cursor too old ▼ Snapshot │ ▼ Current state │ ▼ Replay recent events │ ▼ Live stream
This is much cheaper than storing and replaying the entire history for every client.
17. Snapshot + Delta Is More Powerful Than Infinite Replay
Suppose a document has:
Current version: 1,000,000
A client last saw:
version 100
Sending:
101 102 103 ... 1,000,000
is absurd.
Instead:
snapshot(version=999,500)
+
events 999,501 → 1,000,000
Now recovery becomes manageable.
This pattern works for:
- collaborative documents
- dashboards
- game state
- account state
- inventory
- presence summaries
- live configuration
The general rule:
Use replay for short gaps. Use snapshots for long gaps.
18. Now Consider the Slow Client
Here's another failure mode.
Your server produces:
1,000 events/sec
A client's network can consume:
100 events/sec
If you buffer everything:
Queue: 100 1,000 10,000 100,000 1,000,000
Eventually:
RAM → 100% Process → OOM Gateway → crash
One slow client has now become everyone else's problem.
Real-time systems need bounded buffers.
Client
│
▼
┌───────────────────┐
│ Outbound Queue │
│ max = 1,000 │
└─────────┬─────────┘
│
▼
Socket
Once the queue exceeds its limit, you need a policy.
There are three major options.
19. Strategy 1: Disconnect
For events that must not be lost:
Queue full
↓
Disconnect
↓
Client reconnects
↓
Replay missed events
This is often much safer than letting memory grow indefinitely.
The client effectively says:
"I can't keep up."
The server responds:
"Then we'll reset you and resynchronize."
That is controlled failure.
20. Strategy 2: Drop
Some events don't matter after they become stale.
For example:
typing_started cursor_moved mouse_position progress = 47%
If the client misses:
cursor x=420
there is little value in replaying it three seconds later.
Drop it.
21. Strategy 3: Conflate
This is even better for high-frequency state.
Suppose we receive:
price = 100 price = 101 price = 102 price = 103 ... price = 200
The client doesn't necessarily need all 101 intermediate states.
It needs:
price = 200
So instead of:
Queue: 100 101 102 103 ... 200
keep:
Latest price: 200
Then periodically send the latest value.
This is called conflation, and it can reduce both bandwidth and memory dramatically for state-like streams. Modern real-time system guidance increasingly treats conflation as a core backpressure technique.
The key is understanding the semantics.
Do not conflate events like:
PaymentCreated PaymentCaptured PaymentRefunded
Those are facts.
You can often conflate:
cursor position typing state progress percentage current temperature current stock price
Those are states.
22. A Better Event Model
This suggests that a production event protocol should classify events.
For example:
{
"id": "evt_18291",
"type": "price_update",
"delivery": "latest",
"sequence": 82912,
"timestamp": 1788684000,
"payload": {
"symbol": "ABC",
"price": 182.42
}
}
versus:
{
"id": "evt_18292",
"type": "payment_captured",
"delivery": "durable",
"sequence": 82913,
"timestamp": 1788684001,
"payload": {
"paymentId": "pay_123"
}
}
Now the gateway knows:
latest
→ may be conflated
durable
→ must be replayable
That decision can save enormous infrastructure cost.
23. Subscription Restoration Is Another Hidden Bottleneck
Imagine a user has:
300 subscriptions
When the socket reconnects, you could send:
SUBSCRIBE 1 SUBSCRIBE 2 SUBSCRIBE 3 ... SUBSCRIBE 300
Now imagine:
100,000 users reconnect
That's:
30,000,000 subscription operations
You just created a second reconnect storm.
The solution is to make session restoration compact.
Instead of reconstructing everything from scratch:
Client │ │ session_id ▼ Session Registry │ ▼ Subscription Set
Or encode the necessary state into a resumable session token where appropriate.
The general goal is:
Make reconnect work proportional to the connection, not proportional to the history of the connection.
24. Don't Make Authentication Your Reconnect Bottleneck
Authentication services are normally designed around ordinary request rates.
A reconnect storm changes the traffic pattern.
Suppose:
Normal authentication: 20,000 requests/sec Reconnect storm: 300,000 requests/sec
Your authentication service becomes the first casualty.
Possible protections include:
- short-lived cached authentication results
- connection-specific session tickets
- local verification for signed tokens
- admission control before expensive authorization
- avoiding unnecessary database reads
- caching tenant and subscription metadata
If your JWT is locally verifiable using a cached public key, you may not need a network call for every reconnect.
That can make an enormous difference.
25. The Reconnect Path Should Be Cheaper Than the Initial Connection Path
This is a powerful design principle.
Initial connection:
authenticate load user load tenant load permissions load subscriptions create session
Reconnect:
validate resume token restore session resume cursor
If every reconnect behaves like a brand-new login, mass failures become much more expensive than normal operation.
A resumable session can turn:
full initialization
into:
resume existing state
The difference becomes enormous at millions of connections.
26. Now Add Multi-Region Failure
So far we've only killed one gateway.
Now let's kill an entire region.
Architecture:
Global Traffic
│
┌────────────┴────────────┐
│ │
▼ ▼
US Region EU Region
│ │
┌─────┴─────┐ ┌─────┴─────┐
│ Gateways │ │ Gateways │
└─────┬─────┘ └─────┬─────┘
│ │
└──────────┬──────────────┘
▼
Durable Events
Suppose US-East contains:
3,000,000 connections
and the entire region disappears.
Those three million clients now need somewhere to go.
But EU-West was not necessarily provisioned to accept three million additional connections instantly.
This gives us another capacity equation.
Let:
N = connections normally served by region H = reserved failover headroom
If:
N = 3M H = 20%
then other regions need enough spare capacity to absorb:
3M connections
not merely the normal traffic level.
This is why multi-region architecture isn't just:
"Put another region next to the first one."
You need failure capacity.
27. Regional Failure Can Create a Global Reconnect Storm
Imagine:
US-East dies
│
▼
3M clients disconnect
│
▼
Global traffic manager
│
▼
EU-West + US-West
│
▼
Reconnect
│
▼
Authentication
│
▼
Replay
If every client immediately chooses the same surviving region, you've simply moved the outage.
Traffic steering needs to consider:
- current connection counts
- regional capacity
- reconnect rate
- latency
- health
- failure state
The global control plane itself becomes part of the recovery architecture.
28. Avoid the "Everything Depends on Redis" Trap
Redis is excellent for many real-time workloads.
But consider this architecture:
Reconnect ↓ Redis ↓ Subscription state ↓ Redis ↓ Session ↓ Redis ↓ Replay
Now Redis becomes the central dependency for recovery.
During a reconnect storm, that's dangerous.
A better architecture separates responsibilities:
Session cache
│
▼
Fast lookup
Durable event log
│
▼
Replay
Database
│
▼
Long-term state
The cache should accelerate recovery.
It shouldn't be the only source of truth for durable events.
This distinction matters because pub/sub systems are not automatically durable replay systems. Production guidance for WebSocket systems commonly warns against treating a transient pub/sub layer as the complete message history.
29. The Full Architecture
Now we can assemble everything.
┌───────────────────────┐
│ Global DNS │
│ / Traffic Manager │
└───────────┬───────────┘
│
┌───────────────────┴───────────────────┐
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Region A │ │ Region B │
└──────┬───────┘ └──────┬───────┘
│ │
┌────────┴────────┐ ┌────────┴────────┐
│ │ │ │
▼ ▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│Gateway │ │Gateway │ │Gateway │ │Gateway │
│ │ │ │ │ │ │ │
└────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘
│ │ │ │
└────────┬────────┘ └────────┬────────┘
│ │
▼ ▼
┌────────────────┐ ┌────────────────┐
│ Local Admission│ │ Local Admission│
│ Controller │ │ Controller │
└───────┬────────┘ └───────┬────────┘
│ │
└────────────────┬────────────────────┘
│
▼
┌──────────────────┐
│ Event Stream │
│ Kafka / Durable │
│ Log │
└────────┬─────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌────────┐ ┌──────────┐ ┌──────────┐
│ Replay │ │ Session │ │ Database │
│ Store │ │ Registry │ │ / State │
└────────┘ └──────────┘ └──────────┘
And each gateway contains:
Gateway ├── connection manager ├── authentication ├── admission control ├── heartbeat manager ├── subscription manager ├── outbound queue ├── backpressure policy ├── cursor tracking ├── replay coordinator └── graceful shutdown controller
Now we're designing a real system.
30. Failure Scenario: One Gateway Dies
Let's walk through it.
Initial state:
Gateway A 100K connections
Gateway A crashes.
Step 1: Detect
Health checks detect the failure.
Step 2: Traffic stops
Load balancer removes Gateway A.
Step 3: Clients detect disconnect
Clients enter reconnect state.
Step 4: Jitter
Clients calculate randomized retry delays.
Step 5: Admission control
Surviving gateways accept only a controlled number of reconnects.
Step 6: Resume
Client sends:
session_id last_processed_event
Step 7: Session restoration
Server loads the compact session state.
Step 8: Replay
Server sends:
last_processed + 1 ... latest_available
Step 9: Live transition
Once caught up:
REPLAY → LIVE
Step 10: Recovery complete
No manual intervention.
That's what failure handling should look like.
31. Failure Scenario: Deployment
Now deploy Gateway B.
Instead of:
kill
we do:
mark draining
↓
remove from load balancer
↓
stop accepting new connections
↓
send reconnect hint
↓
clients reconnect gradually
↓
drain outbound work
↓
close remaining connections
↓
terminate
The key difference:
The server controls when the clients come back.
That prevents the deployment from becoming a synchronized reconnect event.
32. Failure Scenario: Slow Client
Client receives messages too slowly.
Queue reaches:
80%
Start monitoring.
At:
100%
apply policy.
If event is:
typing_update
drop or conflate.
If event is:
payment_captured
don't drop.
Disconnect and force replay.
The client reconnects and receives the durable event from the log.
This is a clean failure boundary.
33. Failure Scenario: Client Was Offline for Two Days
The replay window is:
24 hours
Client asks for:
event 4,000,000
but the earliest available event is:
5,000,000
The server responds:
RESYNC_REQUIRED
Then:
snapshot
↓
snapshot_version
↓
replay from snapshot_version
↓
live
This is much safer than pretending that every disconnect can be repaired through infinite event replay.
34. The Metrics That Actually Matter
If you're operating this architecture, don't monitor only:
CPU RAM connections
Those aren't enough.
Track:
Connection metrics
active_connections new_connections/sec closed_connections/sec reconnects/sec connection_failure_rate
Recovery metrics
replay_requests/sec events_replayed/sec average_replay_depth max_replay_depth resync_rate resume_success_rate
Admission metrics
connection_admission_rate connection_rejection_rate retry_after_distribution
Backpressure metrics
buffer_depth buffer_overflow_rate slow_client_count conflated_events/sec disconnect_due_to_backpressure
Event metrics
event_lag consumer_lag delivery_latency p50 p95 p99
The metric I would watch most closely
reconnects/sec
Because that number tells you whether the recovery system is under stress before many of the downstream systems fail.
35. A Useful SRE Metric: Recovery Amplification
Here's a metric worth introducing.
Define:
Recovery Amplification = backend operations caused by one reconnect
Suppose one reconnect triggers:
1 auth check 2 cache reads 1 subscription lookup 1 replay query
That's:
5 backend operations
If 100,000 connections reconnect:
500,000 backend operations
Now imagine you reduce the reconnect workflow from five operations to two.
You just reduced recovery load by:
60%
without changing the number of connections.
This is why optimizing the reconnect path can matter more than optimizing the steady-state socket.
36. Another Useful Metric: Recovery Debt
Suppose a gateway fails.
You have:
100,000 disconnected clients
and you're safely reconnecting:
2,000/sec
Your recovery debt is:
100,000
Then:
98,000 96,000 94,000 ...
You can graph it.
If the number isn't falling quickly enough, the system isn't recovering.
This is often more useful during an incident than looking at CPU.
A healthy system should show:
Recovery debt
100K ───────╲
╲
╲
╲
╲──── 0
An unhealthy system shows:
Recovery debt
100K ─────╲
╲
────────
╲
╲
Recovery has stalled.
37. The Most Dangerous Architecture
Here is the design I would actively reject:
Client ↓ WebSocket ↓ Redis ↓ Database
with:
on disconnect:
reconnect immediately
and:
on reconnect:
authenticate
reload everything
replay everything
This architecture often works beautifully in development.
Then production gives it 50,000 simultaneous disconnects.
And it collapses.
The problem isn't that WebSockets don't scale.
The problem is that recovery wasn't designed as part of the system.
38. The Design Principles
If you remember only a few ideas from this article, remember these.
1. Connections are state
A persistent connection isn't equivalent to a normal HTTP request.
2. Recovery is a separate workload
A system can handle millions of steady-state connections and still fail during mass reconnects.
3. Control reconnect admission
Don't allow unlimited reconnect traffic to hit your expensive services.
4. Jitter everything
Synchronized clients create synchronized load.
5. Make reconnect cheap
Resume state instead of reconstructing everything.
6. Use cursors
A reconnecting client needs to tell the server where it stopped.
7. Prefer at-least-once plus idempotency
Duplicate events are usually easier to handle than missing events.
8. Bound every queue
Unlimited buffering is just a delayed out-of-memory error.
9. Separate durable events from ephemeral state
A payment event and a cursor position don't have the same delivery requirements.
10. Drain connections during deploys
Never casually kill thousands of long-lived sessions.
11. Design for regional failure
Your failover region needs spare connection capacity.
12. Measure recovery itself
Watch reconnect rate, replay depth, recovery debt, and admission pressure.
39. The Final Architecture
A mature real-time system doesn't look like:
WebSocket + Redis + Load Balancer
It looks more like:
┌─────────────────┐
│ Global Traffic │
│ Manager │
└────────┬────────┘
│
┌─────────────┴─────────────┐
│ │
▼ ▼
┌────────────┐ ┌────────────┐
│ Region A │ │ Region B │
└─────┬──────┘ └─────┬──────┘
│ │
┌────────┴────────┐ ┌────────┴────────┐
│ │ │ │
▼ ▼ ▼ ▼
Gateway Gateway Gateway Gateway
│ │ │ │
└────────┬────────┘ └────────┬────────┘
│ │
▼ ▼
Admission Admission
Controller Controller
│ │
└─────────────┬─────────────┘
│
▼
Durable Event Log
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
Replay Session Database
Store Registry State
And inside every gateway:
Gateway
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Admission Connection Heartbeat
Control Manager Manager
│ │
│ ▼
│ Subscription
│ Manager
│ │
│ ▼
│ Outbound Queue
│ │
│ ▼
│ Backpressure
│ Policy
│
▼
Resume
Protocol
│
▼
Replay
That's the difference between:
"We use WebSockets."
and:
"We designed a real-time distributed system."
40. The Question You Should Ask in Every Real-Time System Design
The next time someone draws:
Client ↓ Load Balancer ↓ WebSocket Servers
don't ask:
"How many connections can one server handle?"
Ask:
"What happens when 100,000 of those connections disappear at the same time?"
Then ask:
"How quickly can we safely recreate them?"
Then:
"What backend systems get hit during reconnect?"
Then:
"How do clients recover missed events?"
Then:
"What happens to slow clients?"
Then:
"What happens during deployment?"
Then:
"What happens when the entire region disappears?"
Those questions reveal the real architecture.
Because scaling a real-time system isn't just about keeping connections alive.
It's about surviving the moment they all die.
Quick reference: production checklist
Before calling a million-connection real-time system production-ready, verify:
[ ] Client exponential backoff [ ] Client retry jitter [ ] Server-controlled reconnect hints [ ] Connection admission control [ ] Per-node connection limits [ ] Graceful connection draining [ ] Durable event IDs [ ] Resume cursor [ ] Replay window [ ] Snapshot fallback [ ] Idempotent event processing [ ] Bounded outbound queues [ ] Slow-client policy [ ] Event conflation where appropriate [ ] Authentication protection [ ] Session restoration path [ ] Durable event storage [ ] Multi-region failover capacity [ ] Reconnect-rate monitoring [ ] Replay-lag monitoring [ ] Recovery-debt monitoring [ ] Load testing with mass disconnects [ ] Chaos testing of gateway failures [ ] Regional failure testing
The final checkbox is the one most teams skip:
[ ] Disconnect 100,000 clients simultaneously
Don't test only whether the system can serve 100,000 connections.
Test whether it can lose 100,000 connections and recover without taking everything else down with them.
That is the real system-design problem.
Comments (0)
No comments yet. Be the first!