Day 58: Sync Engine — Reconciling a Stale Client After Reconnection
The Spring Boot Trap
A junior engineer writes this:
List<Event> missed = eventRepo.findByGuildIdAndTimestampAfter(guildId, session.lastSeen());
socket.send(objectMapper.writeValueAsBytes(missed));It works in staging with five users. It catastrophically fails when 400,000 mobile clients reconnect after a 20-minute outage — the scenario Discord engineers call a “reconnect storm.”
The hidden costs:
findByGuildIdAndTimestampAfterdoes a full index scan with no upper bound. At 100 missed events per client, you are deserializing 40 millionEventobjects simultaneously. The Eden space fills in seconds. The JVM fires stop-the-world GC collections lasting 800ms each, exactly while the thread pool is already saturated processing reconnects. The database connection pool exhausts. The entire Gateway pod crashes and takes its neighbors with it via a cascading failure.The framework hid every one of those failure modes behind a clean repository interface.
The Failure Mode: Unbounded Delta Fetching
The naive sync strategy has three structural flaws:
No upper bound on delta size. A client offline for six hours in a busy guild has missed tens of thousands of events. Materializing all of them into heap memory is an OOM waiting to happen.
No event prioritization. Presence changes, role updates, and membership events affect what the client can render. Dumping chat messages on top of them means the client shows a broken UI while waiting for structural data to arrive.
No cursor continuity. Using wall-clock timestamps for sync is incorrect. Clocks drift. Events written within the same millisecond have undefined ordering under
timestamp >= ?. You need a monotonic sequence number, not a timestamp.
The Flux Architecture: Three-Phase Cursor Sync
