Day 56: Global Rate Limiting — Distributed Rate Limiter for Gateway Logins

The Spring Boot Trap

A junior engineer reaches for Spring’s @RateLimiter annotation from Resilience4j, or wires up Bucket4j with an in-memory ConcurrentHashMap. The code is clean. Tests pass. In staging, with a single node and synthetic load, it holds.

Then production happens.

Your Gateway cluster has twelve nodes behind an L4 load balancer. A credential-stuffing bot opens 120 connections — ten per node. Each node’s local AtomicInteger sees ten IDENTIFY packets. Your limit is 50 per minute. Every node reports: “Under limit.” The bot authenticates 120 sessions in thirty seconds. Your in-memory rate limiter protected nothing.

That is the locality problem. Any rate limiter that does not share state across nodes is theatre.

---

The Failure Mode

Why the naive fix also fails: The obvious next step is a shared Redis key with INCR + EXPIRE. Junior engineers write this as two round-trips:

long count = redis.incr(key);           // round-trip 1
if (count == 1) redis.expire(key, 60);  // round-trip 2

Under concurrent load, two nodes hit INCR simultaneously before either fires EXPIRE. The key never expires. You’ve built a permanent ban instead of a sliding window. This is a classic TOCTOU race on a distributed counter.

The thread cost: If each IDENTIFY triggers a synchronous Redis call on a platform thread, at 50,000 logins/second across the cluster you need 50,000 threads blocked on network I/O. At 1 MB stack each, that is 50 GB of thread stack memory — before your application does a single unit of work.

GC pressure from key strings: A naive Map<String, RateLimitBucket> where the key is "ratelimit:identify:" + ip allocates a new String object per request. At scale the Eden space saturates in milliseconds. Stop-the-world minor GC pauses hit every 200–400 ms, causing cascading IDENTIFY timeouts on legitimate clients.

---

The Flux Architecture