An 8-minute outage from a dead NLB and a JVM that cached DNS forever… — DevOps&SRE Library — TG.ME

An 8-minute outage from a dead NLB and a JVM that cached DNS forever

We drained a Network Load Balancer during a planned migration, and one internal service kept hammering the dead IPs for 8 minutes. The cause wasn't the failover. It was a JVM caching the DNS record forever. The fix was a 30-second TTL and a health-check tweak, not a smarter system.


https://dev.to/claire_nguyen/an-8-minute-outage-from-a-dead-nlb-and-a-jvm-that-cached-dns-forever-4dmj
September 13, 2026 2.3K 17