Article URL: https://keel-iot.eu/blog/state-oriented-consistency.html Comments URL: https://news.ycombinator.com/item?id=49201540 Points: 4 # Comments: 0

The useful question turned out to be: "Which consistency guarantee does this specific piece of state actually need?" Those sound similar. They aren't. The first one assumes a single answer exists for the whole system. The second one assumes it doesn't — and it's the second one that this whole piece converges on, in the form of one table that, by the end, does most of the explaining for us. We discovered the difference the hard way while building a clustered message broker. The lesson underneath has very little to do with MQTT. Any distributed system holding more than one kind of state runs into this question eventually. Get it right, and the rest of the design gets simpler almost automatically. Two pods, five minutes apart, killed by the kernel's OOM handler. Memory limit: 512Mi. Nothing exotic — a normal container limit for a normal stateless service. The first hypothesis was the obvious one: load-balancer imbalance, probably compounding into a reconnect cascade. Plausible. Wrong. That's not the shape imbalance produces. A pod serving 130x more connections than another should not have nearly identical memory. Either the metric was lying, or the mental model was. Reading the actual code path responsible for session state turned up the real cause: on every pod boot, a persistence hook loaded every client's stored session state — the entire fleet's, not just the fraction that would ever reconnect to this specific pod. One unfiltered read, called once at startup, and every row it returned became a live in-memory object. Why would anyone write it that way? In a stateless cluster behind a non-sticky load balancer, there's genuinely no way to know in advance which clients will reconnect to this pod. So the simplest correct-seeming implementation was: load everything, everywhere, and let whatever connects find its state waiting. Not a typo-bug — a design that quietly assumed a node should be ready to serve any client that might show up. A small, local instance of exactly the wrong question from the opening: it optimized for "the cluster can serve anyone," not "this specific piece of state has this specific requirement."