Article URL: https://buildkite.engineering/how-a-flaky-test-exposed-a-redis-use-after-free/ Comments URL: https://news.ycombinator.com/item?id=49000250 Points: 37 # Comments: 11

It can be hard to fix a bug even when it happens deterministically; replicating the circumstances that trigger it is often the hardest step. When a bug is non-deterministic this becomes harder: we need to execute the code that reproduces it many times in order to observe the behaviour, which adds significant time to our feedback loop. Harder still is when the cause of the bug happens prior to its manifestation —when data is corrupted but we don’t immediately notice it. Our usual tools focus on capturing the state of the system at the time a crash happens, which doesn’t tell us why the corruption happened in the first place, just that it happened. This is the tale of a memory corruption bug we found in the redis-client library and what helped us get there. Our story starts with David, an engineering manager on our Scalability team. One idle Tuesday, David saw a build of his had failed due to what appeared to be a number of flaky tests. He started a thread in Slack to alert others and included a screenshot from the Test Engine dashboard. As a continuous integration company, we are intimately familiar with flaky tests —they are an inevitable part of any large codebase. Within 15 minutes, the team pulled the Andon Cord; these tests were so flaky they were preventing us deploying code to production. As David summarised it, “[the] priority is getting main unblocked”, so we worked to exclude the tests from our suite temporarily. With the urgency reduced, David and the many other developers blocked by this hiccup were free to go back to their original tasks. David, though, decided to take a slightly deeper look by inspecting the reliability of these tests using Buildkite’s Test Engine reliability score, to see if he could correlate when they suddenly became unstable. It seems the three worst flakeys went from ~100% reliable to… not around [2-4 days prior]. I’ve had a look through merged PRs to see if I can spot what might have contributed to that change, but can’t see anything obvious… Only thing that jumps out is the redis upgrade? That Redis upgrade was done on the prior Friday afternoon by a staff engineer, and part time blog post narrator, Patrick Robinson. Patrick loves few things more than talking about himself in the third person; one of those things is getting deep into gnarly bugs. Here’s our first clue: the Redis gem upgrade had gone smoothly with no hint of any production issues. At this point, though, a second hypothesis appeared: the tests were flaky because they depended on a certain sequence of events. See, the flaky tests were part of our feature test suite; they utilised not just RSpec but also a Selenium headless browser and a test environment comprising of a web/API server, database and Redis server. In these situations, race conditions between the different components are a common cause of flaky tests. Unfortunately, this would lead our investigators off track.