Article URL: https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-opus-5-on-slop-code-bench.md Comments URL: https://news.ycombinator…

Last Friday I dug into SlopCodeBench, a new-ish (March 2026) long-horizon coding benchmark from @GOrlanski's lab at UW Madison. It addresses the thing that bothers me most about coding benchmarks - that even "larger" more complex benchmarks still divulge the whole problem up front: In contrast, each challenge in SlopCodeBench has multiple "checkpoints" - the model doesn't know the whole problem up front, it has to evolve the codebase over time as new requirements are divulged. What's cool about this benchmark is that it is unsaturated - at the time of running, the best models available, GPT-5.4 and Opus 4.6, got 11% and 17% strict pass rates, respectively. On Friday I ran three claude models (Opus 4.8, Sonnet 5, and Opus 5) through a subset of SlopCodeBench and watched it live for six hours. Opus 5 wins technically but none of them did a very good job IMO. Will post more results soon with Fable and 5.6 Sol in the mix. The big headline is that Opus 5 got a 24% on the small subset of the benchmark that I ran - not much higher than Opus 4.6's 17% strict pass rate in the original paper. All of the tested models showed a pretty significant increase in verbosity, complexity, and a bunch of other code smell metrics over the course of each challenge, with Opus 5 writing five times the number of functions/callables than Opus 4.8 over the course of the same set of challenges. My personal read of this 23% pass rate is that SlopCodeBench finally gives some signal for a thing I've only ever been able to argue from vibes - that for real-shaped software engineering work, building one issue at a time, today's models can't be relied on to run lights-off without steering. i had claude pick out 3 problems from the repo, 17 checkpoints total, a mix of easy/medium/hard labeled problems: There's an appendix at the end with all 17 checkpoints explained in detail but I won't put that all here.