Article URL: https://modal.com/blog/scaling-to-1-million-concurrent-sandboxes-in-seconds Comments URL: https://news.ycombinator.com/item?id=48940231 Points: 15 # Comments: 1

At Modal, we build sandboxes, among other things. Agents run in sandboxes, and agents are eating software. Today, Modal runs millions of sandboxes per day, supports up to fifty thousand concurrent sandboxes per customer, and supports a variety of use cases at scale, from reinforcement learning to background agents. Increasingly, our users require more and more sandboxes, created at higher and higher rates. Reinforcement learning can require running millions of sandboxes concurrently, and creating bursts of hundreds of thousands of sandboxes at the beginning of rollouts. Similarly, agents increasingly require massive scale and high concurrent creation rates to deal with traffic bursts. Our existing sandbox platform is really good, but it wasn’t designed for these scales; nor is any other existing solution. We’re obsessed with scale and performance, and we want our infrastructure to accelerate the growth of agents, not add friction. So we went back to the drawing board. Over the last few months, we’ve rebuilt our core sandbox platform from the ground up for both scale and reliability. On our new system, users can run millions of sandboxes concurrently and create tens of thousands of sandboxes per second. We’ve removed all central bottlenecks from our control plane so there are no practical scaling limits, and we’ve optimized every part of container scheduling and startup, simplifying the scheduling path to a layer of load balancers which create containers directly on our worker fleet. As a demonstration of what our platform is capable of, we’ve run a million sandboxes concurrently, creating all 1 million in under a minute. Running 1 million sandboxes pushes the limits of any container platform, both because of the sheer number of containers, but also because running this many sandboxes requires many tens of thousands of compute nodes. There will be many operations which are either O(containers), O(nodes), or both, which will cause traditional container platforms to hit scaling limits. Kubernetes can be scaled, but it requires serious work. To run large numbers of nodes, etcd generally must be rewritten or replaced. Supporting high scheduling throughput requires building a complex scatter-gather system to parallelize the scheduling algorithm while still maintaining a single source of truth for pod state. Sharding and parallelization is not easy by default, because Kubernetes relies on strong consistency as a backbone of its design. Modal’s original sandbox architecture has similar issues. Like Kubernetes, we rely on strong consistency throughout our backend, so creating and scheduling sandboxes requires global coordination, and O(sandboxes) writes to Postgres, which we cannot trivially shard.