Scaling to 1 million concurrent sandboxes in seconds
To optimize for scale, we decided that everything taking O(sandboxes) or O(nodes) load must be horizontally scalable by default, the sandbox creation path should be as simple as possible, and everything else should be secondary. The solution we came to is notably different than existing systems. We completely dispensed with any sort of central coordination, and traded global consistency for scalability and performance everywhere on the critical path for running and creating sandboxes.
I have nowhere near the same requirements for my personal sandbox setup but it still super interesting to see how what is needed to reach massive scale.