Two million workflow runs a day sounds like a big-infrastructure problem, and in some ways it is. But most of the engineering effort went into a smaller problem: making sure a single noisy customer’s traffic never slows down anyone else’s.

Isolation over sharing

Each workflow run is scheduled on a queue partitioned by workspace, not by a global FIFO. A workspace running 50,000 runs a month cannot starve a workspace running 500. We learned this the hard way in the first six months, when one customer’s retry storm added seconds of latency to every other account on the platform for about twenty minutes.

The runtime itself is stateless between steps. Every step reads its input from durable storage and writes its output back before the next step is scheduled, which means a worker can crash mid-run and another worker picks up exactly where it left off, with no in-memory state lost.

What broke at scale

The first version stored full payloads in the run’s metadata row, which worked fine until a customer piped a 4MB JSON blob through a workflow and the metadata table started timing out on writes. Large payloads now live in object storage with a reference in the run row, and that one change cut our p99 step latency by more than half.