Making agents fast - what I've learnt over the last 12 months

Open original

Martin Alderson walks through what he's learned about speeding up agentic AI workflows, drawing on his web performance background. The core argument is that model inference is often not the biggest bottleneck; tool calls, shell startup, polling loops, and hardware choices frequently matter more.

  • Inference has two parts to watch: time to first token (dominated by prefill and provider queueing) and tokens per second. Which one matters depends on whether your agent sends short back-and-forth turns or long outputs, and thinking budget can dwarf both.
  • Halving inference time only cuts about 20% off a task if provisioning, tool calls, and suspend/resume stay the same. Tool call traces often reveal slow things like xargs spawning hundreds of thousands of processes or a slow .bashrc (nvm is a common culprit).
  • Polling loops with sleep are a red flag; the agent is waiting for something that should push a notification instead. Faster underlying hardware (CPU type, local NVMe versus networked disks) and careful RAM management during suspend/resume also make a measurable difference.

Speed in agent workflows splits into two layers, and the author's main point is that people over-focus on the first. Inference speed has two parts: time to first token (TTFT), which covers network, queueing and prompt prefill, and tokens per second (tok/s), which is how fast output streams once it starts. Which one matters depends on the work. Chatty agents with many short turns live and die by TTFT, while agents writing long code or documents care more about tok/s. Prompt caching is essential for TTFT, though the author says it matters less in practice than he expected. Provider choice swings speed enormously for the same model, and performance varies with time of day and load because of "noisy neighbour" effects. The sneakiest lever is thinking budget: a fast-tier model set to maximum reasoning can be far slower than a slow-tier model on low thinking, so reducing thinking can outweigh any tok/s gain. The bigger point is that inference is often not where the time goes. Halving inference time only cuts about 20% off a task if provisioning, tool execution and suspend/resume stay the same. Tool calls deserve classic profiling. The author cites an agent running `xargs -I{}` across many files, which spawns hundreds of thousands of processes and is painfully slow in sandboxes. `strace` helps find these bottlenecks but adds overhead of its own. Output size matters too, since uncached input tokens drive TTFT, so SQL-style access that returns exactly what's needed beats broad `awk` dumps. Shell startup is another hidden cost: if agents launch a fresh shell per command, a heavy `.bashrc` with nvm can cost more time than the command itself. The most original section is about building the tools agents want rather than just making existing ones faster. Watching what an agent does can reveal that it keeps grepping markdown files, which argues for a searchable database instead. Open-weights models help here because their thinking traces show what the agent was trying to do just before a slow call. Two other patterns stand out. A `for` loop with `sleep` is a red flag for polling, often at a 10 to 20 second minimum, and the fix is usually an async callback from the service. And parallelising tool calls helps, but weaker models may break things, so shipping CLI or MCP tools that do parallel work internally can be safer than relying on prompting alone.

Ask about this article