@andrewchen - Andrew Chen Shares Homelab

Open original

@andrewchen on X Andrew Chen lays out his homelab for running AI models locally, built around a Framework Desktop, an RTX 5090 eGPU, and two DGX Sparks. He routes most requests to local models through a small router model, aiming for 100% local use, and admits the setup is overbuilt and regrets the eGPU.

  • A Qwen 3.8 27B model on a 5090 eGPU hits 150+ tokens per second, making it the fast local option.
  • Two DGX Sparks running DeepSeek V4 Flash handle slower batch work like cron jobs and long dev builds.
  • A homegrown Hermes plugin uses a small Arch-Router model to decide between local and cloud, currently keeping about 60-70% of work local.

Andrew Chen lays out his homelab for running AI locally, and the core design is a routing setup. Hermes, a box built on a Framework Desktop with an AMD AI Max+ 395, is the default for everything. A 5090 eGPU runs Qwen 3.8 27B for quick interactive work, sometimes hitting 150+ tokens per second. Two DGX Sparks run DeepSeek V4 Flash 0731, which he calls better but slower. A Pi 5 handles monitoring, and a Mac mini serves as the dev machine, where he switches between Herdr, ohmypi, Codex, and Claude depending on the task. The whole thing sits in a 10-inch DeskPi mini rack, mostly. The interesting part is how the routing works. A homegrown plugin in Hermes sends each request to Arch-Router, a small model chosen for low time-to-first-token, and that model decides whether the job stays local or goes to a cloud frontier model. He's currently at about 60 to 70 percent local and wants to push toward 100. The Sparks get the batch work: cron jobs, longer dev builds, background runs. So the split is roughly interactive on the eGPU, heavy and unattended on the Sparks, and the cloud as an escape hatch. He's candid that he doesn't need most of it. The Mac mini came first and the rest accumulated. His one firm recommendation is don't buy an eGPU, which he regrets.

Ask about this article