The author argues that despite improvements in open-weight models, most AI inference will remain in datacenters because local models can't match frontier performance and are actually more expensive to run. Batching hundreds of users' requests together and specialized datacenter GPUs make cloud inference roughly 30x more efficient than running models at home, and users will always prefer the strongest available model in their budget.

• Datacenter inference beats local by ~30x on efficiency due to request batching and specialized GPUs (e.g., B200 vs RTX 4090)
• A home GPU rig's upfront cost plus $50-300/month in power outweighs just paying for years of API access
• Users always gravitate to the strongest model they can afford, so smaller local models keep losing ground even as they improve
• Local models will persist only in niches like low-latency voice interfaces, privacy-focused use, or unreliable internet—not as the dominant paradigm

The article argues that enterprises should measure AI infrastructure economics by cost per token rather than raw compute metrics like FLOPS per dollar. It shows how maximizing delivered tokens—through hardware, software and system optimizations—drives down real-world cost and boosts revenue, citing NVIDIA Blackwell’s 35× lower token cost versus Hopper.

• Cost per token (total infra cost ÷ tokens generated), not FLOPS/dollar or GPU hourly rate, is the real measure of AI infrastructure efficiency.
• Blackwell GB300 NVL72 costs almost 2x more per GPU-hour than Hopper H200 ($2.65 vs $1.41), but delivers 65x the tokens/sec per GPU (6,000 vs 90).
• That throughput gap translates to 50x more tokens per megawatt and a 35x lower cost per million tokens ($0.12 vs $4.20).
• Techniques like FP4 precision, speculative decoding, KV-cache offloading, and disaggregated serving are necessary, not optional, to actually achieve these lower token costs.