Generative AI, particularly Large Language Models (LLMs), is much cheaper to operate than commonly believed, with costs decreasing significantly in recent years. A comparison of LLM pricing to web search APIs shows that LLMs can be an order of magnitude less expensive, challenging misconceptions about their operational costs and sustainability. The article aims to clarify these points for those who hold the opposite view.

Together AI offers a powerful API for running inference on over 200 open-source models, providing a cost-effective and fast solution compared to major competitors like OpenAI and Azure. The service is designed for scalability, utilizing optimized NVIDIA GPUs and proprietary technologies to enhance performance while maintaining privacy standards. Flexible deployment options cater to various customer needs, from managed serverless solutions to dedicated GPU clusters.

PyTorch and vLLM have been integrated to enhance generative AI applications by implementing Prefill/Decode Disaggregation, which improves inference efficiency at scale. This collaboration has optimized Meta's internal inference stack by allowing independent scaling of prefill and decode processes, resulting in better performance metrics. Key optimizations include enhanced KV cache transfer and load balancing, ultimately leading to reduced latency and increased throughput.