Mooncake has joined the PyTorch Ecosystem, enhancing the capabilities for deploying large language models (LLMs). It addresses the “memory wall” issue in LLM serving, which becomes a bottleneck as context lengths and model sizes increase. By decoupling key-value (KV) cache from specific GPU workers, Mooncake allows for greater throughput and scalability. Its features include the separation of prefill and decoding tasks, global reuse of KV cache across requests, and fault-tolerant distributed backend support. The technology originated from a collaboration between Moonshot AI and Tsinghua University. It has since gained traction among major organizations like Alibaba Cloud, JD.com, and Tencent, who use it to optimize GPU resources for millions of users. Mooncake’s architecture has been tested in demanding production settings, proving its effectiveness in improving resource utilization and service reliability. The article details a joint solution that integrates Mooncake with leading inference engines and tools. For instance, it outlines a setup using RoleBasedGroup (RBG) to orchestrate the deployment of SGLang and the Shepherd Model Gateway (SMG). This configuration allows efficient routing of requests based on cache locality and system load. The deployment examples include specific YAML configurations for different roles within the architecture, demonstrating how Mooncake can enhance both prefill and decode tasks using RDMA for high-speed KV cache transfer.