KServe v0.15 has been released, enhancing capabilities for serving generative AI models, including support for large language models (LLMs) and advanced caching mechanisms. Key features include integration with Envoy AI Gateway, multi-node inference, and autoscaling with KEDA, aimed at improving performance and scalability for AI workloads. The update also introduces a dedicated documentation section for generative AI and various performance optimizations.

The guide outlines how to deploy large language models (LLMs) at scale using Google Kubernetes Engine (GKE) and the GKE Inference Gateway, which optimizes load balancing by considering AI-specific metrics. It provides a step-by-step walkthrough for setting up an inference pipeline with the vLLM framework, ensuring efficient resource management and performance for AI workloads. Key features include intelligent load balancing, simplified operations, and support for multiple models and hardware configurations.