Where the Blocks Live: Tier-Aware KV-Cache Routing for vLLM at Scale

Video by PyTorch via YouTube

When GPU memory fills up, vLLM offloads the KV cache to CPU or storage. llm-d tracks where every block lives, and sends each request to its cache. At PyTorch Conference North America, Bongwoo Bak from Moreh will present a poster on "Where the Blocks Live." You’ll see when CPU is enough, and when shared storage pays off.

Join us in San Jose, October 20-21: https://hubs.la/Q04v4SL60

Source