Video by PyTorch via YouTube
When you run a model over a long context, the KV cache gets huge, so you have to throw some of it away. There are competing rules for choosing what to drop: keep the most recent tokens, keep the ones that get the most attention, and so on. The problem is that every method ships its rule welded to its own kernel. So if one method beats another, you can’t tell whether the rule was smarter or the code was just faster.
At PyTorch Conference North America, Huanran Wang of AMD will present the poster: FlexBudget: Interchangeable KV Importance Signals under One Compiled FlexAttention Budget
We hold the kernel and the memory budget fixed, and swap only the rule. Same code, same allowance, honest comparison. It’s training-free, and it’s all core PyTorch.
Join us in San Jose, October 20-21 to learn more: https://hubs.la/Q04v4SL60
#PyTorchCon