Interchangeable KV Importance Signals under One Compiled FlexAttention Budget with FlexBudget
Video by PyTorch via YouTube When you run a model over a long context, the KV cache gets huge, so you have to throw some of it away. There are competing rules for choosing what to drop: keep the most recent tokens, keep the ones that get the most attention, and so on. The problem … Read more