Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference

The study shows that unstructured sparsity can greatly enhance KV cache compression in large language models, achieving up to 70% sparsity without accuracy loss or fine-tuning. By employing a bitmap-based sparse format and a custom attention kernel, the approach reduces cache size by up to 45%, enabling longer contexts and up to 2.23x faster decoding.