QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining

The paper introduces QLIP, a drop-in replacement for the CLIP vision encoder, designed to overcome limitations such as fixed input resolution and inability to produce separate embeddings for dissimilar images, by utilizing an image quadtree with content-aware patchification. QLIP can be integrated into existing MLLMs with minimal code, significantly improving visual understanding and question answering accuracyup to 13.6% on the V* benchmarkwithout requiring retraining or fine-tuning.