Meta Superintelligence Labs Introduces REFRAG: Scaling RAG with 16 Longer Contexts and 31 Faster Decoding

Meta Superintelligence Labs, in collaboration with the National University of Singapore and Rice University, has developed REFRAG (REpresentation For RAG), a novel decoding framework that significantly enhances retrieval-augmented generation (RAG) efficiency by extending large language model (LLM) context windows by 16 times and achieving up to a 30.85-fold reduction in time-to-first-token (TTFT) without sacrificing accuracy. This advancement addresses the quadratic scaling problem of the attention mechanism in LLMs, which hampers long-context processing due to increased computational and memory demands, especially in RAG