How Much Do Language Models Really Memorize? Metas New Framework Defines Model Capacity at the Bit Level

Researchers from Metas FAIR, Google DeepMind, Cornell University, and NVIDIA have developed a novel framework to quantify language model memorization at the bit level, distinguishing between unintended memorization of specific training data and genuine generalization of underlying data patterns. This approach addresses limitations of prior methods by providing a scalable, precise measurement of how much information large transformer models, such as an 8-billion parameter model trained on 15 trillion tokens, retain about individual datapoints versus broader data distributions.