How much information do LLMs really memorize? Now we know, thanks to Meta, Google, Nvidia and Cornell

Researchers have discovered that GPT-style language models possess a fixed memorization capacity of approximately 3.6 bits per parameter, indicating a consistent limit to how much information these models can store. This finding provides a deeper understanding of the models' information retention capabilities and has implications for optimizing model design and assessing potential privacy risks associated with memorized data.