From Transformers to Associative Memory, How Titans and MIRAS Rethink Long Context Modeling

Google Research has introduced Titans and MIRAS, innovative approaches to enhance sequence models with usable long-term memory while maintaining parallel training and near-linear inference efficiency. Titans is a novel architecture that integrates a deep neural memory modulea multi-layer perceptroninto a Transformer backbone to provide precise long-term memory, whereas MIRAS offers a general framework interpreting sequence models as online optimization over associative memory, addressing the quadratic scaling limitations of traditional attention mechanisms and improving performance on tasks requiring extremely long context, such as genomic modeling.