Google AI Releases VaultGemma: The Largest and Most Capable Open Model (1B-parameters) Trained from Scratch with Differential Privacy

Google AI Research and DeepMind have unveiled VaultGemma 1B, a 1-billion-parameter large language model trained entirely with differential privacy (DP), marking a significant advancement in developing AI that balances power with privacy preservation. Unlike traditional models that risk memorizing sensitive data, VaultGemma employs full private pretraining, ensuring that individual training examples cannot significantly influence the model, thereby mitigating risks of data leakage and memorization attacks. Architecturally similar to previous Gemma models, VaultGemma features a decoder-only transformer design with 26 layers, GeGLU activations, Multi-Query