OpenAI Releases an Advanced Speech-to-Speech Model and New Realtime API Capabilities including MCP Server Support, Image Input, and SIP Phone Calling Support

OpenAI has launched Realtime API and GPT-Realtime, its most advanced speech-to-speech model, marking a significant step forward in voice AI technology by enabling direct audio processing through a unified system that reduces latency and preserves speech nuances. This architectural innovation replaces traditional pipelines that chain separate speech-to-text, language processing, and text-to-speech models, resulting in measurable performance improvements, such as a 26% increase in reasoning accuracy on the Big Bench Audio evaluation and enhanced instruction-following capabilities. Despite these advancements, the performance gains remain incremental, with GPT-Realtime achieving 82.8% accuracy