SCORPIO: Serving the Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference

The paper presents SCORPIO, an SLO-focused LLM serving system that enhances system goodput and adherence to service level objectives like TTFT and TPOT by leveraging adaptive scheduling and predictive modules. Evaluations show SCORPIO achieves up to 14.4 times higher goodput and 46.5% better SLO compliance compared to existing systems.