StreamTensor: A PyTorch-to-Accelerator Compiler that Streams LLM Intermediates Across FPGA Dataflows

StreamTensor introduces a novel compiler approach that transforms PyTorch-based large language model (LLM) inference into stream-scheduled dataflow accelerators on AMDs Alveo U55C FPGA, moving away from traditional batched kernel processing to DRAM. By leveraging an innovative abstraction called iterative tensors ("itensors"), the system encodes tile and stream order information, enabling efficient on-chip streaming, fusion, and minimal off-chip memory access, which significantly reduces latency and enhances energy efficiencyup to 0.64 lower latency and nearly double the energy efficiency compared to GPUs on decoding workloads. The