VERINA: Benchmarking Verifiable Code Generation

A new benchmark called Verina has been introduced to evaluate the ability of large language models (LLMs) to generate verifiable code, including code, specifications, and proofs, across 189 curated tasks in Lean. The evaluation reveals significant challenges, with the best model achieving only 61.4% correct code and minimal success in proof generation, highlighting the need for advancements in LLM-based verification methods.