AxDafny: Agentic Verified Code Generation in Dafny
Benjamin Breen, Austin Letson, Borja Requena Pozo, Leopoldo Sarra
Read on arXiv →
Key claim
AxDafny achieves 92.7% verification success in code generation.
AxDafny improves the success rate of code verification significantly compared to previous models. It achieves a 92.7% verification success rate on a new benchmark, demonstrating its effectiveness in generating both code and necessary proof artifacts.
In plain English
Imagine you're trying to create software that not only runs correctly but also proves that it does so. This is a tough challenge because generating code that meets both functional and formal verification requirements is complex. Currently, many models can generate code, but they often fail to provide the necessary proofs, leading to unreliable software. This is what's called a verification gap, where the code might work but lacks the formal backing to ensure it does so under all conditions.
The approach in this paper addresses this gap by introducing a framework that guides the code generation process with verification in mind. It iteratively creates not just the code but also the proofs needed to verify that the code behaves as expected. This means that instead of just hoping the generated code is correct, the model actively works to ensure it can be verified.
What’s particularly noteworthy is the introduction of a new benchmark, LiveCodeBench-Pro-Dafny, which consists of 250 programming problems designed for this purpose. The results show that AxDafny significantly outperforms previous models, achieving a 92.7% verification success rate, which is a notable improvement over existing methods. For anyone building systems that require reliable code generation, this framework offers a promising solution to ensure both functionality and correctness.
The introduction of a verifier-guided repair framework for code generation is a significant advancement in the field.
The paper provides strong experimental results with a new benchmark and clear comparisons to existing methods.
Deep reliability assessment
The methodology supports the claim that verifier-guided repair can improve verification success in Dafny, but it may overclaim general applicability due to the limited scope of benchmarks and the specific nature of Dafny's formal verification environment.
Reproducibility
Yes, the paper mentions a public GitHub repository with code, benchmark data, and evaluation harness: https://github.com/Axiomatic-AI/ax-dafny.
Discussion questions
- 1.The authors claim that AxDafny significantly improves verification success over GPT-5.5. Is the comparison with GPT-5.5 a fair baseline, or should they have considered a different model for a more rigorous comparison?
- 2.The paper highlights that verification success and runtime test performance measure different aspects of generated code. In your experience, how often do these two metrics align, and which do you prioritize in your projects?
- 3.Has anyone here worked with formal verification in programming languages like Dafny? What challenges did you face, and how does this paper's approach address or overlook those challenges?
- 4.The authors note that Dafny specifications enforce functional correctness but not asymptotic complexity. Would you prefer a system that balances both, or is focusing on correctness sufficient for your needs?
- 5.The paper introduces a new benchmark, LCB-Pro-Dafny. How important is it to have benchmarks specifically tailored to formal verification, and do you think this benchmark will be widely adopted?
Key figure
Figure 1 illustrates a verified Dafny method for computing an arithmetic sum, highlighting executable statements, method specifications, and proof annotations.