Benchmarking Correctness and Security in Multi-Turn Code Generation
Abstract
AbstractAI coding assistants powered by large language models (LLMs) have transformed software development, significantly boosting productivity. While existing benchmarks evaluate the correctness and security of LLM-generated code, they are limited to single-turn tasks that cannot capture the conversational nature of real-world development, where requirements evolve across turns and tools increasingly generate incremental diffs rather than complete programs. We introduce MT-Sec, the first benchmark to jointly evaluate correctness and security in multi-turn coding scenarios. MT-Sec covers four interaction types (expansion, editing, refactoring, and debugging) reflecting common development workflows. We construct multi-turn tasks using human-curated instructions and feedback-driven interactions grounded in dynamic test suites. Evaluating 31 LLMs and 3 agent scaffolds, we find that models struggle most with expansion and editing, but improve significantly in debugging when provided with feedback. Agent-based systems often underperform their base models in automated multi-turn settings due to brittle tool use. We also evaluate multi-turn code-diff generation, an unexplored but practical setting, and observe higher failure rates compared to full-program generation. Our findings underscore the need for multi-turn benchmarks to drive safer AI-assisted development.