AI infrastructure startup Datacurve has open-sourced a new programming agent benchmark called DeepSWE, designed to evaluate cutting-edge large language models on extremely long and complex real-world software engineering tasks.
Compared to the mainstream SWE-Bench Pro, DeepSWE focuses on long-horizon planning and editing large codebases. The benchmark initially includes 113 real development tasks spanning five programming languages: TypeScript, Go, Python, JavaScript, and Rust. In DeepSWE tests, the reference solutions average 668 lines of code across 7 different files — 5.5 times the code size and file complexity of SWE-Bench Pro. To mimic how human developers delegate work, agents receive prompts averaging just 2,158 characters, forcing them to rely on minimal instructions to perform deep reasoning and navigate the codebase.
To address the common problems of pretraining contamination and model memorization in public datasets, DeepSWE uses entirely original tasks written from scratch. During evaluation, DeepSWE strips away model-specific scaffolding tools and runs all models using the open-source framework mini-swe-agent, ensuring results reflect underlying capabilities. The benchmark also replaces traditional automated scorers that depend on internal details with hand-crafted behavior verifiers that test the observable behavior of the final code, providing high-fidelity assessment accuracy.
In the first round of evaluations on frontier models, the performance gap was stark. OpenAI's flagship model gpt-5.5 took the lead with a 70% solve rate, followed by gpt-5.4 at 56%. Anthropic's claude-opus-4.7 came close at 54%, while the lightweight command-line tool claude-sonnet-4.6 managed only 32%. Google's gemini-3.5-flash hit 28%, beating gpt-5.4-mini and kimi-k2.6, both at 24%. Chinese models mimo-v2.5-pro and glm-5.1 scored 19% and 18%, respectively. Deepseek-v4-pro, which prioritizes cost efficiency, struggled on long-horizon engineering tasks with just an 8% solve rate, exposing weaknesses in multi-step planning and self-correction for large-file edits.