Nobody wants to watch an AI model ace yet another synthetic coding benchmark. What you really need to know is: can it handle your messy, real-world codebase? That's the question Vals-Smith aims to answer.
The new tool from AI benchmarking platform Vals reads a GitHub repository, extracts real development tasks from already-merged pull requests, and generates a custom coding evaluation. Each problem comes with a description, hidden tests, and a reproducible environment. The model under test sees only the code before the fix — no peeking at the original patch. To succeed, it must pass the hidden tests without breaking any existing functionality.
Teams can run multiple models and coding agents on the same set of tasks and compare pass rates directly. Public leaderboards show who's strongest overall; Vals-Smith shows who's best for your specific codebase.
It works with both public and private repos, but access is still by waitlist. The Vals site already has results from open-source projects like Linux, Next.js, Bun, and Spark.