Chinese AI startup Moonshot AI, the company behind the open-source Kimi model series, has just open-sourced a new tool called Kimi Vendor Verifier (KVV). It’s designed to check whether third-party APIs running Kimi models are actually performing as well as the official ones.
Moonshot AI has been building some serious models, including the open-source Kimi K2.6 released on April 21, 2026, which it claims matches the performance of Anthropic's Claude Opus 4.6.
The company says it had been getting frequent feedback from the community about "weird benchmark scores." After investigating, it found that most of the time the issue was incorrect decoding parameters. Moonshot fixed it by forcing certain parameters on its own API side.
But then it looked at third-party APIs and found significant differences compared to its official one. Some vendors, like Fireworks AI, are putting in the work to get the best performance out of the model. Others are serving up low-quality versions.
That’s the tricky part of open models: anyone can run them, but quality control becomes a nightmare. If a user gets a lousy response from a Kimi model via a third-party vendor, they can't tell if the problem is with Moonshot's model or the vendor's setup. That’s a quick way to lose user trust.
So Moonshot built KVV, an open-source tool to verify if an AI is performing correctly and matching the official specs.
KVV checks model performance across six benchmarks:
・Pre-validation Verifies that API parameters are applied correctly.
・OCRBench A five-minute smoke test for multimodal pipelines.
・MMMU Pro Tests diverse visual inputs and validates preprocessing.
・AIME2025 A long-output stress test that catches KV cache bugs and quantization degradation not found in short benchmarks.
・K2VV ToolCall Measures consistency and JSON schema accuracy, detecting tool errors.
・SWE-Bench A full agent-style coding test. It's not included in the open-source release due to its dependency on a sandbox.
Moonshot also plans to give vendors early access to test models so they can check their systems before launch. The company will also verify vendor APIs itself and publish the results in a ranking.
Running the full KVV suite took about 15 hours on a server with two NVIDIA H20 8GPUs.
Announcing KVV, Moonshot AI said, "Making a model open is only half the battle. The other half is ensuring it runs correctly elsewhere." The company is clearly motivated to get third-party vendors to implement Kimi K2.6 properly. This push for quality comes as Moonshot's latest model, Kimi K2.6, is making bold claims about matching top-tier closed-source models in coding performance.