AI startup Recursive has released the first results from its research system — an automated engine that proposes ideas, writes code, runs experiments, and validates outcomes. The system beat the best publicly known results on three separate benchmarks: fixed-budget training, a NanoGPT speedrun, and GPU kernel optimization. The takeaway? In tasks with clear objectives and fast feedback loops, the AI is already finding optimization headroom that humans missed.
In a five-minute NanoChat Autoresearch training run, the system pushed validation loss down to 0.9109 BPB, cutting the time needed to reach the same loss by roughly 23% (a 1.3× speedup). The key tweak: enhancing short-context memory by hashing binary and ternary token combinations into a fixed embedding table, then mixing them into the attention value path via learnable gating — letting the model exploit local information at minimal cost.
On the NanoGPT Speedrun benchmark — already optimized by the community for over two years — the system shaved the time to hit a target loss from 79.7 seconds to 77.5 seconds. It did this by pushing FP8 forward computation into the attention path to boost throughput, and by rewriting the fused MLP kernel to store only the squared ReLU activations and recompute intermediate values during backpropagation, cutting memory bandwidth.
On SOL-ExecBench, a GPU kernel optimization benchmark, the system ran on Nvidia's B200 GPU and raised the average SOL score (the percentage of theoretical peak performance) from 0.699 to 0.754, closing the gap to the physical limit by 18%. Its generated solutions included absorbing GRN scaling into subsequent linear layer weights, packing expert routing scores and indices into key-value pairs for intra-warp reduction, and using low-level PTX instructions to pack FP4 in an NVFP4 MoE kernel while retaining FP32 for intermediate calculations to limit error accumulation. To keep the AI from gaming the metric, the system includes multi-level correctness audits that filter out invalid speedups.