Xiaomi's large model application team has released and open-sourced ControlFoley, a framework for generating video sound effects. Previous AI video dubbing primarily relied on the model inferring sounds from visuals, giving creators little control over the audio style. ControlFoley's focus is "controllability": it can generate sounds from video frames, but also accepts text descriptions or reference audio to let creators dictate the audio style. For example, changing a knock on a door to "metallic clang" or matching a drum sound to a tennis racket swing — the model maintains audio-visual sync while adhering to the specified style. Under the hood, ControlFoley uses a spatiotemporal audio-video encoder based on CAV-MAE and introduces a "time-timbre decoupling" strategy, leaving sound timing to the video and timbre to a reference audio clip.
In multi-task evaluations set by the paper, ControlFoley achieves open-source state-of-the-art on several conventional video dubbing benchmarks. Even when text instructions strongly contradict the visual content, the model still balances textual adherence and temporal synchronization. Compared to the commercial closed-source system Kling-Foley, ControlFoley is competitive in semantic alignment, synchronization, and perceptual quality. However, it still lags on some KL divergence matching metrics in Kling-Audio-Eval and MovieGen-Audio-Bench. The technical report, code, model weights, and demo are all available now.