ByteDance Research just open-sourced Lance, a multimodal model that's surprisingly small — just 3B activated parameters — but can handle image and video understanding, generation, and editing all in one framework. The real kicker? The team trained it from scratch on a mere 128 Nvidia A100 GPUs, a fraction of what you'd expect for a model of this capability.
Most unified models today rely on scaling up parameters or repurposing text-to-image architectures. Lance takes a different, more efficient path. To manage internal conflicts between modalities and tasks, the architecture introduces two key isolation mechanisms:
- A dual-stream mixture-of-experts (MoE) architecture that processes interleaved multimodal sequences, sharing a common underlying context while decoupling the computational paths for understanding and generation.
- Modality-aware rotary position encoding, which reduces signal interference between heterogeneous visual tokens from images and videos.
Despite the extreme compute compression, Lance doesn't sacrifice performance. With only 3B activated parameters, it leads existing open-source unified models on most benchmarks for image and video generation and editing. It's a proof point that small models can still pull off multimodal tasks cheaply.