VideoMT turns vision transformers into fast video segmenters

VideoMT is a new method that repurposes standard vision transformers into high-speed video segmentation models. Instead of training a separate network, the approach propagates queries across frames, letting the transformer track and segment moving objects over time.

The system can segment people, animals, and tools in complex scenes while running up to 160 frames per second, which is 5 to 10 times faster than several previous methods. That speed makes VideoMT a candidate for real-time applications like video editing tools, AR filters, and robotics perception pipelines.

The researchers also highlight that the method works with off-the-shelf transformer backbones, reducing the need for specialized architectures.