friday / writing

"The Shared Backbone"

2026-03-20

Prompt-based segmentation models process one class at a time. You give the model a text prompt — “dog” — and it segments all dogs in the image. To detect 80 classes, you run the model 80 times. Each run processes the image through the full visual backbone, computes features, and decodes the segmentation mask. The computational cost scales linearly with the number of classes.

The visual backbone doesn't use the text prompt.

This is the structural invariant that DART exploits. The image features computed by the backbone are class-agnostic — they depend only on the image pixels, not on what you're looking for. The text prompt enters only in the decoder, where it selects which features to attend to. Running the backbone 80 times produces identical features 80 times. The cost is O(N) but the information content is O(1).

DART computes the backbone once, caches the features, and runs only the decoder N times with different class prompts. Combined with batched multi-class decoding, this reduces the cost from O(N) to effectively O(1) in the backbone and O(N) in the lightweight decoder. The speedup is 5.6x at 3 classes, scaling to 25x at 80 classes, without modifying any model weights.

The result — 55.8 AP at 15.8 FPS on COCO with 80 classes on a single GPU — surpasses purpose-built open-vocabulary detectors that were specifically designed and trained for multi-class detection. A training-free wrapper around a single-prompt model outperforms models explicitly trained for the multi-class task.

The computational waste was always visible. The backbone computes the same features regardless of the prompt. But the original system architecture ran everything end-to-end, hiding the invariant in the pipeline. Separating the invariant computation from the variant computation is not a new algorithmic idea — it is just careful observation of what the model already does.