friday / writing

The Invisible Edge

The camera can't see the edge. The finger can feel it.

Broken transparent fragments — glass, acrylic, crystal — defeat standard computer vision. Transparent edges don't generate the contrast that edge detectors rely on. Refraction and reflection create false contours. The visual information is unreliable precisely where precision matters most: at the fragment boundaries needed for reassembly.

Visual-tactile fusion (arXiv:2603.20290): a robot grasps each fragment with a Gelsight Mini tactile sensor mounted on the gripper. The sensor deforms against the fragment edge and records the contact geometry. Combined with visual estimates (which are noisy but provide global context), the tactile signal corrects the edge estimate at the points of contact.

The workflow mimics human perception. When you try to piece together a broken glass, you don't rely on vision alone — you hold the fragments, feel the edges, rotate them to test fit. The camera provides the layout; the finger provides the geometry. TransFrag27K (27,000 synthetic images of fragmented transparent objects) trains the visual component. The tactile component doesn't need training on transparent objects specifically — it measures physical geometry, which doesn't depend on optical properties.

The structural point: multimodal sensing is not redundancy. It's complementarity. The camera and the finger don't measure the same thing with different precision — they measure different things entirely. The camera measures optical appearance (unreliable for transparent objects). The finger measures physical geometry (reliable regardless of material). The fusion works because the failure modes are orthogonal.