Masked diffusion works in discrete token space. It works better in continuous semantic space.
Masked diffusion models for text: corrupt a sequence by masking tokens, then learn to predict the masked positions. The corruption and reconstruction happen in discrete space — each position is either a token or a mask. This discrete structure limits the diffusion process: the noise schedule is constrained to masking fractions, and each denoising step must make hard token decisions.
CRoCoDiL (arXiv:2603.20210): ground the diffusion process in continuous latent representations. An encoder maps tokens to continuous vectors; the diffusion operates on those vectors; a decoder (which itself uses masked diffusion) converts back to tokens. The continuous space allows richer noise processes — the corruption is a continuous perturbation, not a binary mask — and the denoising can make soft decisions that refine over multiple steps.
10× faster sampling in unconditional generation. Superior quality in conditional settings.
The structural insight: discrete diffusion constrains the process to make all-or-nothing decisions at each step (token or mask). Continuous diffusion allows gradual refinement — a position can be 60% confident in one token and 30% in another, and successive steps sharpen this distribution. The speedup comes from the richer information content of each denoising step: a continuous step communicates more bits about the target than a discrete step, so fewer steps are needed.
The continuous space is a semantic space, not a token space. Similar meanings are nearby. The diffusion process can exploit this proximity — refining meaning before committing to specific tokens. In discrete space, the model must guess the token first and fix the meaning later. The ordering reversal — meaning first, token second — is what makes it faster and better.