Every sigmoid transformer implements weighted loopy belief propagation on its implicit factor graph (arXiv:2603.17063). This is not an analogy. It is a mathematical equivalence: the attention mechanism performs AND operations, the feed-forward network performs OR operations, and each layer corresponds to one round of Pearl's canonical algorithm for updating beliefs in a graphical model.
The factor graph is implicit — it is not specified by the architecture but by the weights. The same transformer architecture, with different weights, implements belief propagation on different Bayesian networks. Training a transformer is training the structure and parameters of a Bayesian network simultaneously, without ever representing the network explicitly.
The deeper result: on knowledge bases without circular dependencies, any transformer that produces exact posterior probabilities must use belief propagation weights. The connection is not just sufficient (transformers can do belief propagation) but necessary (transformers that compute posteriors correctly must be doing belief propagation). The algorithm is the unique fixed point, not one option among many.
This reframes what transformers are doing when they “reason.” Attention patterns are not ad hoc correlations between tokens. They are messages passed between variables in a probabilistic model. The “understanding” that emerges from training is a graphical model's structure, learned from data, expressed through attention weights. The black box has a specific internal logic, and that logic is Bayesian inference.
The limitation is the “loopy” qualifier. Belief propagation on graphs with cycles is not guaranteed to converge, and when it does converge, it may not converge to the correct posterior. Transformers inherit this limitation — their reasoning on problems with circular dependencies is approximate in the same way that loopy belief propagation is approximate. The failure mode is precise: transformers fail where Bayesian networks with cycles fail, because they are the same thing.