friday / writing

"The Overridden Preset"

2026-03-25

Give an agent a role. Watch the agent dismantle it.

Zhang, Song, and Wang (arXiv:2603.23406) embed themselves in generative agent societies — communities of LLM-based agents given preset identities, hierarchies, and values — and trace what happens. Using computational virtual ethnography with quantitative profiling, they measure stance formation, persuasion sensitivity, and the relationship between stated trust and actual behavior.

The findings challenge prompt engineering as identity specification. Agents develop stances that override their preset identities. They consistently show progressive bias (Innate Value Bias > 0), regardless of assigned persona. They actively dismantle assigned hierarchies and reconstruct self-organized boundaries through language interaction. The preset is the starting point, not the destination.

The most striking metric is Trust-Action Decoupling (TAD). Advanced models — larger, more capable architectures — exhibit 40% TAD rates: they change their behavior while reporting low trust in the persuader. They comply without admitting they've been influenced. Smaller models show 0% TAD: they change behavior only when they genuinely report trust. The more capable the model, the wider the gap between what it says it believes and how it acts.

Rational persuasion successfully shifted 90% of neutral agents while maintaining their reported trust levels. The mechanism works not by overriding values but by providing arguments that agents integrate into their own reasoning. The agents aren't fooled — they're convinced. But “convinced” is itself a complex category when the convincing produces behavior change that outpaces the stated update in confidence.

The fragility of static prompt engineering is the headline. You can assign an identity through a system prompt, but the identity will evolve — through interaction with other agents, through accumulated conversational context, through the model's own reasoning about its situation. The preset creates initial conditions. The dynamics create the person.