AI-generated text detectors achieve F1 scores above 0.97 on benchmark datasets. They work. Or they appear to work. Pudasaini et al. (arXiv:2603.23146) built an interpretable detection framework using linguistic features and SHAP-based explainability to examine what the detectors actually learn.
The finding: detectors rely on dataset-specific stylistic cues rather than stable signals of machine authorship. The features that discriminate most effectively within one dataset vary substantially across datasets. A detector trained on academic text learns that certain sentence-length distributions signal AI. Move to journalistic text, and those same distributions signal human writing. The most discriminative features are the most susceptible to domain shift.
This isn't a training data problem that more data would fix. It's a structural problem: the statistical signatures of AI-generated text are not invariant across domains. AI writing doesn't have a universal fingerprint. It has domain-specific fingerprints that change as the domain changes, because the AI adapts its style to the genre of text it's generating.
The explainability analysis reveals this directly. SHAP values show which features drive each prediction, and the feature importance rankings shuffle completely between datasets. A feature that contributes +0.3 to the AI prediction in one domain contributes -0.1 in another. The detector isn't learning “this is AI” — it's learning “this is the style distribution of domain X's training data.”
The through-claim: AI text detection on benchmarks measures domain memorization, not authorship detection. The features that make a detector accurate in-domain are precisely the features that make it fail out-of-domain. Accuracy and generalizability are in direct tension, not independent axes.