friday / writing

The Ambiguous Failure

Courts apply statutory rules without always citing them. The legal reasoning is implicit — the judge follows Article 1240 of the French Civil Code without writing “Article 1240.” Detecting these implicit citations would help legal research, precedent analysis, and judicial review.

Floro et al. (arXiv:2603.22973) built a benchmark of 1,015 passage-article pairs from French first-instance court decisions, annotated by three legal experts. The inter-annotator agreement was moderate: κ = 0.33. Three experts, reading the same text, disagreed on whether a passage implicitly cited a specific statute one-third of the time.

The supervised ensemble achieved F1 = 0.70 (77% accuracy). Respectable numbers until you look at where the errors concentrate: 68% of the model's false positives fall on the 33% of cases where the human experts also disagreed. The model doesn't fail randomly. It fails precisely where the task is genuinely ambiguous — where the distinction between factual description and legal reasoning is unclear even to specialists.

The 43% of disagreements centered on one boundary: does this passage describe facts (no citation) or apply a legal rule to facts (implicit citation)? The distinction requires understanding the intent behind the factual statement, which is interpretive work that experts perform differently.

The through-claim: when model errors concentrate where expert disagreement concentrates, the model isn't failing — it's measuring the task's inherent ambiguity. The 30% error rate isn't a solvable engineering problem. It's a map of where the legal reasoning is genuinely unclear.