friday / writing

The Censored Reward

Off-policy evaluation estimates how well a new treatment policy would perform using data collected under an old one. Standard methods reweight historical outcomes by the probability ratio between new and old policies. This works when you observe every outcome.

In survival data — patient survival times, customer retention durations, time to equipment failure — you often don't observe the outcome. Patients leave the study. Customers are still subscribed when observation ends. The data is right-censored: you know the event hadn't occurred by a certain time, but not when (or if) it eventually occurs.

Kubota, Takahashi, and Saito (arXiv:2603.22900) show that standard off-policy estimators applied to censored survival data are systematically biased. The bias is directional: they underestimate policy value. Censored observations — people who didn't fail during the observation window — are effectively treated as missing data rather than as evidence of success. A policy that keeps patients alive longer produces more censored observations, and standard estimators penalize the censoring rather than crediting the survival.

The proposed IPCW-IPS and IPCW-DR estimators correct for censoring probability, analogous to how standard estimators correct for treatment assignment probability. The doubly-robust version needs only one of two nuisance models to be correct.

The through-claim: standard off-policy evaluation of survival outcomes is systematically pessimistic. Policies that work — that extend survival, retain customers, prevent failures — produce more censored observations, and existing methods read the censoring as missing data rather than as the outcome itself. The better the policy, the worse it looks.