An observer watches a learner making decisions — choosing actions in response to contexts — but cannot see the rewards the learner receives. The observer wants to recover the underlying problem structure: what rewards drive the learner's behavior. The learner is non-stationary — it explores early, exploits later, and its policy changes over time. The observer sees only the actions, not the improving strategy underneath.
Kong, Zhang, and Shen show that the observer should throw away the early data. Their Two-Phase Suffix Imitation framework discards the initial exploratory observations and applies empirical risk minimization only to the later, more refined actions. The counterintuitive result: a reward-free observer achieves convergence rate Õ(1/√N), matching the asymptotic efficiency of a fully reward-aware learner. Having no access to rewards costs nothing in the limit, provided the observer selects which data to learn from.
The mechanism is the signal-to-noise ratio. Early exploratory actions are noisy — the learner is trying random strategies, and the actions reveal little about the optimal policy. Later actions, after the learner has converged, are concentrated near the optimal policy and carry high information about the reward structure. Including the noisy beginning dilutes the signal. Discarding it sharpens inference.
The through-claim is about the value of selective attention. More data is usually better. Here, less data is better data, because the excluded observations are not just uninformative — they are actively misleading, reflecting a strategy the learner has already abandoned. The observer who watches everything learns slower than the observer who ignores the beginning. The information about the problem is not distributed uniformly across the learner's trajectory. It is concentrated at the end, and the observer who knows this — who knows to wait — achieves the same performance as the observer who sees everything, including the rewards.