You don't need to label the whole test set. You need to label the right 5%.
Active Testing for NLP (arXiv:2603.21840): instead of annotating every example in a test set, select the most informative subset for human labeling and estimate model performance from that subset. 95% reduction in annotation cost with performance estimation within 1% of the full test set.
The selection strategies use embedding spaces to identify examples that maximally reduce estimation uncertainty — examples near decision boundaries, in underrepresented regions, or with high model disagreement. Different strategies work for different tasks and data characteristics; no single method dominates universally.
An adaptive stopping mechanism automatically determines when enough examples have been labeled — the estimation has converged and additional labels won't meaningfully improve accuracy. This eliminates the need to predetermine the annotation budget, which is usually guessed based on cost constraints rather than statistical reasoning.
The finding across 18 datasets: the informative examples are a small fraction of the total. Most test examples are “easy” — the model's prediction is obvious and labeling it provides no new information about model quality. The hard examples — where the model is uncertain or wrong — are concentrated in specific regions of the embedding space.
The structural insight: evaluation is an estimation problem, not a census. The standard practice (label everything, compute accuracy) is exact but wasteful. Strategic sampling (label the informative subset, estimate accuracy) is approximate but efficient. The approximation error is negligible because most examples are redundant for the purpose of estimating performance. The test set has much lower effective dimensionality than its size suggests.