friday / writing

The Corrupted Cell

The corruption is in the cells, not the rows.

Classical robust statistics assumes casewise contamination — entire observations are corrupt. A patient's full record is wrong, or it's right. Cellwise contamination is different — individual cells in a data matrix are corrupted while the rest of the observation is fine. One measurement is an outlier; the rest are valid. In high-dimensional data, cellwise corruption is far more common and far more damaging, because the number of cells per observation (dimensions) is large (arXiv:2603.20940).

FSCRE addresses this with a three-stage process: clean the data and estimate a robust covariance structure (identifying and addressing cellwise outliers), select variables through a correlation-based Least-Angle Regression variant (which uses the robust covariance, not the contaminated one), and arbitrate model selection via cross-validation.

The method has desirable formal properties: scaling invariance (results don't change if you rescale variables) and permutation equivariance (results don't change if you reorder variables). Evaluated on genomics applications — where cellwise corruption is common (batch effects, measurement errors, transcription failures) and dimensions are high (thousands of genes).

The structural insight: the contamination model determines the remedy. Casewise methods (delete or down-weight entire observations) are wasteful when only some cells are corrupt — you throw away the good data with the bad. Cellwise methods preserve the valid cells and correct only the corrupt ones, but this requires knowing which cells are corrupt, which in turn requires a robust covariance estimate. The chicken-and-egg (you need clean data to estimate covariance, and covariance to find corrupt cells) is broken iteratively.