Bilateral Distribution Compression: Reducing Both Data Size and Dimensionality
This addresses the challenge of handling modern large-scale datasets efficiently for machine learning applications, though it appears incremental as it builds on existing distribution compression methods.
The paper tackles the problem of compressing large datasets in both sample size and dimensionality by proposing Bilateral Distribution Compression (BDC), a two-stage framework that reduces data size and dimensionality while preserving the distribution, with experiments showing it achieves comparable or superior performance to ambient-space compression at lower cost.
Existing distribution compression methods reduce dataset size by minimising the Maximum Mean Discrepancy (MMD) between original and compressed sets, but modern datasets are often large in both sample size and dimensionality. We propose Bilateral Distribution Compression (BDC), a two-stage framework that compresses along both axes while preserving the underlying distribution, with overall linear time and memory complexity in dataset size and dimension. Central to BDC is the Decoded MMD (DMMD), which quantifies the discrepancy between the original data and a compressed set decoded from a low-dimensional latent space. BDC proceeds by (i) learning a low-dimensional projection using the Reconstruction MMD (RMMD), and (ii) optimising a latent compressed set with the Encoded MMD (EMMD). We show that this procedure minimises the DMMD, guaranteeing that the compressed set faithfully represents the original distribution. Experiments show that across a variety of scenarios BDC can achieve comparable or superior performance to ambient-space compression at substantially lower cost.