Differentially Private CutMix for Split Learning with Vision Transformer
This work addresses privacy and efficiency issues in distributed learning for computer vision, particularly with vision transformers, but is incremental as it builds on existing split learning and differential privacy techniques.
The paper tackles the problem of privacy leakage and high communication costs in split learning with vision transformers by proposing DP-CutMixSL, a differentially private framework that uses patch-level randomized CutMix, achieving higher accuracy than vanilla split learning while improving privacy and efficiency.
Recently, vision transformer (ViT) has started to outpace the conventional CNN in computer vision tasks. Considering privacy-preserving distributed learning with ViT, federated learning (FL) communicates models, which becomes ill-suited due to ViT' s large model size and computing costs. Split learning (SL) detours this by communicating smashed data at a cut-layer, yet suffers from data privacy leakage and large communication costs caused by high similarity between ViT' s smashed data and input data. Motivated by this problem, we propose DP-CutMixSL, a differentially private (DP) SL framework by developing DP patch-level randomized CutMix (DP-CutMix), a novel privacy-preserving inter-client interpolation scheme that replaces randomly selected patches in smashed data. By experiment, we show that DP-CutMixSL not only boosts privacy guarantees and communication efficiency, but also achieves higher accuracy than its Vanilla SL counterpart. Theoretically, we analyze that DP-CutMix amplifies Rényi DP (RDP), which is upper-bounded by its Vanilla Mixup counterpart.