TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimization
This work improves the feasibility of privacy attacks in federated learning with transformers, particularly under differential privacy defenses, for security researchers and practitioners concerned with data leakage.
TIGER introduces a continuous gradient inversion attack that optimizes token embeddings to minimize their distance to a subspace identified from attention gradients, achieving higher reconstruction quality and robustness under differential privacy compared to prior attacks on transformer models.
Federated learning allows multiple clients to jointly train a shared model by sending gradient updates to a central server while keeping raw inputs local. However, prior gradient inversion attacks show that these updates can reveal enough information to reconstruct client inputs. Existing attacks on transformers either optimize dummy inputs to match the true client updates, which is costly and unstable for modern models, or exploit the low rank of attention gradients to identify a subspace containing the true layer embeddings, followed by a discrete membership test for candidate tokens. However, this token test is brittle under numerical noise, i.e., from quantization or Differential Privacy (DP), and scales poorly for encoder models with non-causal attention. We introduce TIGER, a continuous gradient inversion attack that turns this subspace signal into a differentiable objective. Instead of searching over tokens or matching full gradients, TIGER directly optimizes token embeddings to minimize their distance to the subspace. Our experiments demonstrate that on encoder-only models, TIGER substantially improves both reconstruction quality and runtime over existing attacks, while on decoder models, TIGER is more robust than prior subspace-based attacks, enabling the first successful reconstructions in DP-defended federated learning settings.