What do your logits know? (The answer may surprise you!)
For model owners and users, this reveals that easily accessible logits can unintentionally leak sensitive information, highlighting a new privacy vulnerability in deployed models.
The paper investigates information leakage in vision-language models, comparing how much task-irrelevant information is retained in low-dimensional projections of the residual stream versus top-k logits. It finds that top-k logits can leak as much information as the full residual stream, posing a privacy risk.
Recent work has shown that probing model internals can reveal a wealth of information not apparent from the model generations. This poses the risk of unintentional or malicious information leakage, where model users are able to learn information that the model owner assumed was inaccessible. Using vision-language models as a testbed, we present the first systematic comparison of information retained at different "representational levels'' as it is compressed from the rich information encoded in the residual stream through two natural bottlenecks: low-dimensional projections of the residual stream obtained using tuned lens, and the final top-k logits most likely to impact model's answer. We show that even easily accessible bottlenecks defined by the model's top logit values can leak task-irrelevant information present in an image-based query, in some cases revealing as much information as direct projections of the full residual stream.