The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?
This work addresses safety and reliability issues in vision-language models for users, but it is incremental as it builds on existing linear probing techniques.
The study tackled the problem of hallucinated or harmful content in large vision-language models by analyzing token distributions, finding that the first token's logits contain hidden knowledge for tasks like recognizing unanswerable questions and defending against attacks, with experiments showing performance improvements on tasks such as uncertainty indication and hallucination mitigation.
Large vision-language models (LVLMs), designed to interpret and respond to human instructions, occasionally generate hallucinated or harmful content due to inappropriate instructions. This study uses linear probing to shed light on the hidden knowledge at the output layers of LVLMs. We demonstrate that the logit distributions of the first tokens contain sufficient information to determine whether to respond to the instructions, including recognizing unanswerable visual questions, defending against jailbreaking attacks, and identifying deceptive questions. Such hidden knowledge is gradually lost in logits of subsequent tokens during response generation. Then, we illustrate a simple decoding strategy at the generation of the first token, effectively improving the generated content. In experiments, we find a few interesting insights: First, the CLIP model already contains a strong signal for solving these tasks, which indicates potential bias in the existing datasets. Second, we observe performance improvement by utilizing the first logit distributions on three additional tasks, including indicating uncertainty in math solving, mitigating hallucination, and image classification. Last, with the same training data, simply finetuning LVLMs improves models' performance but is still inferior to linear probing on these tasks.