Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMs
This addresses token efficiency for users of multimodal LLMs, though it is incremental as it applies an existing visual input capability to a new compression task.
The paper tackled the problem of reducing token usage in multimodal LLMs by compressing textual inputs as images, showing that this method yields substantial token savings (often nearly half) without degrading performance on benchmarks like RULER and CNN/DailyMail.
Large language models (LLMs) and their multimodal variants can now process visual inputs, including images of text. This raises an intriguing question: can we compress textual inputs by feeding them as images to reduce token usage while preserving performance? In this paper, we show that visual text representations are a practical and surprisingly effective form of input compression for decoder LLMs. We exploit the idea of rendering long text inputs as a single image and provide it directly to the model. This leads to dramatically reduced number of decoder tokens required, offering a new form of input compression. Through experiments on two distinct benchmarks RULER (long-context retrieval) and CNN/DailyMail (document summarization) we demonstrate that this text-as-image method yields substantial token savings (often nearly half) without degrading task performance.