Loop as a Bridge: Can Looped Transformers Truly Link Representation Space and Natural Language Outputs?
This addresses a fundamental problem in AI for improving model interpretability and output quality, but the findings are incremental as they highlight limitations rather than a breakthrough.
The paper tackles the gap between internal knowledge and linguistic outputs in Large Language Models by empirically investigating if Looped Transformers can bridge it through iterative introspection, finding that while increased loops narrow the gap, it partly degrades internal knowledge and perception does not improve across loops.
Large Language Models (LLMs) often exhibit a gap between their internal knowledge and their explicit linguistic outputs. In this report, we empirically investigate whether Looped Transformers (LTs)--architectures that increase computational depth by iterating shared layers--can bridge this gap by utilizing their iterative nature as a form of introspection. Our experiments reveal that while increasing loop iterations narrows the gap, it is partly driven by a degradation of their internal knowledge carried by representations. Moreover, another empirical analysis suggests that current LTs' ability to perceive representations does not improve across loops; it is only present in the final loop. These results suggest that while LTs offer a promising direction for scaling computational depth, they have yet to achieve the introspection required to truly link representation space and natural language.