A transformer architecture alteration to incentivise externalised reasoning
This addresses efficiency issues in LLMs for AI researchers, but it appears incremental as it builds on existing early-exit methods.
The paper tackles the problem of making LLMs more efficient reasoners by introducing an early-exit mechanism in transformers to reduce unnecessary computations, with preliminary results showing adaptive computation reduction across tokens.
We propose a new architectural change, and post-training pipeline, for making LLMs more verbose reasoners by teaching a model to truncate forward passes early. We augment an existing transformer architecture with an early-exit mechanism at intermediate layers and train the model to exit at shallower layers when the next token can be predicted without deep computation. After a calibration stage, we incentivise the model to exit as early as possible while maintaining task performance using reinforcement learning. We provide preliminary results to this effect for small reasoning models, showing that they learn to adaptively reduce computations across tokens. We predict that, applied at the right scale, our approach can minimise the amount of excess computation that reasoning models have at their disposal to perform non-myopic planning using their internal activations, reserving this only for difficult-to-predict tokens.