Agentic System as Compressor: Quantifying System Intelligence in Bits
For AI researchers evaluating agentic systems, this work offers a principled, quantitative measure of system intelligence based on compression, though the experiments are small-scale and the approach is incremental.
The paper proposes a method to quantify the intelligence of agentic systems by measuring how many bits are needed to reconstruct a target object under a fixed task distribution, interface, and compute budget, using compression as a proxy. In five controlled settings (reversed text, chess moves, protein sequences, retrieval-augmented QA, and semantic story compression), agentic components consistently reduce codelength.
Large language models are turning from isolated predictors into agentic systems: they call tools, retrieve evidence, obey environment constraints, use verifiers, and complete tasks through search and multi-turn interaction. We adopts an analytical viewpoint based on "compression is intelligence": under a fixed task distribution, interface, and compute budget, a stronger agentic system lets a target object be reconstructed with fewer bits. We operationalize the measure with arithmetic coding, seed coding, and a fallback, and evaluate it in five settings: reversed text, chess moves, protein sequences, retrieval-augmented question answering, and semantic story compression; in all of them agentic components reduce codelength. These small, controlled experiments cover component types typical of real agentic systems, show that codelength can analyze how components, observers, and budgets change residual uncertainty, and offer guidance for evaluating real agent systems.