Unsupervised Cycle Detection in Agentic Applications
This addresses a specific observability issue for developers of agentic applications, though it is incremental as it builds on existing structural and semantic analysis techniques.
The paper tackles the problem of detecting hidden execution cycles in agentic applications powered by Large Language Models, which cause resource inefficiencies, and presents an unsupervised framework that achieves an F1 score of 0.72, outperforming baseline methods.
Agentic applications powered by Large Language Models exhibit non-deterministic behaviors that can form hidden execution cycles, silently consuming resources without triggering explicit errors. Traditional observability platforms fail to detect these costly inefficiencies. We present an unsupervised cycle detection framework that combines structural and semantic analysis. Our approach first applies computationally efficient temporal call stack analysis to identify explicit loops and then leverages semantic similarity analysis to uncover subtle cycles characterized by redundant content generation. Evaluated on 1575 trajectories from a LangGraph-based stock market application, our hybrid approach achieves an F1 score of 0.72 (precision: 0.62, recall: 0.86), significantly outperforming individual structural (F1: 0.08) and semantic methods (F1: 0.28). While these results are encouraging, there remains substantial scope for improvement, and future work is needed to refine the approach and address its current limitations.