CROP: Token-Efficient Reasoning in Large Language Models via Regularized Prompt Optimization
For practitioners deploying LLMs in production, CROP offers a pragmatic solution to reduce latency and cost without significant accuracy loss.
CROP introduces a regularized prompt optimization method that reduces token consumption by 80.6% while maintaining competitive accuracy on complex reasoning tasks (GSM8K, LogiQA, BIG-Bench Hard).
Large Language Models utilizing reasoning techniques improve task performance but incur significant latency and token costs due to verbose generation. Existing automatic prompt optimization(APO) frameworks target task accuracy exclusively at the expense of generating long reasoning traces. We propose Cost-Regularized Optimization of Prompts (CROP), an APO method that introduces regularization on response length by generating textual feedback in addition to standard accuracy feedback. This forces the optimization process to produce prompts that elicit concise responses containing only critical information and reasoning. We evaluate our approach on complex reasoning datasets, specifically GSM8K, LogiQA and BIG-Bench Hard. We achieved an 80.6\% reduction in token consumption while maintaining competitive accuracy, seeing only a nominal decline in performance. This presents a pragmatic solution for deploying token-efficient and cost-effective agentic AI systems in production pipelines.