Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?
This addresses the challenge of scalable constraint handling in LLMs for practical applications, though it is incremental as it builds on existing constraint generation research.
The paper tackles the problem of whether Large Language Models (LLMs) can parse large, fine-grained constraint lists, introducing the Large-Scale Constraint Generation (LSCG) problem and evaluating it with a Words Checker instance. The result shows that existing solutions suffer significant performance drops with more constraints, while the proposed FoCusNet model boosts accuracy by 8-13%.
Recent research has explored the constrained generation capabilities of Large Language Models (LLMs) when explicitly prompted by few task-specific requirements. In contrast, we introduce Large-Scale Constraint Generation (LSCG), a new problem that evaluates whether LLMs can parse a large, fine-grained, generic list of constraints. To examine the LLMs' ability to handle an increasing number constraints, we create a practical instance of LSCG, called Words Checker. In Words Checker, we evaluate the impact of model characteristics (e.g., size, family) and steering techniques (e.g., Simple Prompt, Chain of Thought, Best of N) on performance. We also propose FoCusNet, a small and dedicated model that parses the original list of constraints into a smaller subset, helping the LLM focus on relevant constraints. Experiments reveal that existing solutions suffer a significant performance drop as the number of constraints increases, with FoCusNet showing an 8-13% accuracy boost.