J6: Jacobian-Driven Role Attribution for Multi-Objective Prompt Optimization in LLMs
This work addresses a fundamental problem in LLM adaptation for researchers and practitioners by providing a novel, interpretable framework for multi-objective optimization, though it is incremental in building on existing gradient-based methods.
The paper tackles the challenge of balancing multiple objectives like factuality and confidence in LLM prompt optimization by proposing J6, a Jacobian-based method that decomposes gradient interactions into six components, enabling dynamic updates and achieving improved performance with interpretable insights.
In large language model (LLM) adaptation, balancing multiple optimization objectives such as improving factuality (heat) and increasing confidence (via low entropy) poses a fundamental challenge, especially when prompt parameters (e.g., hidden-layer insertions h and embedding modifications w) interact in non-trivial ways. Existing multi-objective optimization strategies often rely on scalar gradient aggregation, ignoring the deeper geometric structure between objectives and parameters. We propose J6, a structured Jacobian-based method that decomposes the gradient interaction matrix into six interpretable components. This decomposition enables both hard decision-making (e.g., choosing the dominant update direction via argmax) and soft strategies (e.g., attention-style weighting via softmax over J6), forming a dynamic update framework that adapts to local conflict and synergy. Moreover, the interpretable structure of J6 provides insight into parameter attribution, task interference, and geometry-aligned adaptation. Our work introduces a principled and extensible mechanism for conflict-aware prompt optimization, and opens a new avenue for incorporating structured Jacobian reasoning into multi-objective neural tuning.