Gated Tree Cross-attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs
This provides a practical checkpoint-compatible route to more syntax-robust decoder-only LLMs, addressing reliability issues for downstream reasoning applications.
The paper tackled the problem of decoder-only large language models being brittle to minor grammatical perturbations by introducing a checkpoint-compatible gated tree cross-attention branch that reads precomputed constituency chunk memory, resulting in strengthened syntactic robustness without compromising performance on tasks like Multiple-Choice QA and commonsense reasoning.
Decoder-only large language models achieve strong broad performance but are brittle to minor grammatical perturbations, undermining reliability for downstream reasoning. However, directly injecting explicit syntactic structure into an existing checkpoint can interfere with its pretrained competence. We introduce a checkpoint-compatible gated tree cross-attention (GTCA) branch that reads precomputed constituency chunk memory while leaving backbone architecture unchanged. Our design uses a token update mask and staged training to control the scope and timing of structural updates. Across benchmarks and Transformer backbones, GTCA strengthens syntactic robustness beyond continued-training baselines without compromising Multiple-Choice QA performance or commonsense reasoning, providing a practical checkpoint-compatible route to more syntax-robust decoder-only LLMs.