Incomplete Prompt Jailbreaks in Large Language Models
For LLM safety researchers, this work identifies a novel vulnerability (IPJ) and proposes a neuron-level defense mechanism, though the findings are incremental as they build on existing jailbreak and interpretability research.
The paper formalizes incomplete prompt jailbreaks (IPJ) in LLMs, showing that incomplete harmful prompts can elicit harmful continuations. It demonstrates that training-based defenses fail to generalize, but neuron-level interventions targeting termination and continuation neurons offer more robust protection.
Large language models (LLMs) are increasingly released as open-weight models with safeguards against harmful requests. Nevertheless, sentence completion remains vulnerable to incomplete harmful prompts. In this work, we formalize this phenomenon as incomplete prompt jailbreaks (IPJ) and provide a systematic empirical characterization of when and how incomplete prompts elicit harmful continuations. We analyze diverse attractor types associated with incomplete sentence continuation and show that LLMs systematically delay refusal until sentence termination. We further demonstrate that training models to refuse incomplete harmful prompts via parameter tuning is insufficient, failing to generalize across both content domains and attractor types. To enable fine-grained control, we identify two functional neurons: termination and continuation neurons. By clarifying their roles in sentence completion, we highlight the potential of neuron-level interventions for more precise and robust IPJ defenses.