Semantic Fusion with Fuzzy-Membership Features for Controllable Language Modelling
This work addresses the need for interpretable and controllable text generation in natural language processing, though it is incremental as it builds on existing Transformer models with lightweight modifications.
The paper tackles the problem of enabling controllable language generation by augmenting a Transformer language model with a parallel semantic feature channel using fuzzy-membership functions, resulting in improved perplexity and precise user-controllable generation of polarity and punctuation on a synthetic corpus.
We propose semantic fusion, a lightweight scheme that augments a Transformer language model (LM) with a parallel, fuzzy-membership feature channel that encodes token-level semantics. Each token is represented by a vector of interpretable features (e.g. part-of-speech cues, shallow roles, boundary flags, sentiment polarity and strength) whose values are graded degrees from differentiable membership functions (e.g. power kernels). These per-token vectors form a sentence-level semantic matrix fused via a gated adapter into the LM. Training uses standard next-token prediction, an auxiliary loss that reconstructs the semantic features from hidden states, and a lightweight uniformizer that regularizes adjective-class distributions. On a synthetic two-clause corpus with held-out adjectives for out-of-distribution (OOD) control, semantic fusion improves perplexity and enables precise, user-controllable generation of polarity and punctuation while maintaining model simplicity. This approach adds only small overhead, remains fully compatible with tied input-output embeddings, and provides an interpretable pathway for conditioned natural language generation.