Langformers: Unified NLP Pipelines for Language Models
This addresses usability issues for non-programmers, beginners, and experienced developers in NLP, though it is incremental as it builds on existing tools and frameworks.
The paper tackles the complexity of using transformer-based language models in NLP by introducing Langformers, an open-source Python library that unifies pipelines for tasks like conversational AI and text classification, resulting in streamlined workflows through a factory-based interface.
Transformer-based language models have revolutionized the field of natural language processing (NLP). However, using these models often involves navigating multiple frameworks and tools, as well as writing repetitive boilerplate code. This complexity can discourage non-programmers and beginners, and even slow down prototyping for experienced developers. To address these challenges, we introduce Langformers, an open-source Python library designed to streamline NLP pipelines through a unified, factory-based interface for large language model (LLM) and masked language model (MLM) tasks. Langformers integrates conversational AI, MLM pretraining, text classification, sentence embedding/reranking, data labelling, semantic search, and knowledge distillation into a cohesive API, supporting popular platforms such as Hugging Face and Ollama. Key innovations include: (1) task-specific factories that abstract training, inference, and deployment complexities; (2) built-in memory and streaming for conversational agents; and (3) lightweight, modular design that prioritizes ease of use. Documentation: https://langformers.com