AttentionSmithy: A Modular Framework for Rapid Transformer Development and Customization
This work addresses the problem of framework implementation barriers for domain experts in AI, offering a tool to accelerate research, though it is incremental as it builds on existing transformer concepts.
The paper tackles the complexity of customizing transformer architectures for domain experts by introducing AttentionSmithy, a modular software package that simplifies transformer innovation through reusable building blocks, enabling rapid prototyping and evaluation of transformer variants without extensive coding, with validation showing over 95% accuracy in cell type classification.
Transformer architectures have transformed AI applications but remain complex to customize for domain experts lacking low-level implementation expertise. We introduce AttentionSmithy, a modular software package that simplifies transformer innovation by breaking down key components into reusable building blocks: attention modules, feed-forward networks, normalization layers, and positional encodings. Users can rapidly prototype and evaluate transformer variants without extensive coding. Our framework supports four positional encoding strategies and integrates with neural architecture search for automated design. We validate AttentionSmithy by replicating the original transformer under resource constraints and optimizing translation performance by combining positional encodings. Additionally, we demonstrate its adaptability in gene-specific modeling, achieving over 95% accuracy in cell type classification. These case studies highlight AttentionSmithy's potential to accelerate research across diverse fields by removing framework implementation barriers.