Adaptive Integrated Layered Attention (AILA)
This work addresses efficiency and performance challenges in deep learning for practitioners across domains like finance, computer vision, and NLP, though it is incremental as it extends existing architectures.
The paper tackles the problem of improving neural network efficiency by proposing Adaptive Integrated Layered Attention (AILA), which combines dense skip connections with adaptive feature reuse across layers, achieving performance matching strong baselines like LSTMs, Transformers, and ResNets at a fraction of the training and inference time on tasks such as price forecasting, image recognition, and sentiment analysis.
We propose Adaptive Integrated Layered Attention (AILA), a neural network architecture that combines dense skip connections with different mechanisms for adaptive feature reuse across network layers. We evaluate AILA on three challenging tasks: price forecasting for various commodities and indices (S&P 500, Gold, US dollar Futures, Coffee, Wheat), image recognition using the CIFAR-10 dataset, and sentiment analysis on the IMDB movie review dataset. In all cases, AILA matches strong deep learning baselines (LSTMs, Transformers, and ResNets), achieving it at a fraction of the training and inference time. Notably, we implement and test two versions of the model - AILA-Architecture 1, which uses simple linear layers as the connection mechanism between layers, and AILA-Architecture 2, which implements an attention mechanism to selectively focus on outputs from previous layers. Both architectures are applied in a single-task learning setting, with each model trained separately for individual tasks. Results confirm that AILA's adaptive inter-layer connections yield robust gains by flexibly reusing pertinent features at multiple network depths. The AILA approach thus presents an extension to existing architectures, improving long-range sequence modeling, image recognition with optimised computational speed, and SOTA classification performance in practice.