Incorporating Exponential Smoothing into MLP: A Simple but Effective Sequence Model
This work provides insights into sequence modeling for researchers, but it is incremental as it builds on existing state space models with a simplified approach.
The paper tackled the problem of modeling long-range dependencies in sequences by investigating whether the success of complex models like S4 stems from their intricate design or simpler state space models, and found that incorporating exponential smoothing into an MLP with minimal parameter increase achieves comparable results to S4 on the LRA benchmark.
Modeling long-range dependencies in sequential data is a crucial step in sequence learning. A recently developed model, the Structured State Space (S4), demonstrated significant effectiveness in modeling long-range sequences. However, It is unclear whether the success of S4 can be attributed to its intricate parameterization and HiPPO initialization or simply due to State Space Models (SSMs). To further investigate the potential of the deep SSMs, we start with exponential smoothing (ETS), a simple SSM, and propose a stacked architecture by directly incorporating it into an element-wise MLP. We augment simple ETS with additional parameters and complex field to reduce the inductive bias. Despite increasing less than 1\% of parameters of element-wise MLP, our models achieve comparable results to S4 on the LRA benchmark.