CodeSSM: Towards State Space Models for Code Understanding
It addresses the problem of high memory usage and context limitations in code understanding for developers and researchers, offering a viable alternative to transformers, though it is incremental as it applies an existing model type to a new domain.
This paper tackles the limitations of transformers in code understanding tasks by introducing CodeSSM, the first State Space Model trained on code corpora, showing that SSMs are more sample-efficient, extrapolate to longer contexts, and reduce memory usage by up to 64% compared to transformers at a context length of 2048.
Although transformers dominate many code-specific tasks, they have significant limitations. This paper explores State Space Models (SSMs) as a promising alternative for code understanding tasks such as retrieval, classification, and clone detection. We introduce CodeSSM, the first SSM-based model trained on code corpora to assess its effectiveness. Our results demonstrate that SSMs are more sample-efficient and can extrapolate to longer contexts beyond the pretraining length. Extensive experiments show that SSMs offer a viable alternative to transformers, addressing several their limitations. Additionally, CodeSSM reduces memory usage by up to 64\% compared to transformers at a context length of 2048, with greater savings as context length grows.