Banyan: Improved Representation Learning with Explicit Structure
This provides a viable alternative for under-represented languages and resource-constrained NLP applications, though it is incremental in improving structured models.
The paper tackles the problem of efficient representation learning in low-resource settings by introducing Banyan, a model that uses explicit hierarchical structure to outperform larger transformer models with only 14 non-embedding parameters.
We present Banyan, a model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers excel at scale, they struggle in low-resource settings. Conversely recent structured models have shown promise as efficient learners, but lack performance. Banyan bridges this gap with two key innovations: an entangled hierarchical tree structure and diagonalized message passing, enabling it to outperform larger transformer models with just 14 non-embedding parameters. It excels in low-resource settings, offering a viable alternative for under-represented languages and highlighting its potential for efficient, interpretable NLP in resource-constrained environments.