Structural Deep Encoding for Table Question Answering
This work addresses scalability and accuracy issues in table question answering, but it is incremental as it builds on prior methods like sparse attention.
The paper tackled the problem of Transformers losing structural information when processing tabular data by analyzing encoding techniques and introducing novel sparse attention masks, resulting in improved computational efficiency and performance.
Although Transformers-based architectures excel at processing textual information, their naive adaptation for tabular data often involves flattening the table structure. This simplification can lead to the loss of essential inter-dependencies between rows, columns, and cells, while also posing scalability challenges for large tables. To address these issues, prior works have explored special tokens, structured embeddings, and sparse attention patterns. In this paper, we conduct a comprehensive analysis of tabular encoding techniques, which highlights the crucial role of attention sparsity in preserving structural information of tables. We also introduce a set of novel sparse attention mask designs for tabular data, that not only enhance computational efficiency but also preserve structural integrity, leading to better overall performance.