TableLoRA: Low-rank Adaptation on Table Structure Understanding for Large Language Models
This addresses the challenge of processing two-dimensional table data in one-dimensional sequences for LLMs in low-parameter settings, offering a domain-specific incremental improvement.
The paper tackled the problem of improving large language models' understanding of tabular data under parameter-efficient fine-tuning by proposing TableLoRA, which uses special tokens and 2D LoRA to encode table structure, resulting in consistent outperformance over vanilla LoRA and other methods on four datasets.
Tabular data are crucial in many fields and their understanding by large language models (LLMs) under high parameter efficiency paradigm is important. However, directly applying parameter-efficient fine-tuning (PEFT) techniques to tabular tasks presents significant challenges, particularly in terms of better table serialization and the representation of two-dimensional structured information within a one-dimensional sequence. To address this, we propose TableLoRA, a module designed to improve LLMs' understanding of table structure during PEFT. It incorporates special tokens for serializing tables with special token encoder and uses 2D LoRA to encode low-rank information on cell positions. Experiments on four tabular-related datasets demonstrate that TableLoRA consistently outperforms vanilla LoRA and surpasses various table encoding methods tested in control experiments. These findings reveal that TableLoRA, as a table-specific LoRA, enhances the ability of LLMs to process tabular data effectively, especially in low-parameter settings, demonstrating its potential as a robust solution for handling table-related tasks.