Exploring Large Language Models for Feature Selection: A Data-centric Perspective
This work addresses feature selection problems for data scientists and ML practitioners by leveraging LLMs, but it is incremental as it builds on existing LLM capabilities without introducing a fundamentally new paradigm.
The paper explored using Large Language Models (LLMs) for feature selection, categorizing methods into data-driven and text-based approaches, and found that text-based methods were effective and robust in classification and regression tasks, with potential demonstrated in a real-world medical application.
The rapid advancement of Large Language Models (LLMs) has significantly influenced various domains, leveraging their exceptional few-shot and zero-shot learning capabilities. In this work, we aim to explore and understand the LLMs-based feature selection methods from a data-centric perspective. We begin by categorizing existing feature selection methods with LLMs into two groups: data-driven feature selection which requires numerical values of samples to do statistical inference and text-based feature selection which utilizes prior knowledge of LLMs to do semantical associations using descriptive context. We conduct experiments in both classification and regression tasks with LLMs in various sizes (e.g., GPT-4, ChatGPT and LLaMA-2). Our findings emphasize the effectiveness and robustness of text-based feature selection methods and showcase their potentials using a real-world medical application. We also discuss the challenges and future opportunities in employing LLMs for feature selection, offering insights for further research and development in this emerging field.