A Comparative Study of Feature Selection in Tsetlin Machines
This work addresses the problem of feature selection for Tsetlin machines, providing a comprehensive baseline for researchers and practitioners, though it is incremental as it adapts existing methods to a new model.
The paper tackled the lack of established feature selection tools for Tsetlin machines by adapting and evaluating various techniques, including novel TM-internal scorers, across 12 datasets, showing that these scorers perform competitively and reveal interacting feature patterns while being computationally efficient.
Feature Selection (FS) is crucial for improving model interpretability, reducing complexity, and sometimes for enhancing accuracy. The recently introduced Tsetlin machine (TM) offers interpretable clause-based learning, but lacks established tools for estimating feature importance. In this paper, we adapt and evaluate a range of FS techniques for TMs, including classical filter and embedded methods as well as post-hoc explanation methods originally developed for neural networks (e.g., SHAP and LIME) and a novel family of embedded scorers derived from TM clause weights and Tsetlin automaton (TA) states. We benchmark all methods across 12 datasets, using evaluation protocols, like Remove and Retrain (ROAR) strategy and Remove and Debias (ROAD), to assess causal impact. Our results show that TM-internal scorers not only perform competitively but also exploit the interpretability of clauses to reveal interacting feature patterns. Simpler TM-specific scorers achieve similar accuracy retention at a fraction of the computational cost. This study establishes the first comprehensive baseline for FS in TM and paves the way for developing specialized TM-specific interpretability techniques.