From Strings to Data Science: a Practical Framework for Automated String Handling
This addresses a practical bottleneck in data science for practitioners dealing with diverse string data, though it appears incremental by building on existing best practices.
The paper tackles the problem of automatically preprocessing categorical string features in tabular datasets, proposing a framework that identifies and encodes them into numerical representations, with an open-source Python implementation showing promising results.
Many machine learning libraries require that string features be converted to a numerical representation for the models to work as intended. Categorical string features can represent a wide variety of data (e.g., zip codes, names, marital status), and are notoriously difficult to preprocess automatically. In this paper, we propose a framework to do so based on best practices, domain knowledge, and novel techniques. It automatically identifies different types of string features, processes them accordingly, and encodes them into numerical representations. We also provide an open source Python implementation to automatically preprocess categorical string data in tabular datasets and demonstrate promising results on a wide range of datasets.