CLJun 24, 2021Code
Evaluation of Representation Models for Text Classification with AutoML ToolsSebastian Brändle, Marc Hanussek, Matthias Blohm et al.
Automated Machine Learning (AutoML) has gained increasing success on tabular data in recent years. However, processing unstructured data like text is a challenge and not widely supported by open-source AutoML tools. This work compares three manually created text representations and text embeddings automatically created by AutoML tools. Our benchmark includes four popular open-source AutoML tools and eight datasets for text classification purposes. The results show that straightforward text representations perform better than AutoML tools with automatically created text embeddings.
AIFeb 12, 2021
VitrAI -- Applying Explainable AI in the Real WorldMarc Hanussek, Falko Kötter, Maximilien Kintz et al.
With recent progress in the field of Explainable Artificial Intelligence (XAI) and increasing use in practice, the need for an evaluation of different XAI methods and their explanation quality in practical usage scenarios arises. For this purpose, we present VitrAI, which is a web-based service with the goal of uniformly demonstrating four different XAI algorithms in the context of three real life scenarios and evaluating their performance and comprehensibility for humans. This work reveals practical obstacles when adopting XAI methods and gives qualitative estimates on how well different approaches perform in said scenarios.
LGDec 7, 2020
Leveraging Automated Machine Learning for Text Classification: Evaluation of AutoML Tools and Comparison with Human PerformanceMatthias Blohm, Marc Hanussek, Maximilien Kintz
Recently, Automated Machine Learning (AutoML) has registered increasing success with respect to tabular data. However, the question arises whether AutoML can also be applied effectively to text classification tasks. This work compares four AutoML tools on 13 different popular datasets, including Kaggle competitions, and opposes human performance. The results show that the AutoML tools perform better than the machine learning community in 4 out of 13 tasks and that two stand out.
SEDec 2, 2020
A Methodology for Deriving Evaluation Criteria for Software SolutionsHarald Papp, Marc Hanussek
Finding a suited software solution for a company poses a resource-intensive task in an ever-widening market. Software should solve the technical task at hand as perfectly as possible and, at the same time, match the company strategy. Based on these two dimensions, domain knowledge and industry context, we propose a methodology for deriving individually tailored evaluation criteria for software solutions to make them assessable. The approach is formalized as a three-layer model, that ensures the encoding of said dimensions, where each layer holds a more refined and individualized criteria list, starting from a general softwareagnostic catalogue we composed. Finally, we exemplarily demonstrate our method for Machine-Learning-as-a-Service platforms (MaaS) for small and medium-sized enterprises (SME).
LGSep 3, 2020
Can AutoML outperform humans? An evaluation on popular OpenML datasets using AutoML BenchmarkMarc Hanussek, Matthias Blohm, Maximilien Kintz
In the last few years, Automated Machine Learning (AutoML) has gained much attention. With that said, the question arises whether AutoML can outperform results achieved by human data scientists. This paper compares four AutoML frameworks on 12 different popular datasets from OpenML; six of them supervised classification tasks and the other six supervised regression ones. Additionally, we consider a real-life dataset from one of our recent projects. The results show that the automated frameworks perform better or equal than the machine learning community in 7 out of 12 OpenML tasks.