On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case
This work addresses the need for faster and more efficient models in high-energy physics, where frequent retraining by small groups with limited hardware is common, though it is incremental in applying existing techniques to a specific domain.
The study tested advanced deep-learning techniques on the Higgs ML dataset to improve classification performance and training speed, achieving equal performance to the original Kaggle challenge winner while significantly reducing training and application time.
Beginning from a basic neural-network architecture, we test the potential benefits offered by a range of advanced techniques for machine learning, in particular deep learning, in the context of a typical classification problem encountered in the domain of high-energy physics, using a well-studied dataset: the 2014 Higgs ML Kaggle dataset. The advantages are evaluated in terms of both performance metrics and the time required to train and apply the resulting models. Techniques examined include domain-specific data-augmentation, learning rate and momentum scheduling, (advanced) ensembling in both model-space and weight-space, and alternative architectures and connection methods. Following the investigation, we arrive at a model which achieves equal performance to the winning solution of the original Kaggle challenge, whilst being significantly quicker to train and apply, and being suitable for use with both GPU and CPU hardware setups. These reductions in timing and hardware requirements potentially allow the use of more powerful algorithms in HEP analyses, where models must be retrained frequently, sometimes at short notice, by small groups of researchers with limited hardware resources. Additionally, a new wrapper library for PyTorch called LUMIN is presented, which incorporates all of the techniques studied.