Training Efficiency and Robustness in Deep Learning
This work addresses training speed and robustness issues for deep learning practitioners, but it appears incremental as it builds on existing techniques like hard negative mining and focuses on specific cases like linear models.
The thesis tackled the inefficiency and vulnerability of deep learning models by introducing methods like hard negative mining and gradient clustering to speed up training and improve generalization, and proved that maximal adversarial robustness can be achieved in linear models through optimizer, regularization, or architecture choices without extra data.
Deep Learning has revolutionized machine learning and artificial intelligence, achieving superhuman performance in several standard benchmarks. It is well-known that deep learning models are inefficient to train; they learn by processing millions of training data multiple times and require powerful computational resources to process large batches of data in parallel at the same time rather than sequentially. Deep learning models also have unexpected failure modes; they can be fooled into misbehaviour, producing unexpectedly incorrect predictions. In this thesis, we study approaches to improve the training efficiency and robustness of deep learning models. In the context of learning visual-semantic embeddings, we find that prioritizing learning on more informative training data increases convergence speed and improves generalization performance on test data. We formalize a simple trick called hard negative mining as a modification to the learning objective function with no computational overhead. Next, we seek improvements to optimization speed in general-purpose optimization methods in deep learning. We show that a redundancy-aware modification to the sampling of training data improves the training speed and develops an efficient method for detecting the diversity of training signal, namely, gradient clustering. Finally, we study adversarial robustness in deep learning and approaches to achieve maximal adversarial robustness without training with additional data. For linear models, we prove guaranteed maximal robustness achieved only by appropriate choice of the optimizer, regularization, or architecture.