Revealing the Underlying Patterns: Investigating Dataset Similarity, Performance, and Generalization
This work addresses the need for efficient model selection and generalization in dynamic environments, though it appears incremental as it builds on existing distance-based analysis methods.
The paper tackles the problem of poor generalization in supervised deep learning models by proposing a distance metric to analyze dataset similarity and model behavior, showing that adding only 1, 3, or 7 unseen images to the training set can improve generalization and reduce costs.
Supervised deep learning models require significant amount of labeled data to achieve an acceptable performance on a specific task. However, when tested on unseen data, the models may not perform well. Therefore, the models need to be trained with additional and varying labeled data to improve the generalization. In this work, our goal is to understand the models, their performance and generalization. We establish image-image, dataset-dataset, and image-dataset distances to gain insights into the model's behavior. Our proposed distance metric when combined with model performance can help in selecting an appropriate model/architecture from a pool of candidate architectures. We have shown that the generalization of these models can be improved by only adding a small number of unseen images (say 1, 3 or 7) into the training set. Our proposed approach reduces training and annotation costs while providing an estimate of model performance on unseen data in dynamic environments.