A Generalization Theory based on Independent and Task-Identically Distributed Assumption
This work addresses a foundational problem in machine learning theory by refining generalization bounds to better interpret and guide practical learning tasks, though it is incremental as it builds on existing theories.
The paper tackles the limitation of existing generalization theories that ignore task properties due to the IID assumption, proposing a new ITID assumption to incorporate task properties into data generation and deriving a generalization bound that highlights hypothesis invariance. Experimental results on image classification datasets show the ITID assumption's reasonableness and the theory's effectiveness in improving generalization performance.
Existing generalization theories analyze the generalization performance mainly based on the model complexity and training process. The ignorance of the task properties, which results from the widely used IID assumption, makes these theories fail to interpret many generalization phenomena or guide practical learning tasks. In this paper, we propose a new Independent and Task-Identically Distributed (ITID) assumption, to consider the task properties into the data generating process. The derived generalization bound based on the ITID assumption identifies the significance of hypothesis invariance in guaranteeing generalization performance. Based on the new bound, we introduce a practical invariance enhancement algorithm from the perspective of modifying data distributions. Finally, we verify the algorithm and theorems in the context of image classification task on both toy and real-world datasets. The experimental results demonstrate the reasonableness of the ITID assumption and the effectiveness of new generalization theory in improving practical generalization performance.