Some observations concerning Off Training Set (OTS) error
This work critiques the applicability of a theoretical result in generalization error, highlighting its limited relevance for real-world machine learning problems.
The paper argues that a theorem about Off Training Set (OTS) error, which states small training error does not guarantee small OTS error, only applies when training and test data distributions have no overlap, a scenario deemed largely irrelevant to practical machine learning.
A form of generalisation error known as Off Training Set (OTS) error was recently introduced in [Wolpert, 1996b], along with a theorem showing that small training set error does not guarantee small OTS error, unless assumptions are made about the target function. Here it is shown that the applicability of this theorem is limited to models in which the distribution generating training data has no overlap with the distribution generating test data. It is argued that such a scenario is of limited relevance to machine learning.