Statistical learning theory and Occam's razor: The core argument
It clarifies a foundational principle in machine learning theory, but is incremental as it refines existing arguments without new empirical results.
The paper distills the core argument from statistical learning theory that simpler hypothesis classes are better due to improved learning guarantees, but notes these guarantees are model-relative and constrained by prior knowledge.
Statistical learning theory is often associated with the principle of Occam's razor, which recommends a simplicity preference in inductive inference. This paper distills the core argument for simplicity obtainable from statistical learning theory, built on the theory's central learning guarantee for the method of empirical risk minimization. This core "means-ends" argument is that a simpler hypothesis class or inductive model is better because it has better learning guarantees; however, these guarantees are model-relative and so the theoretical push towards simplicity is checked by our prior knowledge.