The Maximum von Neumann Entropy Principle: Theory and Applications in Machine Learning
This work offers a foundational framework for von Neumann entropy methods in kernel learning, addressing a theoretical gap for researchers in machine learning and quantum information, though it is incremental as it builds on existing maximum entropy principles.
The paper tackles the lack of a principled decision-theoretic and game-theoretic interpretation for maximizing von Neumann entropy in machine learning by extending the minimax formulation of the maximum entropy principle to this setting, providing a robust justification and applying it to kernel representation selection and matrix completion.
Von Neumann entropy (VNE) is a fundamental quantity in quantum information theory and has recently been adopted in machine learning as a spectral measure of diversity for kernel matrices and kernel covariance operators. While maximizing VNE under constraints is well known in quantum settings, a principled analogue of the classical maximum entropy framework, particularly its decision theoretic and game theoretic interpretation, has not been explicitly developed for VNE in data driven contexts. In this paper, we extend the minimax formulation of the maximum entropy principle due to Grünwald and Dawid to the setting of von Neumann entropy, providing a game-theoretic justification for VNE maximization over density matrices and trace-normalized positive semidefinite operators. This perspective yields a robust interpretation of maximum VNE solutions under partial information and clarifies their role as least committed inferences in spectral domains. We then illustrate how the resulting Maximum VNE principle applies to modern machine learning problems by considering two representative applications, selecting a kernel representation from multiple normalized embeddings via kernel-based VNE maximization, and completing kernel matrices from partially observed entries. These examples demonstrate how the proposed framework offers a unifying information-theoretic foundation for VNE-based methods in kernel learning.