Optimal information deletion and Bayes' theorem
Provides a theoretical foundation for information deletion in Bayesian inference, relevant to statisticians and machine learning researchers.
The paper revisits Zellner's optimal information processing rule from the perspective of information deletion, proving that the optimal rule for updating a posterior when data is removed is the leave-data-out posterior from Bayes' theorem.
Arnold Zellner published a seminal paper on Bayes' theorem as an optimal information processing rule, a result that led to the variational formulation of Bayes' theorem, and a central idea in generalized variational inference. Almost 40 years later, we revisit these ideas, but from the perspective of information deletion. We investigate rules that update a posterior distribution into an antedata distribution when a portion of data is removed. In such context, a rule that does not destroy or create nonexistent information is called the optimal information deletion rule and we prove that it coincides with the leave-data-out posterior from Bayes' theorem.