Exploring Large Action Sets with Hyperspherical Embeddings using von Mises-Fisher Sampling
This provides a scalable solution for reinforcement learning applications with large action sets, such as recommender systems, though it is incremental as it builds on existing exploration methods.
The paper tackles the problem of exploring large action sets in reinforcement learning by introducing von Mises-Fisher exploration (vMF-exp), a scalable method that uses hyperspherical embeddings and sampling to avoid the computational bottlenecks of alternatives like Boltzmann Exploration, with validation on simulated data, real-world datasets, and deployment in a global music streaming service's recommender system.
This paper introduces von Mises-Fisher exploration (vMF-exp), a scalable method for exploring large action sets in reinforcement learning problems where hyperspherical embedding vectors represent these actions. vMF-exp involves initially sampling a state embedding representation using a von Mises-Fisher distribution, then exploring this representation's nearest neighbors, which scales to virtually unlimited numbers of candidate actions. We show that, under theoretical assumptions, vMF-exp asymptotically maintains the same probability of exploring each action as Boltzmann Exploration (B-exp), a popular alternative that, nonetheless, suffers from scalability issues as it requires computing softmax values for each action. Consequently, vMF-exp serves as a scalable alternative to B-exp for exploring large action sets with hyperspherical embeddings. Experiments on simulated data, real-world public data, and the successful large-scale deployment of vMF-exp on the recommender system of a global music streaming service empirically validate the key properties of the proposed method.