Sobhan Mohammadpour

Sobhan Mohammadpour, Emma Frejinger

We address the problem of simultaneously estimating arc travel times in a network \emph{and} parameters of route choice models for strategic and tactical network planning purposes. Hitherto, these interdependent tasks have been approached separately in the literature on road traffic networks. We illustrate that ignoring this interdependence can lead to erroneous route choice model parameter estimates. We propose a method for maximum likelihood estimation to solve the simultaneous estimation problem that is applicable to any differentiable route choice model. Moreover, our approach allows to naturally mix observations at varying levels of granularity, including noisy or partial path data. Numerical results based on real taxi data from New York City show strong performance of our method, even in comparison to a benchmark method focused solely on arc travel time estimation.

Sobhan Mohammadpour, Emmanuel Bengio, Emma Frejinger et al.

Generative Flow Networks (GFNs) have emerged as a powerful tool for sampling discrete objects from unnormalized distributions, offering a scalable alternative to Markov Chain Monte Carlo (MCMC) methods. While GFNs draw inspiration from maximum entropy reinforcement learning (RL), the connection between the two has largely been unclear and seemingly applicable only in specific cases. This paper addresses the connection by constructing an appropriate reward function, thereby establishing an exact relationship between GFNs and maximum entropy RL. This construction allows us to introduce maximum entropy GFNs, which, in contrast to GFNs with uniform backward policy, achieve the maximum entropy attainable by GFNs without constraints on the state space.

Sobhan Mohammadpour, Emma Frejinger, Pierre-Luc Bacon

Regularized reinforcement learning (RL), particularly the entropy-regularized kind, has gained traction in optimal control and inverse RL. While standard unregularized RL methods remain unaffected by changes in the number of actions, we show that it can severely impact their regularized counterparts. This paper demonstrates the importance of decoupling the regularizer from the action space: that is, to maintain a consistent level of regularization regardless of how many actions are involved to avoid over-regularization. Whereas the problem can be avoided by introducing a task-specific temperature parameter, it is often undesirable and cannot solve the problem when action spaces are state-dependent. In the state-dependent action context, different states with varying action spaces are regularized inconsistently. We introduce two solutions: a static temperature selection approach and a dynamic counterpart, universally applicable where this problem arises. Implementing these changes improves performance on the DeepMind control suite in static and dynamic temperature regimes and a biological sequence design task.