Spyridon Pougkakiotis

Jane H. Lee, Baturay Saglam, Spyridon Pougkakiotis et al.

Constrained optimization provides a common framework for dealing with conflicting objectives in reinforcement learning (RL). In most of these settings, the objectives (and constraints) are expressed though the expected accumulated reward. However, this formulation neglects risky or even possibly catastrophic events at the tails of the reward distribution, and is often insufficient for high-stakes applications in which the risk involved in outliers is critical. In this work, we propose a framework for risk-aware constrained RL, which exhibits per-stage robustness properties jointly in reward values and time using optimized certainty equivalents (OCEs). Our framework ensures an exact equivalent to the original constrained problem within a parameterized strong Lagrangian duality framework under appropriate constraint qualifications, and yields a simple algorithmic recipe which can be wrapped around standard RL solvers, such as PPO. Lastly, we establish the convergence of the proposed algorithm under common assumptions, and verify the risk-aware properties of our approach through several numerical experiments.

Hassaan Hashmi, Spyridon Pougkakiotis, Dionysis Kalogerias

We consider the problem of maximizing weighted sum rate in a multiple-input single-output (MISO) downlink wireless network with emphasis on user rate reliability. We introduce a novel risk-aggregated formulation of the complex WSR maximization problem, which utilizes the Conditional Value-at-Risk (CVaR) as a functional for enforcing rate (ultra)-reliability over channel fading uncertainty/risk. We establish a WMMSE-like equivalence between the proposed precoding problem and a weighted risk-averse MSE problem, enabling us to design a tailored unfolded graph neural network (GNN) policy function approximation (PFA), named α-Robust Graph Neural Network (αRGNN), trained to maximize lower-tail (CVaR) rates resulting from adverse wireless channel realizations (e.g., deep fading, attenuation). We empirically demonstrate that a trained αRGNN fully eliminates per user deep rate fades, and substantially and optimally reduces statistical user rate variability while retaining adequate ergodic performance.

Pantelis Bouboulis, Spyridon Pougkakiotis, Sergios Theodoridis

We present a new framework for online Least Squares algorithms for nonlinear modeling in RKH spaces (RKHS). Instead of implicitly mapping the data to a RKHS (e.g., kernel trick), we map the data to a finite dimensional Euclidean space, using random features of the kernel's Fourier transform. The advantage is that, the inner product of the mapped data approximates the kernel function. The resulting "linear" algorithm does not require any form of sparsification, since, in contrast to all existing algorithms, the solution's size remains fixed and does not increase with the iteration steps. As a result, the obtained algorithms are computationally significantly more efficient compared to previously derived variants, while, at the same time, they converge at similar speeds and to similar error floors.