Sumedh Gupte

Sumedh Gupte, Prashanth L. A., Sanjay P. Bhat

We consider the problems of estimation and optimization of utility-based shortfall risk (UBSR). We extend UBSR to cover possibly unbounded random variables. We cover prominent risk measures such as entropic risk, expectile risk, Value-at-Risk, and quadratic risk as special cases of the UBSR. In the context of estimation, we derive non-asymptotic bounds on the mean absolute error (MAE) and the mean-squared error (MSE) of the classical sample-average approximation (SAA) estimator for the UBSR. In the context of optimization, we derive an expression for the gradient of UBSR under a smooth parameterization. We propose a gradient estimator for the UBSR and derive non-asymptotic bounds on MAE and MSE for this estimator. We incorporate the aforementioned gradient estimator into a stochastic gradient (SG) optimization algorithm and derive non-asymptotic bounds on the convergence rate of our SG algorithm for optimizing UBSR under three objectives, namely, strongly convex, convex and non-convex. Finally, we conduct experiments on financial applications to demonstrate the performance of our proposed UBSR estimation and optimization algorithms.

Sumedh Gupte, Prashanth L. A., Sanjay P. Bhat

We consider the problems of estimation and optimization of utility-based shortfall risk (UBSR), which is a popular risk measure in finance. In the context of UBSR estimation, we derive a non-asymptotic bound on the mean-squared error of the classical sample average approximation (SAA) of UBSR. Next, in the context of UBSR optimization, we derive an expression for the UBSR gradient under a smooth parameterization. This expression is a ratio of expectations, both of which involve the UBSR. We use SAA for the numerator as well as denominator in the UBSR gradient expression to arrive at a biased gradient estimator. We derive non-asymptotic bounds on the estimation error, which show that our gradient estimator is asymptotically unbiased. We incorporate the aforementioned gradient estimator into a stochastic gradient (SG) algorithm for UBSR optimization. Finally, we derive non-asymptotic bounds that quantify the rate of convergence of our SG algorithm for UBSR optimization.

Sumedh Gupte, Shrey Rakeshkumar Patel, Soumen Pachal et al.

We propose risk-sensitive reinforcement learning algorithms catering to three families of risk measures, namely expectiles, utility-based shortfall risk and optimized certainty equivalent risk. For each risk measure, in the context of a finite horizon Markov decision process, we first derive a policy gradient theorem. Second, we propose estimators of the risk-sensitive policy gradient for each of the aforementioned risk measures, and establish $\mathcal{O}\left(1/m\right)$ mean-squared error bounds for our estimators, where $m$ is the number of trajectories. Further, under standard assumptions for policy gradient-type algorithms, we establish smoothness of the risk-sensitive objective, in turn leading to stationary convergence rate bounds for the overall risk-sensitive policy gradient algorithm that we propose. Finally, we conduct numerical experiments to validate the theoretical findings on popular RL benchmarks.