Xiao Liu

Shimao Zhang, Xiao Liu, Xin Zhang et al.

Large Language Models have demonstrated outstanding performance across various downstream tasks and have been widely applied in multiple scenarios. Human-annotated preference data is used for training to further improve LLMs' performance, which is constrained by the upper limit of human performance. Therefore, Self-Rewarding method has been proposed, where LLMs generate training data by rewarding their own outputs. However, the existing self-rewarding paradigm is not effective in mathematical reasoning scenarios and may even lead to a decline in performance. In this work, we propose the Process-based Self-Rewarding pipeline for language models, which introduces long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization within the self-rewarding paradigm. Our new paradigm successfully enhances the performance of LLMs on multiple mathematical reasoning benchmarks through iterative Process-based Self-Rewarding, demonstrating the immense potential of self-rewarding to achieve LLM reasoning that may surpass human capabilities.

Xiao Liu, Jingyi Feng, Xinchao Liu

Projection-based model reduction is among the most widely adopted methods for constructing parametric Reduced-Order Models (ROM). Utilizing the snapshot data from solving full-order governing equations, the Proper Orthogonal Decomposition (POD) computes the optimal basis modes that represent the data, and a ROM can be constructed in the low-dimensional vector subspace spanned by the POD basis. For parametric governing equations, a potential challenge arises when there is a need to update the POD basis to adapt ROM that accurately capture the variation of a system's behavior over its parameter space (in design, control, uncertainty quantification, digital twins applications, etc.). In this paper, we propose a Projected Gaussian Process (pGP) and formulate the problem of adapting the POD basis as a supervised statistical learning problem, for which the goal is to learn a mapping from the parameter space to the Grassmann manifold that contains the optimal subspaces. A mapping is firstly established between the Euclidean space and the horizontal space of an orthogonal matrix that spans a reference subspace in the Grassmann manifold. A second mapping from the horizontal space to the Grassmann manifold is established through the Exponential/Logarithm maps between the manifold and its tangent space. Finally, given a new parameter, the conditional distribution of a vector can be found in the Euclidean space using the Gaussian Process (GP) regression, and such a distribution is then projected to the Grassmann manifold that enables us to predict the optimal subspace for the new parameter. As a statistical learning approach, the proposed pGP allows us to optimally estimate (or tune) the model parameters from data and quantify the statistical uncertainty associated with the prediction. The advantages of the proposed pGP are demonstrated by numerical experiments.