Han Qi

Jiayi Fu, Xuandong Zhao, Chengyuan Yao et al. · berkeley

Reinforcement Learning from Human Feedback (RLHF) is essential for aligning large language models (LLMs) with human values. However, RLHF is susceptible to \emph{reward hacking}, where the agent exploits flaws in the reward function rather than learning the intended behavior, thus degrading alignment. Although reward shaping helps stabilize RLHF and partially mitigate reward hacking, a systematic investigation into shaping techniques and their underlying principles remains lacking. To bridge this gap, we present a comprehensive study of the prevalent reward shaping methods. Our analysis suggests two key design principles: (1) the RL reward should be bounded, and (2) the RL reward benefits from rapid initial growth followed by gradual convergence. Guided by these insights, we propose Preference As Reward (PAR), a novel approach that leverages the latent preferences embedded within the reward model as the signal for reinforcement learning. We evaluated PAR on two base models, Gemma2-2B, and Llama3-8B, using two datasets, Ultrafeedback-Binarized and HH-RLHF. Experimental results demonstrate PAR's superior performance over other reward shaping methods. On the AlpacaEval 2.0 benchmark, PAR achieves a win rate of at least 5 percentage points higher than competing approaches. Furthermore, PAR exhibits remarkable data efficiency, requiring only a single reference reward for optimal performance, and maintains robustness against reward hacking even after two full epochs of training. The code is available at https://github.com/PorUna-byte/PAR, and the Work done during the internship at StepFun by Jiayi Fu.

Yihan Xia, Panpan You, Taotao Wang et al.

A growing body of work explores how Large Language Models (LLMs) can be embedded in trading systems as agents that perceive market information, retrieve context, reason about decisions, emit tradable actions, and adapt under market feedback. This paper reframes LLM-based trading agents as expert-system decision pipelines and presents an audit-oriented evidence map of 77 included studies in a protocol-coded snapshot screened through 2026-03-09. A primary empirical subset (n=19) satisfies the minimum boundary of Action Output plus Closed-Loop Evaluation; the remaining 58 included studies are retained as background and design context. The central empirical finding is protocol incomparability: within the primary subset, only 2/19 studies report extractable time-consistent split protocols, 1/19 reports an explicit transaction-cost model, 1/19 documents universe or survivorship handling, 11/19 report execution timing or semantics, 15/19 are coded as R0, and no study reaches R3 reproducibility. We therefore use Architecture-Capability-Adaptation as a working analytical lens rather than a validated taxonomy, and we foreground the evidence ledger, reproducibility audit, and reporting checklist as the main contributions. The resulting survey shows that architectural experimentation is expanding rapidly, while comparable evaluation protocols, execution semantics, and reproducible artifacts remain the field's immediate bottlenecks.

Yaxing Wang, Qiong Li, Qi Han et al.

As the Quantum Key Distribution (QKD) technology supporting the pointto-point application matures, the need to build the Quantum Secure Communication Network (QSCN) to guarantee the security of a large scale of nodes becomes urgent. Considering the project time and expense control, it is the first choice to build the QSCN based on an existing classical network. Suitable modeling and simulation are very important to construct a QSCN successfully and efficiently. In this paper, a practical QSCN model, which can reflect the network state well, is proposed. The model considers the volatile traffic demand of the classical network and the real key generation capability of the QKD devices, which can enhance the accuracy of simulation to a great extent. In addition, two unique QSCN performance indicators, ITS (information-theoretic secure) communication capability and ITS communication efficiency, are proposed in the model, which are necessary supplements for the evaluation of a QSCN except for those traditional performance indicators of classical networks. Finally, the accuracy of the proposed QSCN model and the necessity of the proposed performance indicators are verified by plentiful simulations results.