Tianci Gao

Tianci Gao, Konstantin A. Neusypin, Dmitry D. Dmitriev et al.

Proximal Policy Optimization (PPO) is widely used in continuous control due to its robustness and stable training, yet it remains sample-inefficient in tasks with expensive interactions and high-dimensional action spaces. This paper proposes PPO-DAP (PPO with Diffusion Action Prior), a strictly on-policy framework that improves exploration quality and learning efficiency without modifying the PPO objective. PPO-DAP follows a two-stage protocol. Offline, we pretrain a conditional diffusion action prior on logged trajectories to cover the action distribution supported by the behavior policy. Online, PPO updates the actor-critic only using newly collected on-policy rollouts, while the diffusion prior is adapted around the on-policy state distribution via parameter-efficient tuning (Adapter/LoRA) over a small parameter subset. For each on-policy state, the prior generates multiple action proposals and concentrates them toward high-value regions using critic-based energy reweighting and in-denoising gradient guidance. These proposals affect the actor only through a low-weight imitation loss and an optional soft KL regularizer to the prior; importantly, PPO gradients are never backpropagated through offline logs or purely synthetic trajectories. We further analyze the method from a dual-proximal perspective and derive a one-step performance lower bound. Across eight MuJoCo continuous-control tasks under a unified online budget of 1.0M environment steps, PPO-DAP consistently improves early learning efficiency (area under the learning curve over the first 40 epochs, ALC@40) and matches or exceeds the strongest on-policy baselines in final return on 6/8 tasks, with modest overhead (1.18+/-0.04x wall-clock time and 1.05+/-0.02x peak GPU memory relative to PPO).

Ruoxuan Feng, Jiangyu Hu, Wenke Xia et al.

Visuo-tactile sensors aim to emulate human tactile perception, enabling robots to precisely understand and manipulate objects. Over time, numerous meticulously designed visuo-tactile sensors have been integrated into robotic systems, aiding in completing various tasks. However, the distinct data characteristics of these low-standardized visuo-tactile sensors hinder the establishment of a powerful tactile perception system. We consider that the key to addressing this issue lies in learning unified multi-sensor representations, thereby integrating the sensors and promoting tactile knowledge transfer between them. To achieve unified representation of this nature, we introduce TacQuad, an aligned multi-modal multi-sensor tactile dataset from four different visuo-tactile sensors, which enables the explicit integration of various sensors. Recognizing that humans perceive the physical environment by acquiring diverse tactile information such as texture and pressure changes, we further propose to learn unified multi-sensor representations from both static and dynamic perspectives. By integrating tactile images and videos, we present AnyTouch, a unified static-dynamic multi-sensor representation learning framework with a multi-level structure, aimed at both enhancing comprehensive perceptual abilities and enabling effective cross-sensor transfer. This multi-level architecture captures pixel-level details from tactile data via masked modeling and enhances perception and transferability by learning semantic-level sensor-agnostic features through multi-modal alignment and cross-sensor matching. We provide a comprehensive analysis of multi-sensor transferability, and validate our method on various datasets and in the real-world pouring task. Experimental results show that our method outperforms existing methods, exhibits outstanding static and dynamic perception capabilities across various sensors.