Sucheng Qian

Zeren Chen, Zhelun Shi, Xiaoya Lu et al.

Achieving generalizability in solving out-of-distribution tasks is one of the ultimate goals of learning robotic manipulation. Recent progress of Vision-Language Models (VLMs) has shown that VLM-based task planners can alleviate the difficulty of solving novel tasks, by decomposing the compounded tasks as a plan of sequentially executing primitive-level skills that have been already mastered. It is also promising for robotic manipulation to adapt such composable generalization ability, in the form of composable generalization agents (CGAs). However, the community lacks of reliable design of primitive skills and a sufficient amount of primitive-level data annotations. Therefore, we propose RH20T-P, a primitive-level robotic manipulation dataset, which contains about 38k video clips covering 67 diverse manipulation tasks in real-world scenarios. Each clip is manually annotated according to a set of meticulously designed primitive skills that are common in robotic manipulation. Furthermore, we standardize a plan-execute CGA paradigm and implement an exemplar baseline called RA-P on our RH20T-P, whose positive performance on solving unseen tasks validates that the proposed dataset can offer composable generalization ability to robotic manipulation agents.

Liu Liu, Wenqiang Xu, Haoyuan Fu et al.

Human life is populated with articulated objects. A comprehensive understanding of articulated objects, namely appearance, structure, physics property, and semantics, will benefit many research communities. As current articulated object understanding solutions are usually based on synthetic object dataset with CAD models without physics properties, which prevent satisfied generalization from simulation to real-world applications in visual and robotics tasks. To bridge the gap, we present AKB-48: a large-scale Articulated object Knowledge Base which consists of 2,037 real-world 3D articulated object models of 48 categories. Each object is described by a knowledge graph ArtiKG. To build the AKB-48, we present a fast articulation knowledge modeling (FArM) pipeline, which can fulfill the ArtiKG for an articulated object within 10-15 minutes, and largely reduce the cost for object modeling in the real world. Using our dataset, we propose AKBNet, a novel integral pipeline for Category-level Visual Articulation Manipulation (C-VAM) task, in which we benchmark three sub-tasks, namely pose estimation, object reconstruction and manipulation. Dataset, codes, and models will be publicly available at https://liuliu66.github.io/articulationobjects/.

Sucheng Qian, Liu Liu, Wenqiang Xu et al.

A large-scale dataset is essential for learning good features in 3D shape understanding, but there are only a few datasets that can satisfy deep learning training. One of the major reasons is that current tools for annotating per-point semantic labels using polygons or scribbles are tedious and inefficient. To facilitate segmentation annotations in 3D shapes, we propose an effective annotation tool, named iSeg for 3D shape. It can obtain a satisfied segmentation result with minimal human clicks (< 10). Under our observation, most objects can be considered as the composition of finite primitive shapes, and we train iSeg3D model on our built primitive-composed shape data to learn the geometric prior knowledge in a self-supervised manner. Given human interactions, the learned knowledge can be used to segment parts on arbitrary shapes, in which positive clicks help associate the primitives into the semantic parts and negative clicks can avoid over-segmentation. Besides, We also provide an online human-in-loop fine-tuning module that enables the model perform better segmentation with less clicks. Experiments demonstrate the effectiveness of iSeg3D on PartNet shape segmentation. Data and codes will be made publicly available.

Jun Lv, Wenqiang Xu, Lixin Yang et al.

3D hand pose estimation and shape recovery are challenging tasks in computer vision. We introduce a novel framework HandTailor, which combines a learning-based hand module and an optimization-based tailor module to achieve high-precision hand mesh recovery from a monocular RGB image. The proposed hand module unifies perspective projection and weak perspective projection in a single network towards accuracy-oriented and in-the-wild scenarios. The proposed tailor module then utilizes the coarsely reconstructed mesh model provided by the hand module as initialization, and iteratively optimizes an energy function to obtain better results. The tailor module is time-efficient, costs only 8ms per frame on a modern CPU. We demonstrate that HandTailor can get state-of-the-art performance on several public benchmarks, with impressive qualitative results on in-the-wild experiments. Code and video are available on our project webpage https://sites.google.com/view/handtailor.