Zipeng Lin

Zipeng Lin, Zhenguo Nie

In this paper, we propose an approach to address the problem of classifying 3D industrial components by introducing a novel framework named IC-classifier (Industrial Component classifier). Our framework is designed to focus on the object's local and global structures, emphasizing the former by incorporating specific local features for embedding the model. By utilizing graphical neural networks and embedding derived from geometric properties, IC-classifier facilitates the exploration of the local structures of the object while using geometric attention for the analysis of global structures. Furthermore, the framework uses point clouds to circumvent the heavy computation workload. The proposed framework's performance is benchmarked against state-of-the-art models, demonstrating its potential to compete in the field.

Yuexiang Zhai, Hao Bai, Zipeng Lin et al. · berkeley

Large vision-language models (VLMs) fine-tuned on specialized visual instruction-following data have exhibited impressive language reasoning capabilities across various scenarios. However, this fine-tuning paradigm may not be able to efficiently learn optimal decision-making agents in multi-step goal-directed tasks from interactive environments. To address this challenge, we propose an algorithmic framework that fine-tunes VLMs with reinforcement learning (RL). Specifically, our framework provides a task description and then prompts the VLM to generate chain-of-thought (CoT) reasoning, enabling the VLM to efficiently explore intermediate reasoning steps that lead to the final text-based action. Next, the open-ended text output is parsed into an executable action to interact with the environment to obtain goal-directed task rewards. Finally, our framework uses these task rewards to fine-tune the entire VLM with RL. Empirically, we demonstrate that our proposed framework enhances the decision-making capabilities of VLM agents across various tasks, enabling 7b models to outperform commercial models such as GPT4-V or Gemini. Furthermore, we find that CoT reasoning is a crucial component for performance improvement, as removing the CoT reasoning results in a significant decrease in the overall performance of our method.