Visual Perception Engine: Fast and Flexible Multi-Head Inference for Robotic Vision Tasks
This work addresses efficiency and integration challenges for robotic vision tasks, though it is incremental as it builds on existing methods like DINOv2 and CUDA MPS.
The paper tackles the problem of redundant computations and large memory footprints when deploying multiple ML models on resource-constrained robotic platforms by introducing Visual Perception Engine (VPEngine), a modular framework that achieves up to 3x speedup compared to sequential execution and maintains real-time performance at ≥50 Hz on NVIDIA Jetson Orin AGX.
Deploying multiple machine learning models on resource-constrained robotic platforms for different perception tasks often results in redundant computations, large memory footprints, and complex integration challenges. In response, this work presents Visual Perception Engine (VPEngine), a modular framework designed to enable efficient GPU usage for visual multitasking while maintaining extensibility and developer accessibility. Our framework architecture leverages a shared foundation model backbone that extracts image representations, which are efficiently shared, without any unnecessary GPU-CPU memory transfers, across multiple specialized task-specific model heads running in parallel. This design eliminates the computational redundancy inherent in feature extraction component when deploying traditional sequential models while enabling dynamic task prioritization based on application demands. We demonstrate our framework's capabilities through an example implementation using DINOv2 as the foundation model with multiple task (depth, object detection and semantic segmentation) heads, achieving up to 3x speedup compared to sequential execution. Building on CUDA Multi-Process Service (MPS), VPEngine offers efficient GPU utilization and maintains a constant memory footprint while allowing per-task inference frequencies to be adjusted dynamically during runtime. The framework is written in Python and is open source with ROS2 C++ (Humble) bindings for ease of use by the robotics community across diverse robotic platforms. Our example implementation demonstrates end-to-end real-time performance at $\geq$50 Hz on NVIDIA Jetson Orin AGX for TensorRT optimized models.