Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
This addresses the problem of efficient and effective AI deployment in autonomous systems, particularly for power/resource-limited or heterogeneous platforms, though it appears incremental as it builds on existing multi-modal multi-task learning techniques.
The paper tackles the challenge of simultaneously optimizing quality of result and quality of service in AI-driven autonomous systems by proposing a software/hardware co-design approach for multi-modal multi-task learning, formulated as a differentiable optimization problem to improve solution quality while reducing power consumption and latency.
Optimizing the quality of result (QoR) and the quality of service (QoS) of AI-empowered autonomous systems simultaneously is very challenging. First, there are multiple input sources, e.g., multi-modal data from different sensors, requiring diverse data preprocessing, sensor fusion, and feature aggregation. Second, there are multiple tasks that require various AI models to run simultaneously, e.g., perception, localization, and control. Third, the computing and control system is heterogeneous, composed of hardware components with varied features, such as embedded CPUs, GPUs, FPGAs, and dedicated accelerators. Therefore, autonomous systems essentially require multi-modal multi-task (MMMT) learning which must be aware of hardware performance and implementation strategies. While MMMT learning has been attracting intensive research interests, its applications in autonomous systems are still underexplored. In this paper, we first discuss the opportunities of applying MMMT techniques in autonomous systems and then discuss the unique challenges that must be solved. In addition, we discuss the necessity and opportunities of MMMT model and hardware co-design, which is critical for autonomous systems especially with power/resource-limited or heterogeneous platforms. We formulate the MMMT model and heterogeneous hardware implementation co-design as a differentiable optimization problem, with the objective of improving the solution quality and reducing the overall power consumption and critical path latency. We advocate for further explorations of MMMT in autonomous systems and software/hardware co-design solutions.