FusionForce: End-to-end Differentiable Neural-Symbolic Layer for Trajectory Prediction
This addresses the sim-to-real gap and out-of-distribution sensitivity for robotics applications, offering a novel approach that is incremental in combining neural and symbolic methods.
The paper tackles the problem of predicting robot trajectories on rough offroad terrain from sensor data by proposing an end-to-end differentiable model that integrates learnable force predictions with a neural-symbolic layer enforcing classical mechanics, resulting in improved generalization on out-of-distribution data and a speed of 10^4 trajectories per second.
We propose end-to-end differentiable model that predicts robot trajectories on rough offroad terrain from camera images and/or lidar point clouds. The model integrates a learnable component that predicts robot-terrain interaction forces with a neural-symbolic layer that enforces the laws of classical mechanics and consequently improves generalization on out-of-distribution data. The neural-symbolic layer includes a differentiable physics engine that computes the robot's trajectory by querying these forces at the points of contact with the terrain. As the proposed architecture comprises substantial geometrical and physics priors, the resulting model can also be seen as a learnable physics engine conditioned on real sensor data that delivers $10^4$ trajectories per second. We argue and empirically demonstrate that this architecture reduces the sim-to-real gap and mitigates out-of-distribution sensitivity. The differentiability, in conjunction with the rapid simulation speed, makes the model well-suited for various applications including model predictive control, trajectory shooting, supervised and reinforcement learning, or SLAM.