Cut-and-Approximate: 3D Shape Reconstruction from Planar Cross-sections with Deep Reinforcement Learning
This addresses the problem of detailed 3D reconstruction from limited cross-sections for applications in medical imaging or computer graphics, though it appears incremental as it builds on existing reinforcement learning and imitation learning techniques.
The paper tackles 3D shape reconstruction from planar cross-sections by introducing a method that uses deep reinforcement learning to iteratively cut and approximate parts of the shape, aiming to capture detailed topology with fewer cross-sections. Experiments show the method learns efficient policies faster and produces visually compelling results.
Current methods for 3D object reconstruction from a set of planar cross-sections still struggle to capture detailed topology or require a considerable number of cross-sections. In this paper, we present, to the best of our knowledge the first 3D shape reconstruction network to solve this task which additionally uses orthographic projections of the shape. Our method is based on applying a Reinforcement Learning algorithm to learn how to effectively parse the shape using a trial-and-error scheme relying on scalar rewards. This method cuts a part of a 3D shape in each step which is then approximated as a polygon mesh. The agent aims to maximize the reward that depends on the accuracy of surface reconstruction for the approximated parts. We also consider pre-training of the network for faster learning using demonstrations generated by a heuristic approach. Experiments show that our training algorithm which benefits from both imitation learning and also self exploration, learns efficient policies faster, which results the agent to produce visually compelling results.