On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset
This provides a dataset for systematically investigating disentanglement method performance on real versus simulated data, addressing a key bottleneck in representation learning for AI researchers.
The authors tackled the problem of evaluating disentanglement methods on real-world data by creating a new dataset of over one million images of physical 3D objects with seven controlled factors of variation, using a robotic arm. Their results indicate that learned models transfer poorly from simulation to real data, but model and hyperparameter selection can effectively transfer information.
Learning meaningful and compact representations with disentangled semantic aspects is considered to be of key importance in representation learning. Since real-world data is notoriously costly to collect, many recent state-of-the-art disentanglement models have heavily relied on synthetic toy data-sets. In this paper, we propose a novel data-set which consists of over one million images of physical 3D objects with seven factors of variation, such as object color, shape, size and position. In order to be able to control all the factors of variation precisely, we built an experimental platform where the objects are being moved by a robotic arm. In addition, we provide two more datasets which consist of simulations of the experimental setup. These datasets provide for the first time the possibility to systematically investigate how well different disentanglement methods perform on real data in comparison to simulation, and how simulated data can be leveraged to build better representations of the real world. We provide a first experimental study of these questions and our results indicate that learned models transfer poorly, but that model and hyperparameter selection is an effective means of transferring information to the real world.