Jiajun Wu

Yunong Liu, Cristobal Eyzaguirre, Manling Li et al. · salesforce, stanford

Shape assembly is a ubiquitous task in daily life, integral for constructing complex 3D structures like IKEA furniture. While significant progress has been made in developing autonomous agents for shape assembly, existing datasets have not yet tackled the 4D grounding of assembly instructions in videos, essential for a holistic understanding of assembly in 3D space over time. We introduce IKEA Video Manuals, a dataset that features 3D models of furniture parts, instructional manuals, assembly videos from the Internet, and most importantly, annotations of dense spatio-temporal alignments between these data modalities. To demonstrate the utility of IKEA Video Manuals, we present five applications essential for shape assembly: assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and furniture assembly based on instructional video manuals. For each application, we provide evaluation metrics and baseline methods. Through experiments on our annotated data, we highlight many challenges in grounding assembly instructions in videos to improve shape assembly, including handling occlusions, varying viewpoints, and extended assembly sequences.

Jiajun Wu, Chengjian Sun, Chenyang Yang

Optimizing resource allocation with predicted information has shown promising gain in boosting network performance and improving user experience. Earlier research efforts focus on optimizing proactive policies under the assumption of knowing the future information. Recently, various techniques have been proposed to predict the required information, and the prediction results were then treated as the true value in the optimization, i.e., "first-predict-then-optimize". In this paper, we introduce a proactive optimization framework for anticipatory resource allocation, where the future information is implicitly predicted under the same objective with the policy optimization in a single step. An optimization problem is formulated to integrate the implicit prediction and the policy optimization, based on the conditional distribution of the future information given the historical observations. To solve such a problem, we transform it equivalently to a problem depending on the joint distribution of future and historical information. Then, we resort to unsupervised learning with neural networks to learn the proactive policy as a function of the past observations via stochastic optimization. We take proactive caching and bandwidth allocation at base stations as a concrete example, where the objective function is the conditional expectation of successful offloading probability taken over the future popularity given the historically observed popularity. We use simulation to validate the proposed framework and compare it with the "first-predict-then-optimize" strategy and a heuristic "end-to-end" optimization strategy with supervised learning.