Yuki Inoue

Yuki Inoue, Hiroki Ohashi

Embodied Instruction Following (EIF) studies how autonomous mobile manipulation robots should be controlled to accomplish long-horizon tasks described by natural language instructions. While much research on EIF is conducted in simulators, the ultimate goal of the field is to deploy the agents in real life. This is one of the reasons why recent methods have moved away from training models end-to-end and take modular approaches, which do not need the costly expert operation data. However, as it is still in the early days of importing modular ideas to EIF, a search for modules effective in the EIF task is still far from a conclusion. In this paper, we propose to extend the modular design using knowledge obtained from two external sources. First, we show that embedding the physical constraints of the deployed robots into the module design is highly effective. Our design also allows the same modular system to work across robots of different configurations with minimal modifications. Second, we show that the landmark-based object search, previously implemented by a trained model requiring a dedicated set of data, can be replaced by an implementation that prompts pretrained large language models for landmark-object relationships, eliminating the need for collecting dedicated training data. Our proposed Prompter achieves 41.53\% and 45.32\% on the ALFRED benchmark with high-level instructions only and step-by-step instructions, respectively, significantly outperforming the previous state of the art by 5.46\% and 9.91\%.

Yuki Inoue, Hiroto Nagayoshi

Pixel-level crack segmentation is widely studied due to its high impact on building and road inspections. While recent studies have made significant improvements in accuracy, they typically heavily depend on pixel-level crack annotations, which are time-consuming to obtain. In earlier work, we proposed to reduce the annotation cost bottleneck by reformulating the crack segmentation problem as a weakly-supervised problem -- i.e. the annotation process is expedited by sacrificing the annotation quality. The loss in annotation quality was remedied by refining the inference with per-pixel brightness values, which was effective when the pixel brightness distribution between cracks and non-cracks are well separated, but struggled greatly for lighter-colored cracks as well as non-crack targets in which the brightness distribution is less articulated. In this work, we propose an annotation refinement approach which takes advantage of the fact that the regions falsely annotated as cracks have similar local visual features as the background. Because the proposed approach is data-driven, it is effective regardless of a dataset's pixel brightness profile. The proposed method is evaluated on three crack segmentation datasets as well as one blood vessel segmentation dataset to test for domain robustness, and the results show that it speeds up the annotation process by factors of 10 to 30, while the detection accuracy stays at a comparable level.

Yuki Inoue, Hiroto Nagayoshi

Automatic crack detection is a critical task that has the potential to drastically reduce labor-intensive building and road inspections currently being done manually. Recent studies in this field have significantly improved the detection accuracy. However, the methods often heavily rely on costly annotation processes. In addition, to handle a wide variety of target domains, new batches of annotations are usually required for each new environment. This makes the data annotation cost a significant bottleneck when deploying crack detection systems in real life. To resolve this issue, we formulate the crack detection problem as a weakly-supervised problem and propose a two-branched framework. By combining predictions of a supervised model trained on low quality annotations with predictions based on pixel brightness, our framework is less affected by the annotation quality. Experimental results show that the proposed framework retains high detection accuracy even when provided with low quality annotations. Implementation of the proposed framework is publicly available at https://github.com/hitachi-rd-cv/weakly-sup-crackdet.