mrCAD: Multimodal Refinement of Computer-aided Designs
This addresses the gap in multimodal refinement capabilities for AI systems, though it is incremental as it focuses on a specific dataset and benchmark.
The paper tackled the problem of generative AI struggling with language-guided modifications of prior outputs by introducing mrCAD, a dataset of multimodal instructions for refining computer-aided designs, which includes 6,082 games and 15,163 rounds from human players, and found that state-of-the-art VLMs perform better on generation than refinement tasks.
A key feature of human collaboration is the ability to iteratively refine the concepts we have communicated. In contrast, while generative AI excels at the \textit{generation} of content, it often struggles to make specific language-guided \textit{modifications} of its prior outputs. To bridge the gap between how humans and machines perform edits, we present mrCAD, a dataset of multimodal instructions in a communication game. In each game, players created computer aided designs (CADs) and refined them over several rounds to match specific target designs. Only one player, the Designer, could see the target, and they must instruct the other player, the Maker, using text, drawing, or a combination of modalities. mrCAD consists of 6,082 communication games, 15,163 instruction-execution rounds, played between 1,092 pairs of human players. We analyze the dataset and find that generation and refinement instructions differ in their composition of drawing and text. Using the mrCAD task as a benchmark, we find that state-of-the-art VLMs are better at following generation instructions than refinement instructions. These results lay a foundation for analyzing and modeling a multimodal language of refinement that is not represented in previous datasets.