Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoning
This addresses the challenge of reliable robotic code generation from demonstrations in varied environments, representing a novel method for a known bottleneck in cross-domain adaptation.
The paper tackles the problem of video-instructed robotic programming under cross-domain shifts, where perceptual and physical differences cause procedural mismatches, and introduces NeSyCR, a neurosymbolic counterfactual reasoning framework that improves task success by 31.14% over the strongest baseline.
Recent advances in Vision-Language Models (VLMs) have enabled video-instructed robotic programming, allowing agents to interpret video demonstrations and generate executable control code. We formulate video-instructed robotic programming as a cross-domain adaptation problem, where perceptual and physical differences between demonstration and deployment induce procedural mismatches. However, current VLMs lack the procedural understanding needed to reformulate causal dependencies and achieve task-compatible behavior under such domain shifts. We introduce NeSyCR, a neurosymbolic counterfactual reasoning framework that enables verifiable adaptation of task procedures, providing a reliable synthesis of code policies. NeSyCR abstracts video demonstrations into symbolic trajectories that capture the underlying task procedure. Given deployment observations, it derives counterfactual states that reveal cross-domain incompatibilities. By exploring the symbolic state space with verifiable checks, NeSyCR proposes procedural revisions that restore compatibility with the demonstrated procedure. NeSyCR achieves a 31.14% improvement in task success over the strongest baseline Statler, showing robust cross-domain adaptation across both simulated and real-world manipulation tasks.