Positional Encoding Field
This work addresses the challenge of improving 3D reasoning and spatial control in visual generation models for applications like novel view synthesis and image editing.
The paper tackled the problem of how Diffusion Transformers (DiTs) organize visual content by discovering that patch tokens are largely independent, with spatial coherence governed by positional encodings (PEs). It introduced the Positional Encoding Field (PE-Field), extending PEs to a 3D field, which achieved state-of-the-art performance on single-image novel view synthesis and enabled controllable spatial image editing.
Diffusion Transformers (DiTs) have emerged as the dominant architecture for visual generation, powering state-of-the-art image and video models. By representing images as patch tokens with positional encodings (PEs), DiTs combine Transformer scalability with spatial and temporal inductive biases. In this work, we revisit how DiTs organize visual content and discover that patch tokens exhibit a surprising degree of independence: even when PEs are perturbed, DiTs still produce globally coherent outputs, indicating that spatial coherence is primarily governed by PEs. Motivated by this finding, we introduce the Positional Encoding Field (PE-Field), which extends positional encodings from the 2D plane to a structured 3D field. PE-Field incorporates depth-aware encodings for volumetric reasoning and hierarchical encodings for fine-grained sub-patch control, enabling DiTs to model geometry directly in 3D space. Our PE-Field-augmented DiT achieves state-of-the-art performance on single-image novel view synthesis and generalizes to controllable spatial image editing.