SciTextures: Collecting and Connecting Visual Patterns, Models, and Code Across Science and Art
This work provides a foundational resource for researchers in AI and visual understanding to study pattern-mechanism relationships across diverse domains.
The authors introduced the SciTextures dataset, a large-scale collection of over 1,200 models and 100,000 images of textures and visual patterns from science, tech, and art, to explore connections between visual patterns and their generating mechanisms. They used this dataset to benchmark AI models, showing that vision-language models can understand and simulate underlying physical systems, such as by generating simulated images from real-world patterns.
The ability to connect visual patterns with the processes that form them represents one of the deepest forms of visual understanding. Textures of clouds and waves, the growth of cities and forests, or the formation of materials and landscapes are all examples of patterns emerging from underlying mechanisms. We present the Scitextures dataset, a large-scale collection of textures and visual patterns from all domains of science, tech, and art, along with the models and code that generate these images. Covering over 1,200 different models and 100,000 images of patterns and textures from physics, chemistry, biology, sociology, technology, mathematics, and art, this dataset offers a way to explore the connection between the visual patterns that shape our world and the mechanisms that produce them. Created by an agentic AI pipeline that autonomously collects and implements models in standardized form, we use SciTextures to evaluate the ability of leading AI models to link visual patterns to the models and code that generate them, and to identify different patterns that emerged from the same process. We also test AIs ability to infer and recreate the mechanisms behind visual patterns by providing a natural image of a real-world pattern and asking the AI to identify, model, and code the mechanism that formed the pattern, then run this code to generate a simulated image that is compared to the real image. These benchmarks show that vision-language models (VLMs) can understand and simulate the physical system beyond a visual pattern. The dataset and code are available at: https://zenodo.org/records/17485502