The FIL Hypothesis: Inductive Biases Help with Kernel Engineering
For AI researchers and practitioners, this work highlights a fundamental scaling limit of data-driven methods in domains with long feedback loops, though the validation is limited to a single task.
The paper introduces the Feedback Information Loop (FIL) hypothesis, arguing that future AI applications with long feedback loops (hours to weeks) will limit purely data-driven methods, and proposes using inductive biases as an orthogonal approach. Initial validation on GPU programming shows superior performance over data-driven methods.
The Bitter Lesson, which posits that general-purpose methods that scale with computation and data ultimately outperform those with built-in human knowledge, has become a dominant paradigm in the era of Large Language Models. We revisit this principle by observing a new and critical scaling dimension: the duration of the Feedback Information Loop (FIL), the time required for a system to receive a verification signal after generating a prediction. Most historic successes in Artificial Intelligence (AI) have benefited from near instantaneous feedback (e.g., games or classification tasks), but we argue that future AI applications in science and the physical world will inherently involve FILs ranging from hours to weeks. This trend poses a fundamental scaling limit, as obtaining enough verification steps required by purely data-driven methods becomes practically impossible. Additionally, we propose a method that is orthogonal to purely data-driven approaches, based on human-inspired expert knowledge. The method relies on inductive biases and constraining the solution space. We provide an initial validation of the hypothesis and the method, by studying the real-world GPU programming task, a domain with non-trivial FIL, and demonstrate that incorporating inductive biases yields superior performance over data-driven approaches. The code is released under: https://github.com/ai-nikolai/robust_kernelbench