Not Every Dependency Is Worth Discovering: Toward Value-Driven Data Dependency Discovery
For data management researchers, it proposes a paradigm shift from validity-driven to value-driven dependency discovery, addressing the problem of irrelevant or costly dependencies.
The paper argues that data dependency discovery should focus on value rather than validity, defining dependency value as expected reduction in task-specific loss minus lifecycle costs, and outlines a research agenda for value-aware discovery.
Data dependency discovery has traditionally focused on identifying dependencies that hold in the data or are statistically strong. Yet a dependency may be valid without being valuable: it may be irrelevant to the governance task, redundant given existing knowledge, or too costly to discover, validate, maintain, and apply. We call for a shift from validity-driven to value-driven dependency discovery. We define dependency use value decision-theoretically as the expected reduction in task-specific loss from incorporating a dependency into the governance process, and define net value by further accounting for lifecycle costs. Building on this framework, we outline principles for value-aware search, validation, dependency-set selection, and maintenance, and identify a research agenda spanning value estimation before full discovery, loss and cost learning, budgeted set selection, lifecycle monitoring, and benchmarking.