Beyond Scaleup: Knowledge-aware Parsimony Learning from Deep Networks
This work addresses the problem of data, computation, and trust bottlenecks in AI development for researchers and practitioners, offering an incremental alternative to scaleup strategies.
The paper tackles the issue of unsustainable brute-force scaling in deep learning by proposing a knowledge-aware parsimony learning framework that uses domain-specific knowledge like symbols and logic as building blocks, achieving results that surpass scaling law-based methods in tasks such as drug-drug interaction prediction.
The brute-force scaleup of training datasets, learnable parameters and computation power, has become a prevalent strategy for developing more robust learning models. However, due to bottlenecks in data, computation, and trust, the sustainability of this strategy is a serious concern. In this paper, we attempt to address this issue in a parsimonious manner (i.e., achieving greater potential with simpler models). The key is to drive models using domain-specific knowledge, such as symbols, logic, and formulas, instead of purely relying on scaleup. This approach allows us to build a framework that uses this knowledge as "building blocks" to achieve parsimony in model design, training, and interpretation. Empirical results show that our methods surpass those that typically follow the scaling law. We also demonstrate our framework in AI for science, specifically in the problem of drug-drug interaction prediction. We hope our research can foster more diverse technical roadmaps in the era of foundation models.