Structured Knowledge Accumulation: An Autonomous Framework for Layer-Wise Entropy Reduction in Neural Learning
This provides a new learning paradigm for neural networks, potentially impacting resource-constrained and parallel computing applications, though it appears incremental as it builds on existing entropy and information theory concepts.
The paper tackles the problem of gradient-based optimization in neural networks by introducing the Structured Knowledge Accumulation (SKA) framework, which reinterprets entropy as a layer-wise measure to enable independent layer optimization and hierarchical entropy reduction, offering a scalable, biologically plausible alternative.
We introduce the Structured Knowledge Accumulation (SKA) framework, which reinterprets entropy as a dynamic, layer-wise measure of knowledge alignment in neural networks. Instead of relying on traditional gradient-based optimization, SKA defines entropy in terms of knowledge vectors and their influence on decision probabilities across multiple layers. This formulation naturally leads to the emergence of activation functions such as the sigmoid as a consequence of entropy minimization. Unlike conventional backpropagation, SKA allows each layer to optimize independently by aligning its knowledge representation with changes in decision probabilities. As a result, total network entropy decreases in a hierarchical manner, allowing knowledge structures to evolve progressively. This approach provides a scalable, biologically plausible alternative to gradient-based learning, bridging information theory and artificial intelligence while offering promising applications in resource-constrained and parallel computing environments.