Chuang Wang

Wenzhuo Liu, Xinjian Wu, Fei Zhu et al.

Deep Neural Network (DNN) has achieved great success on datasets of closed class set. However, new classes, like new categories of social media topics, are continuously added to the real world, making it necessary to incrementally learn. This is hard for DNN because it tends to focus on fitting to new classes while ignoring old classes, a phenomenon known as catastrophic forgetting. State-of-the-art methods rely on knowledge distillation and data replay techniques but still have limitations. In this work, we analyze the causes of catastrophic forgetting in class incremental learning, which owes to three factors: representation drift, representation confusion, and classifier distortion. Based on this view, we propose a two-stage learning framework with a fixed encoder and an incrementally updated prototype classifier. The encoder is trained with self-supervised learning to generate a feature space with high intrinsic dimensionality, thus improving its transferability and generality. The classifier incrementally learns new prototypes while retaining the prototypes of previously learned data, which is crucial in preserving the decision boundary.Our method does not rely on preserved samples of old classes, is thus a non-exemplar based CIL method. Experiments on public datasets show that our method can significantly outperform state-of-the-art exemplar-based methods when they reserved 5 examplers per class, under the incremental setting of 10 phases, by 18.24% on CIFAR-100 and 9.37% on ImageNet100.

Tian Wang, Chuang Wang

Pretraining methods gain increasing attraction recently for solving PDEs with neural operators. It alleviates the data scarcity problem encountered by neural operator learning when solving single PDE via training on large-scale datasets consisting of various PDEs and utilizing shared patterns among different PDEs to improve the solution precision. In this work, we propose the Latent Neural Operator Pretraining (LNOP) framework based on the Latent Neural Operator (LNO) backbone. We achieve universal transformation through pretraining on hybrid time-dependent PDE dataset to extract representations of different physical systems and solve various time-dependent PDEs in the latent space through finetuning on single PDE dataset. Our proposed LNOP framework reduces the solution error by 31.7% on four problems and can be further improved to 57.1% after finetuning. On out-of-distribution dataset, our LNOP model achieves roughly 50% lower error and 3$\times$ data efficiency on average across different dataset sizes. These results show that our method is more competitive in terms of solution precision, transfer capability and data efficiency compared to non-pretrained neural operators.

Lineghuan Meng, Chuang Wang

This letter presents a high-dimensional analysis of the training dynamics for a single-layer nonlinear contrastive learning model. The empirical distribution of the model weights converges to a deterministic measure governed by a McKean-Vlasov nonlinear partial differential equation (PDE). Under L2 regularization, this PDE reduces to a closed set of low-dimensional ordinary differential equations (ODEs), reflecting the evolution of the model performance during the training process. We analyze the fixed point locations and their stability of the ODEs unveiling several interesting findings. First, only the hidden variable's second moment affects feature learnability at the state with uninformative initialization. Second, higher moments influence the probability of feature selection by controlling the attraction region, rather than affecting local stability. Finally, independent noises added in the data argumentation degrade performance but negatively correlated noise can reduces the variance of gradient estimation yielding better performance. Despite of the simplicity of the analyzed model, it exhibits a rich phenomena of training dynamics, paving a way to understand more complex mechanism behind practical large models.

Stefano Angieri, Alberto García-Martínez, Bingyang Liu et al.

The current system to manage the global pool of IP addresses is centralized in five transnational organizations, the Regional Internet Registries (RIRs). Each of these RIRs manage the address pool for a large number of countries. Because the RIRs are private organizations, they are subject to the legal framework of the country where they are based. This configuration results in a jurisdictional overflow from the legal framework of the countries where the RIR is based to all the countries that the RIRs are serving (the countries served by the RIRs de facto become subjects of the legal system of the country where the RIR is hosted). The situation is aggravated by the deployment of new security techniques such as the RPKI and BGPsec, that enable enforcement of allocations by the RIRs. In this paper we present InBlock, a blockchain-based distributed governance body aimed to provide de-centralized management of IP addresses. InBlock also aims to fulfil the same objectives as the current IP address allocation system, namely, uniqueness, fairness, conservation, aggregation, registration and minimized overhead. InBlock is implemented as a Decentralized Autonomous Organization, i.e., as a set of blockchain's smart contracts in Ethereum. Any entity may request an allocation of addresses to the InBlock registry by solely performing a (crypto)currency transfer to the InBlock. The fee required, along with the annual renewal fee, serves as a mechanism to deter stockpiling and other wasteful practices. As with any novel technology, there are many open questions about the usage of blockchains to build an IP address registry. For this reason, we believe that practical experimentation is required in order to have hands-on experiences about such a system. We propose to conduct an experiment on distributed address management using InBlock as a starting point to inform future directions in this area.