Parameter Convex Neural Networks
This work addresses optimization challenges in neural networks for researchers and practitioners, but it appears incremental as it builds on existing architectures with convexity modifications.
The authors tackled the lack of convexity in deep neural networks, which hinders optimization and generalization, by proposing parameter convex neural networks (PCNNs) and an exponential multilayer neural network (EMLP) that is convex under certain conditions, and they demonstrated improved performance with an exponential graph convolutional network (EGCN) outperforming GCN and GAT on graph classification datasets.
Deep learning utilizing deep neural networks (DNNs) has achieved a lot of success recently in many important areas such as computer vision, natural language processing, and recommendation systems. The lack of convexity for DNNs has been seen as a major disadvantage of many optimization methods, such as stochastic gradient descent, which greatly reduces the genelization of neural network applications. We realize that the convexity make sense in the neural network and propose the exponential multilayer neural network (EMLP), a class of parameter convex neural network (PCNN) which is convex with regard to the parameters of the neural network under some conditions that can be realized. Besides, we propose the convexity metric for the two-layer EGCN and test the accuracy when the convexity metric changes. For late experiments, we use the same architecture to make the exponential graph convolutional network (EGCN) and do the experiment on the graph classificaion dataset in which our model EGCN performs better than the graph convolutional network (GCN) and the graph attention network (GAT).