Historical Document Image Segmentation with LDA-Initialized Deep Neural Networks
This addresses a specific bottleneck in training efficiency for historical document analysis, offering an incremental improvement over existing initialization methods.
The paper tackles the problem of slow and unstable weight initialization in deep neural networks for historical document image segmentation by proposing a Linear Discriminant Analysis (LDA)-based method, which outperforms state-of-the-art random initialization techniques.
In this paper, we present a novel approach to perform deep neural networks layer-wise weight initialization using Linear Discriminant Analysis (LDA). Typically, the weights of a deep neural network are initialized with: random values, greedy layer-wise pre-training (usually as Deep Belief Network or as auto-encoder) or by re-using the layers from another network (transfer learning). Hence, many training epochs are needed before meaningful weights are learned, or a rather similar dataset is required for seeding a fine-tuning of transfer learning. In this paper, we describe how to turn an LDA into either a neural layer or a classification layer. We analyze the initialization technique on historical documents. First, we show that an LDA-based initialization is quick and leads to a very stable initialization. Furthermore, for the task of layout analysis at pixel level, we investigate the effectiveness of LDA-based initialization and show that it outperforms state-of-the-art random weight initialization methods.