Lempel-Ziv Networks
This work addresses the challenge of sequence processing for long sequences in machine learning, but it is incremental as it presents a negative result and highlights baseline tuning issues.
The paper tackled the problem of processing very long sequences by extending the Lempel-Ziv Jaccard Distance (LZJD) to continuous domains with a deep-learning analog called Lempel-Ziv Network, but it did not improve meaningfully on standard LSTM performance across various datasets and tasks.
Sequence processing has long been a central area of machine learning research. Recurrent neural nets have been successful in processing sequences for a number of tasks; however, they are known to be both ineffective and computationally expensive when applied to very long sequences. Compression-based methods have demonstrated more robustness when processing such sequences -- in particular, an approach pairing the Lempel-Ziv Jaccard Distance (LZJD) with the k-Nearest Neighbor algorithm has shown promise on long sequence problems (up to $T=200,000,000$ steps) involving malware classification. Unfortunately, use of LZJD is limited to discrete domains. To extend the benefits of LZJD to a continuous domain, we investigate the effectiveness of a deep-learning analog of the algorithm, the Lempel-Ziv Network. While we achieve successful proof of concept, we are unable to improve meaningfully on the performance of a standard LSTM across a variety of datasets and sequence processing tasks. In addition to presenting this negative result, our work highlights the problem of sub-par baseline tuning in newer research areas.