A Statistical Investigation of Long Memory in Language and Music
This addresses a fundamental bottleneck in sequence modeling for machine learning applications, though it appears incremental as it adapts existing statistical theory to deep learning contexts.
The authors tackled the problem of measuring long-range dependence in sequence data and deep models, which currently relies on heuristic tools, by developing a statistical framework based on long memory stochastic process theory. They explored testable implications about how long memory in real-world data relates to learned representations in deep architectures through a semiparametric approach adapted to high dimensions.
Representation and learning of long-range dependencies is a central challenge confronted in modern applications of machine learning to sequence data. Yet despite the prominence of this issue, the basic problem of measuring long-range dependence, either in a given data source or as represented in a trained deep model, remains largely limited to heuristic tools. We contribute a statistical framework for investigating long-range dependence in current applications of deep sequence modeling, drawing on the well-developed theory of long memory stochastic processes. This framework yields testable implications concerning the relationship between long memory in real-world data and its learned representation in a deep learning architecture, which are explored through a semiparametric framework adapted to the high-dimensional setting.