Memorization capacity of deep ReLU neural networks characterized by width and depth
This provides a foundational theoretical characterization of memorization capacity in deep learning, offering insights into the width-depth trade-off for researchers and practitioners.
The paper tackles the problem of determining the minimal network size needed for deep ReLU neural networks to memorize N data points with separation δ, establishing that width W and depth L must satisfy W^2L^2 = Θ(N log(δ^{-1})) for optimal memorization, with their construction achieving this bound up to logarithmic factors.
This paper studies the memorization capacity of deep neural networks with ReLU activation. Specifically, we investigate the minimal size of such networks to memorize any $N$ data points in the unit ball with pairwise separation distance $δ$ and discrete labels. Most prior studies characterize the memorization capacity by the number of parameters or neurons. We generalize these results by constructing neural networks, whose width $W$ and depth $L$ satisfy $W^2L^2= \mathcal{O}(N\log(δ^{-1}))$, that can memorize any $N$ data samples. We also prove that any such networks should also satisfy the lower bound $W^2L^2=Ω(N \log(δ^{-1}))$, which implies that our construction is optimal up to logarithmic factors when $δ^{-1}$ is polynomial in $N$. Hence, we explicitly characterize the trade-off between width and depth for the memorization capacity of deep neural networks in this regime.