Sharpness-aware Second-order Latent Factor Model for High-dimensional and Incomplete Data
This work addresses a specific bottleneck in representation learning for high-dimensional incomplete data, offering an incremental improvement to existing methods.
The paper tackles the optimization difficulty of second-order latent factor models for high-dimensional incomplete data by proposing a sharpness-aware variant that uses Hessian-vector products to find flat minima, resulting in consistent outperformance over state-of-the-art baselines on industrial datasets.
Second-order Latent Factor (SLF) model, a class of low-rank representation learning methods, has proven effective at extracting node-to-node interaction patterns from High-dimensional and Incomplete (HDI) data. However, its optimization is notoriously difficult due to its bilinear and non-convex nature. Sharpness-aware Minimization (SAM) has recently proposed to find flat local minima when minimizing non-convex objectives, thereby improving the generalization of representation-learning models. To address this challenge, we propose a Sharpness-aware SLF (SSLF) model. SSLF embodies two key ideas: (1) acquiring second-order information via Hessian-vector products; and (2) injecting a sharpness term into the curvature (Hessian) through the designed Hessian-vector products. Experiments on multiple industrial datasets demonstrate that the proposed model consistently outperforms state-of-the-art baselines.