Demystifying Network Foundation Models
This work identifies critical flaws in NFMs for network analysis, offering insights to enhance their reliability and effectiveness in real-world applications.
The paper systematically investigates the latent knowledge in Network Foundation Models (NFMs) through embedding geometry, metric alignment, and causal sensitivity analyses, revealing significant limitations such as anisotropy and inconsistent feature sensitivity, and shows that addressing these can improve performance by up to +0.35 F1 score without architectural changes.
This work presents a systematic investigation into the latent knowledge encoded within Network Foundation Models (NFMs) that focuses on hidden representations analysis rather than pure downstream task performance. Different from existing efforts, we analyze the models through a three-part evaluation: Embedding Geometry Analysis to assess representation space utilization, Metric Alignment Assessment to measure correspondence with domain-expert features, and Causal Sensitivity Testing to evaluate robustness to protocol perturbations. Using five diverse network datasets spanning controlled and real-world environments, we evaluate four state-of-the-art NFMs, revealing that they all exhibit significant anisotropy, inconsistent feature sensitivity patterns, an inability to separate the high-level context, payload dependency, and other properties. Our work identifies numerous limitations across all models and demonstrates that addressing them can significantly improve model performance (by up to +0.35 $F_1$ score without architectural changes).