ML-based tunnel detection and tunneled application classification
This work addresses the challenge of maintaining visibility on encrypted tunneling for network security, particularly against malicious uses, but it is incremental as it builds on existing approaches with extended protocol coverage and analysis.
The paper tackled the problem of detecting encrypted tunneling protocols and classifying applications inside them using network traffic metadata and machine learning, achieving performance improvements by addressing OpenVPN and Wireguard, proposing a complete pipeline, and analyzing domain generalization and adversarial learning.
Encrypted tunneling protocols are widely used. Beyond business and personal uses, malicious actors also deploy tunneling to hinder the detection of Command and Control and data exfiltration. A common approach to maintain visibility on tunneling is to rely on network traffic metadata and machine learning to analyze tunnel occurrence without actually decrypting data. Existing work that address tunneling protocols however exhibit several weaknesses: their goal is to detect application inside tunnels and not tunnel identification, they exhibit limited protocol coverage (e.g. OpenVPN and Wireguard are not addressed), and both inconsistent features and diverse machine learning techniques which makes performance comparison difficult. Our work makes four contributions that address these limitations and provide further analysis. First, we address OpenVPN and Wireguard. Second, we propose a complete pipeline to detect and classify tunneling protocols and tunneled applications. Third, we present a thorough analysis of the performance of both network traffic metadata features and machine learning techniques. Fourth, we provide a novel analysis of domain generalization regarding background untunneled traffic, and, both domain generalization and adversarial learning regarding Maximum Transmission Unit (MTU).