CAN We Trust Your Results? A Cross-Dataset Study of Automotive IDS Evaluation
For researchers evaluating CAN IDS, this work demonstrates the unreliability of single-dataset evaluations and provides a framework for more robust assessment.
The paper introduces a benchmarking framework for consistent evaluation of CAN Intrusion Detection Systems across seven datasets, showing that detection performance varies significantly across datasets, highlighting the need for cross-dataset benchmarking.
The increasing connectivity of modern vehicles has made securing in-vehicle communication networks a critical challenge. Intrusion Detection Systems (IDS) have been widely studied as a defense mechanism for detecting malicious activities on the Controller Area Network (CAN) bus. However, the evaluation of CAN IDS methods remains difficult due to inconsistencies in experimental setups and the lack of standardized benchmarking frameworks. As a result, reported performance often depends on dataset-specific characteristics and may not reflect how detection methods behave in different environments. This work introduces a benchmarking framework for consistent evaluation of CAN IDSs across multiple datasets. Using the proposed framework, we integrate seven publicly available CAN IDS datasets collected under different experimental conditions and perform cross-dataset evaluation of five conceptually different IDS approaches. Our results highlight how detection performance can vary significantly across datasets, demonstrating the importance of cross-dataset benchmarking for assessing the robustness and generalization capabilities of CAN IDS methods.