Evaluation and Calibration of Cross-Dataset Robustness in IoT Intrusion Detection Systems: A Deployment-Aware Approach
DOI:
https://doi.org/10.65080/mjai.v1.CM2601105008Keywords:
IoT security, intrusion detection, deep feature extraction, denoising autoencoder, domain shift, cross-dataset evaluationAbstract
Introduction: The rapid expansion of Internet of Things (IoT) deployments has increased the cyber-attack surface and introduced heterogeneous traffic behaviour across devices, gateways, edge services, and network environments. Although many IoT intrusion detection studies report high performance under independent and identically distributed test conditions, such results often provide limited evidence of real deployment reliability, particularly for unseen hosts, cross-dataset transfers, and calibration drifts.
Methodology: This study presents a deployment-aware evaluation and calibration framework for IoT intrusion detection using a Denoising Autoencoder-Based Deep Feature Extraction (DAE-DFE) backbone. Rather than proposing a new neural architecture, this study focuses on robustness-oriented evaluation protocols and calibration-aware decision-making under domain shifts. The framework was evaluated using conventional IID splits, identifier-removed testing, and source-IP-based GroupSplit evaluation on NF-ToN-IoT-v2 to reduce the memorisation of the host. Cross-dataset robustness was assessed by converting Edge-IIoTset packet/protocol logs into pseudo-flows and testing Edge→NF and NF→Edge transfer using unsupervised threshold adaptation based on positive rate matching.
Results: On the NF-ToN-IoT-v2 GroupSplit, the framework achieved an F1 score of 0.9919 and ROC-AUC of 0.9997. In the Edge→NF cross-dataset setting, the model retained a meaningful ranking performance with ROC-AUC = 0.7962, while unsupervised threshold adaptation improved the target-domain accuracy from 0.4959 to 0.8878 and F1 score from 0.6630 to 0.8981.
Conclusion: The findings show that calibration-aware thresholding and deployment-realistic evaluation are essential for assessing IoT IDS reliability beyond the conventional IID accuracy.
References
Ali O, Ishak MK, Bhatti MKL, Khan I, Kim KI. A comprehensive review of internet of things: technology stack, middlewares, and fog/edge computing interface. Sensors. 2022; 22(3): 995. https://doi.org/10.3390/s22030995
Asadi M, Jamali MAJ, Heidari A, Navimipour NJ. Botnets unveiled: a comprehensive survey on evolving threats and defense strategies. Trans Emerg Telecommun Technol. 2024; 35(11): e5056. https://doi.org/10.1002/ett.5056
Mukherjee A. The complete guide to defense in depth: Learn to identify, mitigate, and prevent cyber threats with a dynamic, layered defense approach. Birmingham: Packt Publishing Ltd; 2024. Available from: https://books.google.com.pk/books?id=F9cTEQAAQBAJ&redir_esc=y
Booij TM, Chiscop I, Meeuwissen E, Moustafa N, Den Hartog FT. ToN_IoT: the role of heterogeneity and the need for standardization of features and attack types in IoT network intrusion data sets. IEEE Internet Things J. 2022; 9(1): 485-496. https://doi.org/10.1109/JIOT.2021.3085194
Chen Q, Tan L, Tang J, Qu X. AI-enabled IoT security: a survey on advances, challenges, and cross-domain collaborative frameworks. In: Proceedings of the 2025 8th International Conference on Computer Information Science and Artificial Intelligence. 2025. p.1615-1621. https://doi.org/10.1145/3773365.3773619
Kipkorir P, Mwangi E, Wasike J. A machine learning-based packet sniffer for detection and classification of the dedenial-of-servicettack packets at the network layer. 2025. https://doi.org/10.51584/IJRIAS.2025.100500051
Xin Q, Xu Z, Guo L, Zhao F, Wu B. IoT traffic classification and anomaly detection method based on deep autoencoders. Preprints. 2024. https://doi.org/10.20944/preprints202407.0530.v1
Meidan Y, Bohadana M, Mathov Y, Mirsky Y, Breitenbacher D, Shabtai A, et al. N-BaIoT: Network-based detection of IoT botnet attacks using deep autoencoders. IEEE Pervasive Comput. 2018; 27(3): 12-22. https://doi.org/10.1109/MPRV.2018.03367731
Alsaedi A, Moustafa N, Tari Z, Mahmood A, Anwar A. TON_IoT telemetry dataset: a new generation dataset of IoT and IIoT for data-driven intrusion detection systems. IEEE Access. 2020; 8: 165130-165150. https://doi.org/10.1109/ACCESS.2020.3022862
Guo C, Pleiss G, Sun Y, Weinberger KQ. On calibration of modern neural networks. In: International Conference on Machine Learning. PMLR; 2017; 70: p.1321-1330. https://proceedings.mlr.press/v70/guo17a.html
Niculescu-Mizil A, Caruana R. Predicting good probabilities with supervised learning. In: Proceedings of the 22nd International Conference on Machine Learning. 2005. p.625-632. https://doi.org/10.1145/1102351.1102430
Rafique SH, Abdallah A, Musa NS, Murugan T. Machine learning and deep learning techniques for internet of things network anomaly detection: current research trends. Sensors. 2024; 24(6): 1968. https://doi.org/10.3390/s24061968
Rahman MM, Al Shakil S, Mustakim MR. A survey on intrusion detection system in IoT networks. Cyber Secur Appl. 2025; 3: 100082. https://doi.org/10.1016/j.csa.2024.100082
Wu J, Wang Y. TriHID: towards verifiable domain adaptation-based IoT intrusion detection in heterogeneous environment. Expert Syst Appl. 2026; 298(A): 129543. https://doi.org/10.1016/j.eswa.2025.129543
Lopes IO, Zou D, Abdulqadder IH, Ruambo FA, Yuan B, Jin H. Effective network intrusion detection via representation learning: a denoising autoencoder approach. Comput Commun. 2022; 194: 55-65. https://doi.org/10.1016/j.comcom.2022.07.027
Khraisat A, Alazab A. A critical review of intrusion detection systems in the internet of things: techniques, deployment strategy, validation strategy, attacks, public datasets and challenges. Cybersecurity. 2021; 4(1): 18. https://doi.org/10.1186/s42400-021-00077-7
Alrayes FS, Zakariah M, Amin SU, Khan ZI, Helal M. Intrusion detection in IoT systems using denoising autoencoder. IEEE Access. 2024; 12: 122401-122425 https://doi.org/10.1109/ACCESS.2024.3451726
Imani M, Joudaki M, Bagheri A, Arabnia HR. Why ROC-AUC alone is misleading for highly imbalanced data: in-depth evaluation of MCC, F2-score, H-measure, and AUC-based metrics across diverse classifiers. Technologies. 2025; 14(1): 54. https://doi.org/10.3390/technologies14010054
Ferrag MA, Friha O, Hamouda D, Maglaras L, Janicke H. Edge-IIoTset: a new comprehensive realistic cyber security dataset of IoT and IIoT applications for centralized and federated learning. IEEE Access. 2022; 10: 40281-40306. https://doi.org/10.1109/ACCESS.2022.3165809
Shahid A. NF TON IOT V2 Full DataSet [dataset]. Kaggle. Available from: https://www.kaggle.com/datasets/shahidabbas76/nf-ton-iot-v2-full-dataset (Accessed on: 5 January 2026).
Pradhan S. Edge-IIoTset-dataset [dataset]. Kaggle. Available from: https://www.kaggle.com/datasets/sibasispradhan/edge-iiotset-dataset (Accessed on: 5 January 2026).
Cullerne Bown W. Sensitivity and specificity versus precision and recall, and related dilemmas. J Classif. 2024; 41(2): 402-426. https://doi.org/10.1007/s00357-024-09478-y
Talukder MA, Islam MM, Uddin MA, Hasan KF, Sharmin S, Alyami SA, et al. Machine learning-based network intrusion detection for big and imbalanced data using oversampling, stacking feature embedding and feature extraction. J Big Data. 2024; 11(1): 33. https://doi.org/10.1186/s40537-024-00886-w
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Ola Madi Mohammed Al Mari (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
