{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T02:00:16Z","timestamp":1783562416168,"version":"3.55.0"},"reference-count":96,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T00:00:00Z","timestamp":1721865600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Computer Security"],"published-print":{"date-parts":[[2024,11,18]]},"abstract":"<jats:p>The development of intrusion detection systems (IDS) for the in-vehicle Controller Area Network (CAN) bus is one of the main efforts being taken to secure the in-vehicle network against various cyberattacks, which have the potential to cause vehicles to malfunction and result in dangerous accidents. These CAN IDS are evaluated in disparate experimental conditions that vary in terms of the workload used, the features used, the metrics reported, etc., which makes direct comparison difficult. Therefore, there have been several benchmarking frameworks and comparative studies designed to evaluate CAN IDS in similar experimental conditions to understand their relative performance and facilitate the selection of the best CAN IDS for implementation in automotive networks. This work provides a comprehensive survey of CAN IDS benchmarking frameworks and comparative studies in the current literature. A CAN IDS evaluation design space is also proposed in this work, which draws from the wider CAN IDS literature. This is not only expected to serve as a guide for designing CAN IDS evaluation experiments but is also used for categorising current benchmarking efforts. The surveyed works have been discussed on the basis of the five aspects in the design space \u2013 namely, IDS type, attack model, evaluation type, workload generation, and evaluation metrics \u2013 and recommendations for future work have been identified.<\/jats:p>","DOI":"10.3233\/jcs-230027","type":"journal-article","created":{"date-parts":[[2024,7,26]],"date-time":"2024-07-26T10:58:33Z","timestamp":1721991513000},"page":"477-507","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":16,"title":["Benchmarking frameworks and comparative studies of Controller Area Network (CAN) intrusion detection systems: A review"],"prefix":"10.1177","volume":"32","author":[{"given":"Shaila","family":"Sharmin","sequence":"first","affiliation":[{"name":"Kulliyyah of Information and Communication Technology, International Islamic University Malaysia, Selangor, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hafizah","family":"Mansor","sequence":"additional","affiliation":[{"name":"Kulliyyah of Information and Communication Technology, International Islamic University Malaysia, Selangor, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andi Fitriah","family":"Abdul Kadir","sequence":"additional","affiliation":[{"name":"Kulliyyah of Information and Communication Technology, International Islamic University Malaysia, Selangor, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Normaziah A.","family":"Aziz","sequence":"additional","affiliation":[{"name":"Kulliyyah of Information and Communication Technology, International Islamic University Malaysia, Selangor, Malaysia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2024,7,25]]},"reference":[{"key":"ref001","doi-asserted-by":"publisher","DOI":"10.1109\/CCST.2016.7815711"},{"key":"ref002","doi-asserted-by":"crossref","unstructured":"P.\u00a0Agbaje, A.\u00a0Anjum, A.\u00a0Mitra, G.\u00a0Bloom and H.\u00a0Olufowobi, A framework for consistent and repeatable controller area network IDS evaluation, in: NDSS Automotive and Autonomous Vehicle Security, (AutoSec) Workshop 2022, 2022.","DOI":"10.14722\/autosec.2022.23031"},{"key":"ref003","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2894183"},{"key":"ref004","doi-asserted-by":"publisher","DOI":"10.1145\/3431233"},{"key":"ref005","doi-asserted-by":"publisher","DOI":"10.1109\/ICIT52682.2021.9491770"},{"key":"ref006","doi-asserted-by":"publisher","DOI":"10.4236\/wet.2018.94007"},{"key":"ref007","doi-asserted-by":"publisher","DOI":"10.1109\/ICTC52510.2021.9621200"},{"key":"ref008","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2937576"},{"key":"ref009","doi-asserted-by":"publisher","DOI":"10.3390\/e22091044"},{"key":"ref010","doi-asserted-by":"publisher","DOI":"10.3390\/fi12070119"},{"key":"ref011","doi-asserted-by":"publisher","DOI":"10.1109\/ICRE.2000.855606"},{"key":"ref012","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-12786-2_6"},{"key":"ref013","doi-asserted-by":"publisher","DOI":"10.14722\/autosec.2021.23013"},{"key":"ref014","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3073057"},{"key":"ref015","unstructured":"R.N.\u00a0Charette, How Software Is Eating the Car, IEEE, 2021, https:\/\/spectrum.ieee.org\/software-eating-car."},{"key":"ref016","doi-asserted-by":"publisher","DOI":"10.1186\/s13040-021-00244-z"},{"key":"ref017","unstructured":"K.T.\u00a0Cho and K.G.\u00a0Shin, Fingerprinting electronic control units for vehicle intrusion detection, in: Proceedings of the 25th USENIX Conference on Security Symposium, SEC\u201916, USENIX Association, USA, 2016, pp.\u00a0911\u2013927. ISBN 978-1-931971-32-4."},{"key":"ref018","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3134001"},{"key":"ref019","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-89432-0_4"},{"key":"ref020","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2018.2812149"},{"key":"ref021","unstructured":"C.\u00a0Corbett, T.\u00a0Basic, T.\u00a0Lukaseder and F.\u00a0Kargl, A testing framework architecture for automotive intrusion detection systems, in: Automotive \u2013 Safety & Security 2017 \u2013 Sicherheit und Zuverl\u00e4ssigkeit f\u00fcr automobile Informationstechnik, P.\u00a0Dencker, H.\u00a0Klenk, H.B.\u00a0Keller and E.\u00a0Pl\u00f6dererder, eds, Gesellschaft f\u00fcr Informatik, Bonn, 2017, pp.\u00a089\u2013102."},{"key":"ref022","unstructured":"S.\u00a0Corrigan, Introduction to the Controller Area Network (CAN), Technical Report, Texas Instruments, 2016."},{"key":"ref023","unstructured":"T.\u00a0Costa, Ca\u00f1ones, Benchmarking framework for Intrusion Detection Systems in Controller Area Networks, Master\u2019s thesis, Politecnico di Milano and Universitat Politecnica de Catalunya, 2021. https:\/\/www.politesi.polimi.it\/handle\/10589\/176269."},{"key":"ref024","doi-asserted-by":"publisher","DOI":"10.1109\/PerComWorkshops48775.2020.9156250"},{"key":"ref025","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVE45908.2019.8965028"},{"key":"ref026","doi-asserted-by":"publisher","DOI":"10.4121\/uuid:b74b4928-C377-4585-9432-2004dfa20a5d"},{"key":"ref027","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpat.2020.102143"},{"key":"ref028","doi-asserted-by":"publisher","DOI":"10.1145\/3359789.3359834"},{"key":"ref029","doi-asserted-by":"publisher","DOI":"10.1109\/SC.Companion.2012.77"},{"key":"ref030","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-09357-9_4"},{"key":"ref031","doi-asserted-by":"publisher","DOI":"10.1109\/STA.2016.7952095"},{"key":"ref032","doi-asserted-by":"publisher","DOI":"10.4271\/2020-01-0721"},{"key":"ref033","doi-asserted-by":"publisher","DOI":"10.1016\/j.vehcom.2018.09.004"},{"key":"ref034","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2982544"},{"key":"ref035","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3029307"},{"key":"ref036","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2020.3025685"},{"key":"ref037","unstructured":"H.\u00a0Jadidbonab, A.\u00a0Tomlinson, H.N.\u00a0Nguyen, T.\u00a0Doan and S.A.\u00a0Shaikh, A real-time in-vehicle network testbed for machine learning-based IDS training and validation, in: Workshop on Artificial Intelligence and Cyber Security (AI-CyberSec 2021), CEUR Workshop Proceedings, 2021."},{"key":"ref038","doi-asserted-by":"publisher","DOI":"10.1109\/TNSE.2021.3059881"},{"key":"ref039","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2848106"},{"key":"ref040","doi-asserted-by":"publisher","DOI":"10.21227\/qvr7-n418"},{"key":"ref041","doi-asserted-by":"publisher","DOI":"10.3390\/electronics11071072"},{"key":"ref042","doi-asserted-by":"publisher","DOI":"10.1109\/FPL57034.2022.00070"},{"key":"ref043","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2021.107840"},{"key":"ref044","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2020.3012749"},{"key":"ref045","doi-asserted-by":"publisher","DOI":"10.1007\/s10586-022-03821-x"},{"key":"ref046","doi-asserted-by":"publisher","DOI":"10.1109\/PST.2017.00017"},{"key":"ref047","doi-asserted-by":"publisher","DOI":"10.1109\/TNSM.2020.3038991"},{"key":"ref048","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2017.7995934"},{"key":"ref049","doi-asserted-by":"publisher","DOI":"10.1109\/RTSI.2016.7740627"},{"key":"ref050","doi-asserted-by":"publisher","DOI":"10.1145\/2808691"},{"key":"ref051","unstructured":"C.\u00a0Miller and C.\u00a0Valasek, A Survey of Remote Automotive Attack Surfaces, in, Black, Hat USA, 2014."},{"key":"ref052","doi-asserted-by":"publisher","DOI":"10.1145\/3064814.3064816"},{"key":"ref053","doi-asserted-by":"publisher","DOI":"10.14722\/autosec.2022.23028"},{"key":"ref054","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3095962"},{"key":"ref055","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2011.5940552"},{"key":"ref056","unstructured":"F.\u00a0Nappi, A survey of intrusion detection systems for controller area networks and FPGA evaluation, Master\u2019s thesis, Politecnico Milano, 2022."},{"key":"ref057","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2023.103166"},{"key":"ref058","doi-asserted-by":"publisher","DOI":"10.1145\/1122445.1122456"},{"key":"ref059","doi-asserted-by":"publisher","DOI":"10.35741\/issn.0258-2724.56.5.14"},{"key":"ref060","doi-asserted-by":"publisher","DOI":"10.1145\/3309171.3309178"},{"key":"ref061","doi-asserted-by":"publisher","DOI":"10.1109\/TVT.2019.2961344"},{"key":"ref062","doi-asserted-by":"publisher","DOI":"10.3390\/math9060690"},{"key":"ref063","doi-asserted-by":"publisher","DOI":"10.1109\/ACMI53878.2021.9528201"},{"key":"ref064","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2022.3158055"},{"key":"ref065","doi-asserted-by":"publisher","DOI":"10.1145\/3570954"},{"key":"ref066","doi-asserted-by":"publisher","DOI":"10.3390\/s22176679"},{"key":"ref067","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-82199-9_49"},{"key":"ref068","doi-asserted-by":"publisher","DOI":"10.1109\/PDP.2017.67"},{"key":"ref069","doi-asserted-by":"publisher","DOI":"10.21227\/24m9-a446"},{"key":"ref070","doi-asserted-by":"publisher","DOI":"10.1109\/TrustCom50675.2020.00041"},{"key":"ref071","doi-asserted-by":"publisher","DOI":"10.1109\/PST.2018.8514157"},{"key":"ref072","doi-asserted-by":"publisher","DOI":"10.1109\/CRC50527.2021.9392627"},{"key":"ref073","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-19-0468-4_10"},{"key":"ref074","doi-asserted-by":"publisher","DOI":"10.1109\/ICOIN.2016.7427089"},{"key":"ref075","doi-asserted-by":"publisher","DOI":"10.1109\/VTCFall.2019.8891068"},{"key":"ref076","doi-asserted-by":"publisher","DOI":"10.23919\/AEIT.2017.8240550"},{"key":"ref077","unstructured":"D.\u00a0Stabili, F.\u00a0Pollicino and A.\u00a0Rota, A benchmark framework for CAN IDS, in: Proceedings of the Italian Conference on Cybersecurity (ITASEC 2021), 2021."},{"key":"ref078","unstructured":"S.\u00a0Stachowski, R.\u00a0Gaynier and D.J.\u00a0LeBlanc, An assessment method for automotive intrusion detection system performance, Technical Report, DOT HA 812 708, University of Michigan, Ann Arbor, Transportation Research Institute, 2019."},{"key":"ref079","doi-asserted-by":"publisher","DOI":"10.1201\/9781315154855-6"},{"key":"ref080","doi-asserted-by":"publisher","DOI":"10.1504\/IJES.2018.10010488"},{"key":"ref081","doi-asserted-by":"publisher","DOI":"10.1109\/ANTS50601.2020.9342791"},{"key":"ref082","doi-asserted-by":"publisher","DOI":"10.1109\/3ICT53449.2021.9581683"},{"key":"ref083","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99587-4_51"},{"key":"ref084","doi-asserted-by":"publisher","DOI":"10.1109\/WCICSS.2015.7420322"},{"key":"ref085","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2018.111145054"},{"key":"ref086","doi-asserted-by":"publisher","DOI":"10.1109\/ASP-DAC52403.2022.9712524"},{"key":"ref087","doi-asserted-by":"publisher","DOI":"10.1145\/3273946.3273950"},{"key":"ref088","doi-asserted-by":"publisher","DOI":"10.4271\/2016-01-0068"},{"key":"ref089","doi-asserted-by":"publisher","DOI":"10.1145\/3568160.3570226"},{"key":"ref090","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2012.14600"},{"key":"ref091","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2019.2908074"},{"key":"ref092","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2021.3090397"},{"key":"ref093","doi-asserted-by":"publisher","DOI":"10.1145\/3309171.3309179"},{"key":"ref094","doi-asserted-by":"publisher","DOI":"10.1016\/j.dib.2020.105149"},{"key":"ref095","doi-asserted-by":"publisher","DOI":"10.1016\/j.adhoc.2019.101974"},{"key":"ref096","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3145007"}],"container-title":["Journal of Computer Security"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JCS-230027","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/JCS-230027","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JCS-230027","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T20:45:50Z","timestamp":1777495550000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/JCS-230027"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,25]]},"references-count":96,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,11,18]]}},"alternative-id":["10.3233\/JCS-230027"],"URL":"https:\/\/doi.org\/10.3233\/jcs-230027","relation":{},"ISSN":["0926-227X","1875-8924"],"issn-type":[{"value":"0926-227X","type":"print"},{"value":"1875-8924","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,25]]}}}