{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,23]],"date-time":"2026-06-23T12:29:00Z","timestamp":1782217740496,"version":"3.54.5"},"publisher-location":"New York, NY, USA","reference-count":44,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T00:00:00Z","timestamp":1604793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,11,8]]},"DOI":"10.1145\/3368089.3417063","type":"proceedings-article","created":{"date-parts":[[2020,11,10]],"date-time":"2020-11-10T21:08:44Z","timestamp":1605042524000},"page":"1240-1250","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":54,"title":["A first look at the integration of machine learning models in complex autonomous driving systems: a case study on Apollo"],"prefix":"10.1145","author":[{"given":"Zi","family":"Peng","sequence":"first","affiliation":[{"name":"Concordia University, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinqiu","family":"Yang","sequence":"additional","affiliation":[{"name":"Concordia University, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tse-Hsun (Peter)","family":"Chen","sequence":"additional","affiliation":[{"name":"Concordia University, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Ma","sequence":"additional","affiliation":[{"name":"Kyushu University, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,11,8]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.110"},{"key":"e_1_3_2_2_2_1","volume-title":"d.]. Apollo. https:\/\/apollo.auto\/. ([n. d.]). Last checked","year":"2020","unstructured":"Baidu. [n. d.]. Apollo. https:\/\/apollo.auto\/. ([n. d.]). Last checked May 2020 . Baidu. [n. d.]. Apollo. https:\/\/apollo.auto\/. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_3_1","unstructured":"Baidu. [n. d.]. Chinese internet giant Baidu ofers free trial robotaxi rides through search and map apps in Changsha. https:\/\/www.scmp.com\/tech\/apps-social\/ article\/3080712\/chinese-internet-giant-baidu-ofers-free-trial-robotaxi-rides. ([n. d.]). Last checked May 2020.  Baidu. [n. d.]. Chinese internet giant Baidu ofers free trial robotaxi rides through search and map apps in Changsha. https:\/\/www.scmp.com\/tech\/apps-social\/ article\/3080712\/chinese-internet-giant-baidu-ofers-free-trial-robotaxi-rides. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_4_1","volume-title":"d.]. CyberRT. https:\/\/cyber-rt.readthedocs.io\/en\/latest. ([n. d.]). Last checked","year":"2020","unstructured":"Baidu. [n. d.]. CyberRT. https:\/\/cyber-rt.readthedocs.io\/en\/latest. ([n. d.]). Last checked May 2020 . Baidu. [n. d.]. CyberRT. https:\/\/cyber-rt.readthedocs.io\/en\/latest. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_5_1","volume-title":"d.]. Cafe. https:\/\/cafe.berkeleyvision.org\/. ([n. d.]). Last checked","author":"BAIR.","year":"2020","unstructured":"BAIR. [n. d.]. Cafe. https:\/\/cafe.berkeleyvision.org\/. ([n. d.]). Last checked May 2020 . BAIR. [n. d.]. Cafe. https:\/\/cafe.berkeleyvision.org\/. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_6_1","unstructured":"BBC. 2020. AI 'outperforms' doctors diagnosing breast cancer. https:\/\/www.bbc. com\/news\/health-50857759  BBC. 2020. AI 'outperforms' doctors diagnosing breast cancer. https:\/\/www.bbc. com\/news\/health-50857759"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCOMSTD.2017.1700015"},{"key":"e_1_3_2_2_8_1","volume-title":"d.]. gflags. https:\/\/github.com\/gflags\/gflags. ([n. d.]). Last checked","year":"2020","unstructured":"gflags. [n. d.]. gflags. https:\/\/github.com\/gflags\/gflags. ([n. d.]). Last checked May 2020 . gflags. [n. d.]. gflags. https:\/\/github.com\/gflags\/gflags. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_9_1","volume-title":"d.]. Protocol Bufers. https:\/\/developers.google.com\/protocol-bufers. ([n. d.]). Last checked","year":"2020","unstructured":"Google. [n. d.]. Protocol Bufers. https:\/\/developers.google.com\/protocol-bufers. ([n. d.]). Last checked May 2020 . Google. [n. d.]. Protocol Bufers. https:\/\/developers.google.com\/protocol-bufers. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_10_1","volume-title":"A Survey of Deep Learning Techniques for Autonomous Driving. CoRR abs\/","author":"Grigorescu Sorin Mihai","year":"1910","unstructured":"Sorin Mihai Grigorescu , Bogdan Trasnea , Tiberiu T. Cocias , and Gigel Macesanu . 2019. A Survey of Deep Learning Techniques for Autonomous Driving. CoRR abs\/ 1910 .07738 ( 2019 ). arXiv: 1910.07738 http:\/\/arxiv.org\/abs\/ 1910.07738 Sorin Mihai Grigorescu, Bogdan Trasnea, Tiberiu T. Cocias, and Gigel Macesanu. 2019. A Survey of Deep Learning Techniques for Autonomous Driving. CoRR abs\/ 1910.07738 ( 2019 ). arXiv: 1910.07738 http:\/\/arxiv.org\/abs\/ 1910.07738"},{"key":"e_1_3_2_2_11_1","first-page":"810","volume-title":"Proceedings of the 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE '19)","author":"Guo Qianyu","year":"2019","unstructured":"Qianyu Guo , Sen Chen , Xiaofei Xie , Lei Ma , Qiang Hu , Hongtao Liu , Yang Liu , Jianjun Zhao , and Xiaohong Li . 2019 . An Empirical Study towards Characterizing Deep Learning Development and Deployment across Diferent Frameworks and Platforms . In Proceedings of the 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE '19) . 810 - 822 . Qianyu Guo, Sen Chen, Xiaofei Xie, Lei Ma, Qiang Hu, Hongtao Liu, Yang Liu, Jianjun Zhao, and Xiaohong Li. 2019. An Empirical Study towards Characterizing Deep Learning Development and Deployment across Diferent Frameworks and Platforms. In Proceedings of the 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE '19). 810-822."},{"key":"e_1_3_2_2_12_1","first-page":"85","volume-title":"Proceedings of the 13th International Conference on Software Testing, Validation and Verification (ICST '20)","author":"Haq F. U.","unstructured":"F. U. Haq , D. Shin , S. Nejati , and L. C. Briand . 2020. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study . In Proceedings of the 13th International Conference on Software Testing, Validation and Verification (ICST '20) . 85 - 95 . F. U. Haq, D. Shin, S. Nejati, and L. C. Briand. 2020. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. In Proceedings of the 13th International Conference on Software Testing, Validation and Verification (ICST '20). 85-95."},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.2205597"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"crossref","unstructured":"Sepp Hochreiter and J\u00fcrgen Schmidhuber. 1997. Long short-term memory. Neural computation 9 8 ( 1997 ) 1735-1780.  Sepp Hochreiter and J\u00fcrgen Schmidhuber. 1997. Long short-term memory. Neural computation 9 8 ( 1997 ) 1735-1780.","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_2_2_15_1","volume-title":"d.]. https:\/\/velodynelidar.com\/. https:\/\/ velodynelidar.com\/. ([n. d.]). Last checked","year":"2020","unstructured":"https:\/\/velodynelidar.com\/. [n. d.]. https:\/\/velodynelidar.com\/. https:\/\/ velodynelidar.com\/. ([n. d.]). Last checked May 2020 . https:\/\/velodynelidar.com\/. [n. d.]. https:\/\/velodynelidar.com\/. https:\/\/ velodynelidar.com\/. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_16_1","volume-title":"Proceedings of the 42nd International Conference on Software Engineering (ICSE '20)","author":"Joshua Garcia","year":"2020","unstructured":"Garcia Joshua , Feng Yang , Shen Junjie , Almanee Sumaya , Xia Yuan , and Chen QiAlfred . 2020 . A Comprehensive Study of Autonomous Vehicle Bugs . In Proceedings of the 42nd International Conference on Software Engineering (ICSE '20) . Garcia Joshua, Feng Yang, Shen Junjie, Almanee Sumaya, Xia Yuan, and Chen QiAlfred. 2020. A Comprehensive Study of Autonomous Vehicle Bugs. In Proceedings of the 42nd International Conference on Software Engineering (ICSE '20)."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"crossref","unstructured":"Rudolph Emil Kalman. 1960. A new approach to linear filtering and prediction problems. ( 1960 ).  Rudolph Emil Kalman. 1960. A new approach to linear filtering and prediction problems. ( 1960 ).","DOI":"10.1115\/1.3662552"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3065386"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"crossref","unstructured":"Harold W Kuhn. 1956. Variants of the Hungarian method for assignment problems. Naval Research Logistics Quarterly 3 4 ( 1956 ) 253-258.  Harold W Kuhn. 1956. Variants of the Hungarian method for assignment problems. Naval Research Logistics Quarterly 3 4 ( 1956 ) 253-258.","DOI":"10.1002\/nav.3800030404"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-68279-0_2"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"crossref","unstructured":"Geesara Kulathunga Aleksandr Buyval and Aleksandr Klimchik. 2019. MultiCamera Fusion in Apollo Software Distribution. IFAC-PapersOnLine 52 8 ( 2019 ) 49-54.  Geesara Kulathunga Aleksandr Buyval and Aleksandr Klimchik. 2019. MultiCamera Fusion in Apollo Software Distribution. IFAC-PapersOnLine 52 8 ( 2019 ) 49-54.","DOI":"10.1016\/j.ifacol.2019.08.047"},{"key":"e_1_3_2_2_23_1","unstructured":"lcov. [n. d.]. lcov. http:\/\/ltp.sourceforge.net\/coverage\/lcov.php. ([n. d.]). Last checked May 2020.  lcov. [n. d.]. lcov. http:\/\/ltp.sourceforge.net\/coverage\/lcov.php. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_24_1","volume-title":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI '18)","author":"Pan Xingang","year":"2018","unstructured":"Xingang Pan , Jianping Shi , Ping Luo , Xiaogang Wang , and Xiaoou Tang . 2018 . Spatial as Deep: Spatial CNN for Trafic Scene Understanding . In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI '18) , Sheila A. McIlraith and Kilian Q. Weinberger (Eds.). 7276-7283. Xingang Pan, Jianping Shi, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2018. Spatial as Deep: Spatial CNN for Trafic Scene Understanding. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI '18), Sheila A. McIlraith and Kilian Q. Weinberger (Eds.). 7276-7283."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_2_26_1","volume-title":"2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). 1027-1038","author":"Pham H. V.","unstructured":"H. V. Pham , T. Lutellier , W. Qi , and L. Tan . 2019. CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries . In 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). 1027-1038 . H. V. Pham, T. Lutellier, W. Qi, and L. Tan. 2019. CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries. In 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). 1027-1038."},{"key":"e_1_3_2_2_27_1","volume-title":"d.]. PyTorch. https:\/\/pytorch.org\/. ([n. d.]). Last checked","year":"2020","unstructured":"PyTorch. [n. d.]. PyTorch. https:\/\/pytorch.org\/. ([n. d.]). Last checked May 2020 . PyTorch. [n. d.]. PyTorch. https:\/\/pytorch.org\/. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.91"},{"key":"e_1_3_2_2_29_1","first-page":"6517","volume-title":"Stronger. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR '17)","author":"Redmon Joseph","year":"2017","unstructured":"Joseph Redmon and Ali Farhadi . 2017 . YOLO9000: Better, Faster , Stronger. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR '17) . IEEE Computer Society , 6517 - 6525 . Joseph Redmon and Ali Farhadi. 2017. YOLO9000: Better, Faster, Stronger. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR '17). IEEE Computer Society, 6517-6525."},{"key":"e_1_3_2_2_30_1","volume-title":"YOLOv3: An Incremental Improvement. CoRR abs\/","author":"Redmon Joseph","year":"1804","unstructured":"Joseph Redmon and Ali Farhadi . 2018. YOLOv3: An Incremental Improvement. CoRR abs\/ 1804 .02767 ( 2018 ). arXiv: 1804.02767 http:\/\/arxiv.org\/abs\/ 1804.02767 Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement. CoRR abs\/ 1804.02767 ( 2018 ). arXiv: 1804.02767 http:\/\/arxiv.org\/abs\/ 1804.02767"},{"key":"e_1_3_2_2_31_1","volume-title":"Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention (Lecture Notes in Computer Science), Nassir Navab, Joachim Hornegger, William M","author":"Ronneberger Olaf","unstructured":"Olaf Ronneberger , Philipp Fischer , and Thomas Brox . 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation . In Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention (Lecture Notes in Computer Science), Nassir Navab, Joachim Hornegger, William M . Wells III, and Alejandro F. Frangi (Eds.), Vol. 9351 . Springer , 234-241. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention (Lecture Notes in Computer Science), Nassir Navab, Joachim Hornegger, William M. Wells III, and Alejandro F. Frangi (Eds.), Vol. 9351. Springer, 234-241."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/2969442.2969519"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1414004.1414030"},{"key":"e_1_3_2_2_34_1","first-page":"3104","volume-title":"Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS '14)","author":"Sutskever Ilya","unstructured":"Ilya Sutskever , Oriol Vinyals , and Quoc V. Le . 2014. Sequence to Sequence Learning with Neural Networks . In Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS '14) . 3104 - 3112 . Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS '14). 3104-3112."},{"key":"e_1_3_2_2_35_1","volume-title":"An Empirical Study of Bugs in Machine Learning Systems. In 2012 IEEE 23rd International Symposium on Software Reliability Engineering. 271-280","author":"Thung F.","unstructured":"F. Thung , S. Wang , D. Lo , and L. Jiang . 2012 . An Empirical Study of Bugs in Machine Learning Systems. In 2012 IEEE 23rd International Symposium on Software Reliability Engineering. 271-280 . F. Thung, S. Wang, D. Lo, and L. Jiang. 2012. An Empirical Study of Bugs in Machine Learning Systems. In 2012 IEEE 23rd International Symposium on Software Reliability Engineering. 271-280."},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8461224"},{"key":"e_1_3_2_2_38_1","volume-title":"d.]. TTesla's Autopilot Was Involved in Another Deadly Car Crash. https:\/\/www.wired.com\/story\/tesla-autopilot-self-driving-crash-california\/. ([n. d.]). Last checked","year":"2020","unstructured":"Wired. [n. d.]. TTesla's Autopilot Was Involved in Another Deadly Car Crash. https:\/\/www.wired.com\/story\/tesla-autopilot-self-driving-crash-california\/. ([n. d.]). Last checked May 2020 . Wired. [n. d.]. TTesla's Autopilot Was Involved in Another Deadly Car Crash. https:\/\/www.wired.com\/story\/tesla-autopilot-self-driving-crash-california\/. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_39_1","unstructured":"Oliver Wyman. [n. d.]. THE TRUE VALUE OF AUTONOMOUS DRIVING. https:\/\/www.oliverwyman.com\/our-expertise\/insights\/2015\/jul\/automotivemanager-2015\/ customer\/the-true-value-of-autonomous-driving. html. ([n. d.]). Last checked May 2020.  Oliver Wyman. [n. d.]. THE TRUE VALUE OF AUTONOMOUS DRIVING. https:\/\/www.oliverwyman.com\/our-expertise\/insights\/2015\/jul\/automotivemanager-2015\/ customer\/the-true-value-of-autonomous-driving. html. ([n. d.]). Last checked May 2020."},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"crossref","unstructured":"J. M. Zhang M. Harman L. Ma and Y. Liu. 2020. Machine Learning Testing: Survey Landscapes and Horizons. IEEE Transactions on Software Engineering ( 2020 ) 1-1.  J. M. Zhang M. Harman L. Ma and Y. Liu. 2020. Machine Learning Testing: Survey Landscapes and Horizons. IEEE Transactions on Software Engineering ( 2020 ) 1-1.","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_2_2_42_1","first-page":"104","volume-title":"Proceedings of the 30th International Symposium on Software Reliability Engineering (ISSRE '19)","author":"Zhang T.","unstructured":"T. Zhang , C. Gao , L. Ma , M. Lyu , and M. Kim . 2019. An Empirical Study of Common Challenges in Developing Deep Learning Applications . In Proceedings of the 30th International Symposium on Software Reliability Engineering (ISSRE '19) . 104 - 115 . T. Zhang, C. Gao, L. Ma, M. Lyu, and M. Kim. 2019. An Empirical Study of Common Challenges in Developing Deep Learning Applications. In Proceedings of the 30th International Symposium on Software Reliability Engineering (ISSRE '19). 104-115."},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"},{"key":"e_1_3_2_2_44_1","volume-title":"Proceedings of the 42nd International Conference on Software Engineering (ICSE '20)","author":"Zhou Husheng","year":"2020","unstructured":"Husheng Zhou , Wei Li , Yuankun Zhu , Yuqun Zhang , Bei Yu , Lingming Zhang , and Cong Liu . 2020 . DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems . In Proceedings of the 42nd International Conference on Software Engineering (ICSE '20) . Husheng Zhou, Wei Li, Yuankun Zhu, Yuqun Zhang, Bei Yu, Lingming Zhang, and Cong Liu. 2020. DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems. In Proceedings of the 42nd International Conference on Software Engineering (ICSE '20)."}],"event":{"name":"ESEC\/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Virtual Event USA","acronym":"ESEC\/FSE '20","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3417063","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3368089.3417063","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:58Z","timestamp":1750197718000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3417063"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,8]]},"references-count":44,"alternative-id":["10.1145\/3368089.3417063","10.1145\/3368089"],"URL":"https:\/\/doi.org\/10.1145\/3368089.3417063","relation":{},"subject":[],"published":{"date-parts":[[2020,11,8]]},"assertion":[{"value":"2020-11-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}