{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T11:25:14Z","timestamp":1781522714696,"version":"3.54.1"},"reference-count":233,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2022,4,1]],"date-time":"2022-04-01T00:00:00Z","timestamp":1648771200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"\u201cBeatriz Galindo\u201d Spanish Program","award":["BEAGAL18\/00064"],"award-info":[{"award-number":["BEAGAL18\/00064"]}]},{"name":"DOGO4ML Spanish research project","award":["PID2020-117191RB-I00"],"award-info":[{"award-number":["PID2020-117191RB-I00"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2022,4,30]]},"abstract":"<jats:p>\n            AI-based systems are software systems with functionalities enabled by at least one AI component (e.g., for image-, speech-recognition, and autonomous driving). AI-based systems are becoming pervasive in society due to advances in AI. However, there is limited synthesized knowledge on\n            <jats:bold>Software Engineering (SE)<\/jats:bold>\n            approaches for building, operating, and maintaining AI-based systems. To collect and analyze state-of-the-art knowledge about SE for AI-based systems, we conducted a systematic mapping study. We considered 248 studies published between January 2010 and March 2020. SE for AI-based systems is an emerging research area, where more than 2\/3 of the studies have been published since 2018. The most studied properties of AI-based systems are dependability and safety. We identified multiple SE approaches for AI-based systems, which we classified according to the SWEBOK areas. Studies related to software testing and software quality are very prevalent, while areas like software maintenance seem neglected. Data-related issues are the most recurrent challenges. Our results are valuable for: researchers, to quickly understand the state-of-the-art and learn which topics need more research; practitioners, to learn about the approaches and challenges that SE entails for AI-based systems; and, educators, to bridge the gap among SE and AI in their curricula.\n          <\/jats:p>","DOI":"10.1145\/3487043","type":"journal-article","created":{"date-parts":[[2022,4,1]],"date-time":"2022-04-01T09:53:52Z","timestamp":1648806832000},"page":"1-59","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":249,"title":["Software Engineering for AI-Based Systems: A Survey"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9928-133X","authenticated-orcid":false,"given":"Silverio","family":"Mart\u00ednez-Fern\u00e1ndez","sequence":"first","affiliation":[{"name":"Universitat Polit\u00e8cnica de Catalunya - BarcelonaTech, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5788-0991","authenticated-orcid":false,"given":"Justus","family":"Bogner","sequence":"additional","affiliation":[{"name":"University of Stuttgart, Institute of Software Engineering, Stuttgart, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9733-8830","authenticated-orcid":false,"given":"Xavier","family":"Franch","sequence":"additional","affiliation":[{"name":"Universitat Polit\u00e8cnica de Catalunya - BarcelonaTech, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1928-7024","authenticated-orcid":false,"given":"Marc","family":"Oriol","sequence":"additional","affiliation":[{"name":"Universitat Polit\u00e8cnica de Catalunya - BarcelonaTech, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7696-0046","authenticated-orcid":false,"given":"Julien","family":"Siebert","sequence":"additional","affiliation":[{"name":"FraunhoferInstitute for Experimental Software Engineering IESE, Kaiserslautern, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adam","family":"Trendowicz","sequence":"additional","affiliation":[{"name":"FraunhoferInstitute for Experimental Software Engineering IESE, Kaiserslautern, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3563-8253","authenticated-orcid":false,"given":"Anna Maria","family":"Vollmer","sequence":"additional","affiliation":[{"name":"FraunhoferInstitute for Experimental Software Engineering IESE, Kaiserslautern, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5256-8429","authenticated-orcid":false,"given":"Stefan","family":"Wagner","sequence":"additional","affiliation":[{"name":"University of Stuttgart, Institute of Software Engineering, Stuttgart, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,4]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180160"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISORC.2018.00025"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338937"},{"key":"e_1_3_3_5_2","article-title":"Characterizing machine learning process: A maturity framework","author":"Akkiraju Rama","year":"2018","unstructured":"Rama Akkiraju, Vibha Sinha, Anbang Xu, Jalal Mahmud, Pritam Gundecha, Zhe Liu, Xiaotong Liu, and John Schumacher. 2018. Characterizing machine learning process: A maturity framework. arXiv (2018).","journal-title":"arXiv"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-35333-9_14"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"issue":"2003","key":"e_1_3_3_8_2","first-page":"1","article-title":"Concrete problems in AI safety","volume":"277","author":"Amodei Dario","year":"2016","unstructured":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man\u00e9. 2016. Concrete problems in AI safety. arXiv 277, 2003 (2016), 1\u201329. arxiv:1606.06565http:\/\/arxiv.org\/abs\/1606.06565","journal-title":"arXiv"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.10.006"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3194085.3194091"},{"key":"e_1_3_3_11_2","first-page":"1","article-title":"Graceful degradation of decision and control responsibility for autonomous systems based on dependability cages","author":"Aniculaesei Adina","year":"2019","unstructured":"Adina Aniculaesei, Jorg Grieser, Andreas Rausch, Karina Rehfeldt, and Tim Warnecke. 2019. Graceful degradation of decision and control responsibility for autonomous systems based on dependability cages. 5th International Symposium on Future Active Safety Technology toward Zero Accidents (FAST-zero\u201919)September (2019), 1\u20136.","journal-title":"5th International Symposium on Future Active Safety Technology toward Zero Accidents (FAST-zero\u201919)"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3088342"},{"key":"e_1_3_3_13_2","article-title":"FactSheets: Increasing trust in AI services through supplier\u2019s declarations of conformity","author":"Arnold M.","year":"2018","unstructured":"M. Arnold, R. K. E. Bellamy, M. Hind, S. Houde, S. Mehta, A. Mojsilovi\u0107, R. Nair, K. Natesan Ramamurthy, D. Reimer, A. Olteanu, D. Piorkowski, J. Tsay, and K. R. Varshney. 2018. FactSheets: Increasing trust in AI services through supplier\u2019s declarations of conformity. arXiv (2018). arxiv:1808.07261.","journal-title":"arXiv"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA.2018.00018"},{"key":"e_1_3_3_15_2","article-title":"Infrastructure for usable machine learning: The Stanford DAWN project","author":"Bailis Peter","year":"2017","unstructured":"Peter Bailis, Kunle Olukotun, Christopher R\u00e9, and Matei Zaharia. 2017. Infrastructure for usable machine learning: The Stanford DAWN project. arXiv (2017). arxiv:1705.07538.","journal-title":"arXiv"},{"key":"e_1_3_3_16_2","article-title":"Requirements assurance in machine learning","volume":"2301","author":"Banks Alec","year":"2019","unstructured":"Alec Banks and Rob Ashmore. 2019. Requirements assurance in machine learning. CEUR Workshop Proceedings 2301 (2019).","journal-title":"CEUR Workshop Proceedings"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-97301-2_4"},{"key":"e_1_3_3_18_2","first-page":"528","volume-title":"Encyclopedia of Software Engineering","author":"Basili V.","year":"1994","unstructured":"V. Basili, G. Caldiera, and H. D. Rombach. 1994. The goal question metric approach. In Encyclopedia of Software Engineering, Vol. 2. John Wiley & Sons, 528\u2013532."},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098021"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.106225"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/REW.2019.00051"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330744"},{"key":"e_1_3_3_23_2","article-title":"Towards corner case detection for autonomous driving","author":"Bolte Jan Aike","year":"2019","unstructured":"Jan Aike Bolte, Andreas B\u00e4r, Daniel Lipinski, and Tim Fingscheidt. 2019. Towards corner case detection for autonomous driving. arXivIv (2019).","journal-title":"arXiv"},{"key":"e_1_3_3_24_2","article-title":"Safely entering the deep: A review of verification and validation for machine learning and a challenge elicitation in the automotive industry","author":"Borg Markus","year":"2018","unstructured":"Markus Borg, Cristofer Englund, Krzysztof Wnuk, Boris Duran, Christoffer Levandowski, Shenjian Gao, Yanwen Tan, Henrik Kaijser, Henrik L\u00f6nn, and Jonas T\u00f6rnqvist. 2018. Safely entering the deep: A review of verification and validation for machine learning and a challenge elicitation in the automotive industry. arXiv preprint arXiv:1812.05389 (2018).","journal-title":"arXiv preprint arXiv:1812.05389"},{"key":"e_1_3_3_25_2","article-title":"Engineering AI systems: A research agenda","author":"Bosch Jan","year":"2020","unstructured":"Jan Bosch, Ivica Crnkovic, and Helena Holmstr\u00f6m Olsson. 2020. Engineering AI systems: A research agenda. arXiv (2020). arxiv:2001.07522.","journal-title":"arXiv"},{"key":"e_1_3_3_26_2","article-title":"SWEBOK Version 3.0","author":"Bourque Pierre","year":"2014","unstructured":"Pierre Bourque and E. Richard. 2014. SWEBOK Version 3.0. IEEE, ISBN-10: 0-7695-5166-1 (2014).","journal-title":"IEEE, ISBN-10: 0-7695-5166-1"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-99927-2_3"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110542"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2017.8258038"},{"key":"e_1_3_3_30_2","first-page":"1","article-title":"Data validation for machine learning","author":"Breck Eric","year":"2019","unstructured":"Eric Breck, Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, and Martin Zinkevich. 2019. Data validation for machine learning. SysML (2019), 1\u201314.","journal-title":"SysML"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2006.07.009"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2017.154"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66284-8_1"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITest.2019.000-6"},{"key":"e_1_3_3_35_2","volume-title":"Proceedings of the Reliable Machine Learning in the Wild - NIPS 2016 Workshop (2016)","author":"Cai Shanqing","year":"2016","unstructured":"Shanqing Cai, Eric Breck, Eric Nielsen, Michael Salib, and D. Sculley. 2016. TensorFlow debugger: Debugging dataflow graphs for machine learning. In Proceedings of the Reliable Machine Learning in the Wild - NIPS 2016 Workshop (2016). https:\/\/storage.googleapis.com\/pub-tools-public-publication-data\/pdf\/45789.pdf."},{"key":"e_1_3_3_36_2","first-page":"1","article-title":"Debugging machine learning tasks","author":"Chakarov Aleksandar","year":"2016","unstructured":"Aleksandar Chakarov, Aditya Nori, Sriram Rajamani, Shayak Sen, and Deepak Vijaykeerthy. 2016. Debugging machine learning tasks. arXiv (2016), 1\u201329. arxiv:1603.07292http:\/\/arxiv.org\/abs\/1603.07292.","journal-title":"arXiv"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITNG.2010.149"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2018.8500679"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-03592-1_16"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1147\/JRD.2019.2915062"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30809-4_31"},{"key":"e_1_3_3_42_2","volume-title":"2021 Proceedings of 24th IberoAmerican Conference on Software Engineering (CIbSE 2021), ESELAW track","author":"Costal Dolors","year":"2021","unstructured":"Dolors Costal, Carles Farr\u00e9, Xavier Franch, and Carme Quer. 2021. How tertiary studies perform quality assessment of secondary studies in software engineering. In 2021 Proceedings of 24th IberoAmerican Conference on Software Engineering (CIbSE 2021), ESELAW track."},{"key":"e_1_3_3_43_2","first-page":"613","article-title":"Clipper: A low-latency online prediction serving system","author":"Crankshaw Daniel","year":"2017","unstructured":"Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A low-latency online prediction serving system. Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2017 (2017), 613\u2013627. arxiv:1612.03079.","journal-title":"Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2017"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2011.36"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2019.8870157"},{"key":"e_1_3_3_46_2","first-page":"17","article-title":"Aloha: A machine learning framework for engineers","author":"Deak Ryan M.","year":"2018","unstructured":"Ryan M. Deak and Jonathan H. Morra. 2018. Aloha: A machine learning framework for engineers. Conference on Systems and Machine Learning (MLSys) (2018), 17\u201319. https:\/\/www.sysml.cc\/doc\/13.pdf.","journal-title":"Conference on Systems and Machine Learning (MLSys)"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2017.2762725"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2019.00027"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-25540-4_25"},{"key":"e_1_3_3_50_2","first-page":"3","article-title":"Semantic adversarial deep learning","volume":"2","author":"Dreossi Tommaso","year":"2018","unstructured":"Tommaso Dreossi, Somesh Jha, and Sanjit A. Seshia. 2018. Semantic adversarial deep learning. arXiv 2 (2018), 3\u201326.","journal-title":"arXiv"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00102"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213858"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009820201126"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-16722-6_10"},{"key":"e_1_3_3_56_2","article-title":"Robust physical-world attacks on deep learning models","author":"Eykholt Kevin","year":"2017","unstructured":"Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2017. Robust physical-world attacks on deep learning models. arXiv (2017). arxiv:1707.08945http:\/\/arxiv.org\/abs\/1707.08945.","journal-title":"arXiv"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66266-4_9"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1978965"},{"key":"e_1_3_3_60_2","article-title":"Beyond the technical challenges for deploying machine learning solutions in a software company","author":"Flaounas Ilias","year":"2017","unstructured":"Ilias Flaounas. 2017. Beyond the technical challenges for deploying machine learning solutions in a software company. arXiv (2017). arxiv:1708.02363.","journal-title":"arXiv"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA.2019.00058"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.07.007"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC45102.2020.9294368"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330566"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2019.00015"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2964386"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-99130-6_4"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377812.3390793"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN-W.2018.00074"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3307630.3342419"},{"key":"e_1_3_3_71_2","article-title":"Counterexample-guided synthesis of perception models and control","author":"Ghosh Shromona","year":"2019","unstructured":"Shromona Ghosh, Hadi Ravanbakhsh, and Sanjit A. Seshia. 2019. Counterexample-guided synthesis of perception models and control. arXiv (2019). arxiv:1911.01523.","journal-title":"arXiv"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111031"},{"key":"e_1_3_3_73_2","article-title":"DeepSafe: A data-driven approach for checking adversarial robustness in neural networks","author":"Gopinath Divya","year":"2017","unstructured":"Divya Gopinath, Guy Katz, Corina S. Pasareanu, and Clark Barrett. 2017. DeepSafe: A data-driven approach for checking adversarial robustness in neural networks. arXiv (2017). arxiv:1710.00486.","journal-title":"arXiv"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00080"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/SYSCON.2018.8369576"},{"key":"e_1_3_3_76_2","first-page":"1927","article-title":"Applying deep learning to Airbnb search","author":"Haldar Malay","year":"2018","unstructured":"Malay Haldar, Mustafa Abdool, Prashant Ramanathan, Tao Xu, Shulin Yang, Huizhong Duan, Qing Zhang, Nick Barrow-Williams, Bradley C. Turnbull, Brendan M. Collins, and Thomas Legrand. 2018. Applying deep learning to Airbnb search. arXiv (2018), 1927\u20131935.","journal-title":"arXiv"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313151.3313166"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2019.8917326"},{"key":"e_1_3_3_79_2","first-page":"123","article-title":"Ethical challenges in data-driven dialogue systems","author":"Henderson Peter","year":"2017","unstructured":"Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2017. Ethical challenges in data-driven dialogue systems. arXiv (2017), 123\u2013129.","journal-title":"arXiv"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3194085.3194090"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2016.7739680"},{"key":"e_1_3_3_82_2","first-page":"1","article-title":"Improving fairness in machine learning systems: What do industry practitioners need?","author":"Holstein Kenneth","year":"2018","unstructured":"Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daum\u00e9, Miroslav Dud\u00edk, and Hanna Wallach. 2018. Improving fairness in machine learning systems: What do industry practitioners need? arXiv (2018), 1\u201316.","journal-title":"arXiv"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2019.00050"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.23940\/ijpe.18.06.p18.12751282"},{"key":"e_1_3_3_85_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-63387-9_1"},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/IC2E.2019.00025"},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/IRC.2019.00059"},{"key":"e_1_3_3_88_2","unstructured":"International Organization For Standardization. 2011. ISO\/IEC 25010 - Systems and Software Engineering - Systems and Software Quality Requirements and Evaluation (SQuaRE) - System and Software Quality Models. 25 pages. http:\/\/www.iso.org\/iso\/iso_catalogue\/catalogue_tc\/catalogue_detail.htm?csnumber=35733."},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-00847-5_39"},{"key":"e_1_3_3_90_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-99229-7_2"},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/CESSER-IP.2019.00009"},{"key":"e_1_3_3_92_2","first-page":"510","article-title":"A comprehensive study on deep learning bug characteristics","author":"Islam Md Johirul","year":"2019","unstructured":"Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. 2019. A comprehensive study on deep learning bug characteristics. arXiv (2019), 510\u2013520.","journal-title":"arXiv"},{"key":"e_1_3_3_93_2","article-title":"What do developers ask about ML libraries? A large-scale study using stack overflow","author":"Islam Md Johirul","year":"2019","unstructured":"Md Johirul Islam, Hoan Anh Nguyen, Rangeet Pan, and Hridesh Rajan. 2019. What do developers ask about ML libraries? A large-scale study using stack overflow. arXivMl (2019). arxiv:1906.11940.","journal-title":"arXiv"},{"key":"e_1_3_3_94_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-010-9146-4"},{"key":"e_1_3_3_95_2","first-page":"10","volume-title":"Proceedings of the 10th European Congress on Embedded Real Time Systems (ERTS)","author":"Jenn Eric","year":"2020","unstructured":"Eric Jenn, Alexandre Albore, Franck Mamalet, Gr\u00e9gory Flandin, Christophe Gabreau, Herv\u00e9 Delseny, Adrien Gauffriau, Hugues Bonnin, Lucian Alecu, J\u00e9r\u00e9my Pirard, Baptiste Lefevre, Jean-Marc Gabriel, Cyril Cappi, Laurent Gard\u00e8s, Sylvaine Picard, Gilles Dulon, Brice Beltran, Jean-Christophe Bianic, Mathieu Damour, Kevin Delmas, and Claire Pagetti. 2020. Identifying challenges to the certification of machine learning for safety critical systems. In Proceedings of the 10th European Congress on Embedded Real Time Systems (ERTS). 10."},{"key":"e_1_3_3_96_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASEW.2019.00025"},{"key":"e_1_3_3_97_2","first-page":"349","article-title":"Model-reuse attacks on deep learning systems","author":"Ji Yujie","year":"2018","unstructured":"Yujie Ji, Xinyang Zhang, Shouling Ji, Xiapu Luo, and Ting Wang. 2018. Model-reuse attacks on deep learning systems. arXiv (2018), 349\u2013363.","journal-title":"arXiv"},{"key":"e_1_3_3_98_2","doi-asserted-by":"publisher","DOI":"10.23940\/ijpe.20.01.p8.6777"},{"key":"e_1_3_3_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICVES.2017.7991928"},{"key":"e_1_3_3_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173748"},{"key":"e_1_3_3_101_2","doi-asserted-by":"publisher","DOI":"10.1109\/BigDataCongress.2018.00013"},{"key":"e_1_3_3_102_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2018.3571224"},{"key":"e_1_3_3_103_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2754374"},{"issue":"2004","key":"e_1_3_3_104_2","first-page":"1","article-title":"Procedures for performing systematic reviews","volume":"33","author":"Kitchenham Barbara","year":"2004","unstructured":"Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele, UK, Keele University 33, 2004 (2004), 1\u201326.","journal-title":"Keele, UK, Keele University"},{"key":"e_1_3_3_105_2","unstructured":"Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. Keele University and University of Durham."},{"key":"e_1_3_3_106_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSREW.2018.00-20"},{"key":"e_1_3_3_107_2","doi-asserted-by":"publisher","DOI":"10.4271\/2016-01-0128"},{"key":"e_1_3_3_108_2","doi-asserted-by":"publisher","DOI":"10.4271\/2018-01-1071"},{"key":"e_1_3_3_109_2","first-page":"4178","article-title":"Efficient autonomy validation in simulation with adaptive stress testing","author":"Koren Mark","year":"2019","unstructured":"Mark Koren and Mykel J. Kochenderfer. 2019. Efficient autonomy validation in simulation with adaptive stress testing. arXiv (2019), 4178\u20134183.","journal-title":"arXiv"},{"key":"e_1_3_3_110_2","doi-asserted-by":"publisher","DOI":"10.5220\/0007245800130024"},{"key":"e_1_3_3_111_2","article-title":"On the interplay between requirements, engineering, and artificial intelligence","volume":"2584","author":"Kostova Blagovesta","year":"2020","unstructured":"Blagovesta Kostova, Seda G\u00fcrses, and Alain Wegmann. 2020. On the interplay between requirements, engineering, and artificial intelligence. CEUR Workshop Proceedings 2584 (2020).","journal-title":"CEUR Workshop Proceedings"},{"key":"e_1_3_3_112_2","doi-asserted-by":"publisher","DOI":"10.24251\/HICSS.2019.630"},{"key":"e_1_3_3_113_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9492-y"},{"key":"e_1_3_3_114_2","doi-asserted-by":"publisher","DOI":"10.1145\/2678025.2701399"},{"key":"e_1_3_3_115_2","article-title":"Trustworthy AI in the age of pervasive computing and big data","author":"Kumar Abhishek","year":"2020","unstructured":"Abhishek Kumar, Tristan Braud, Sasu Tarkoma, and Pan Hui. 2020. Trustworthy AI in the age of pervasive computing and big data. arXiv (2020). arxiv:2002.05657.","journal-title":"arXiv"},{"key":"e_1_3_3_116_2","doi-asserted-by":"publisher","DOI":"10.3233\/IDT-190160"},{"key":"e_1_3_3_117_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSREW.2019.00035"},{"key":"e_1_3_3_118_2","article-title":"Open Problems in Engineering Machine Learning Systems and the Quality Model","author":"Kuwajima Hiroshi","year":"2019","unstructured":"Hiroshi Kuwajima, Hirotoshi Yasuoka, and Toshihiro Nakae. 2019. Open Problems in Engineering Machine Learning Systems and the Quality Model. arXiv (2019). arxiv:1904.00001v1.","journal-title":"arXiv"},{"key":"e_1_3_3_119_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-020-05872-w"},{"key":"e_1_3_3_120_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377814.3381714"},{"key":"e_1_3_3_121_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEST.2018.8624723"},{"key":"e_1_3_3_122_2","first-page":"11","article-title":"Learning with safety requirements: State of the art and open questions","volume":"1745","author":"Leofante Francesco","year":"2016","unstructured":"Francesco Leofante, Luca Pulina, and Armando Tacchella. 2016. Learning with safety requirements: State of the art and open questions. CEUR Workshop Proceedings 1745 (2016), 11\u201325.","journal-title":"CEUR Workshop Proceedings"},{"key":"e_1_3_3_123_2","doi-asserted-by":"publisher","DOI":"10.1145\/3297280.3297411"},{"key":"e_1_3_3_124_2","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3054202"},{"key":"e_1_3_3_125_2","article-title":"Machine learning model development from a software engineering perspective: A systematic literature review","author":"Lorenzoni Giuliano","year":"2021","unstructured":"Giuliano Lorenzoni, Paulo Alencar, Nathalia Nascimento, and Donald Cowan. 2021. Machine learning model development from a software engineering perspective: A systematic literature review. arXiv preprint arXiv:2102.07574 (2021).","journal-title":"arXiv preprint arXiv:2102.07574"},{"key":"e_1_3_3_126_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-19034-7_14"},{"key":"e_1_3_3_127_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106368"},{"key":"e_1_3_3_128_2","first-page":"614","article-title":"Combinatorial testing for deep learning systems","author":"Ma Lei","year":"2018","unstructured":"Lei Ma, Fuyuan Zhang, Minhui Xue, Bo Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. Combinatorial testing for deep learning systems. arXiv (2018), 614\u2013618. arxiv:1806.07723.","journal-title":"arXiv"},{"key":"e_1_3_3_129_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236082"},{"key":"e_1_3_3_130_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN-W.2019.00017"},{"key":"e_1_3_3_131_2","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC47002.2019.00058"},{"key":"e_1_3_3_132_2","article-title":"Paracosm: A language and tool for testing autonomous driving systems","author":"Majumdar Rupak","year":"2019","unstructured":"Rupak Majumdar, Aman Mathur, Marcus Pirron, Laura Stegner, and Damien Zufferey. 2019. Paracosm: A language and tool for testing autonomous driving systems. arXiv (2019). arxiv:1902.01084.","journal-title":"arXiv"},{"key":"e_1_3_3_133_2","doi-asserted-by":"publisher","DOI":"10.1145\/3195555.3195558"},{"key":"e_1_3_3_134_2","article-title":"Research directions for developing and operating artificial intelligence models in trustworthy autonomous systems","author":"Mart\u00ednez-Fern\u00e1ndez Silverio","year":"2020","unstructured":"Silverio Mart\u00ednez-Fern\u00e1ndez, Xavier Franch, Andreas Jedlitschka, Marc Oriol, and Adam Trendowicz. 2020. Research directions for developing and operating artificial intelligence models in trustworthy autonomous systems. arXiv (2020). arxiv:2003.05434.","journal-title":"arXiv"},{"key":"e_1_3_3_135_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2018.00061"},{"key":"e_1_3_3_136_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-33742-1_15"},{"key":"e_1_3_3_137_2","article-title":"Towards a framework for safety assurance of autonomous systems","volume":"2419","author":"McDermid John","year":"2019","unstructured":"John McDermid, Yan Jia, and Ibrahim Habli. 2019. Towards a framework for safety assurance of autonomous systems. CEUR Workshop Proceedings 2419 (2019).","journal-title":"CEUR Workshop Proceedings"},{"key":"e_1_3_3_138_2","unstructured":"Meenu Mary John Helena Holmstr\u00f6m Olsson and Jan Bosch. [n.d.]. Architecting AI deployment: A systematic review of state-of-the-art and state-of-practice literature."},{"key":"e_1_3_3_139_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2019.2954841"},{"key":"e_1_3_3_140_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN-W.2017.14"},{"key":"e_1_3_3_141_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2986259"},{"key":"e_1_3_3_142_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106294"},{"key":"e_1_3_3_143_2","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA.2019.00030"},{"key":"e_1_3_3_144_2","doi-asserted-by":"publisher","DOI":"10.1109\/GCCE.2018.8574766"},{"key":"e_1_3_3_145_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-13651-2_2"},{"key":"e_1_3_3_146_2","doi-asserted-by":"publisher","DOI":"10.1109\/TAAI48200.2019.8959916"},{"key":"e_1_3_3_147_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-21290-2_39"},{"key":"e_1_3_3_148_2","article-title":"Software engineering for artificial intelligence and machine learning software: A systematic literature review","author":"Nascimento Elizamary","year":"2020","unstructured":"Elizamary Nascimento, Anh Nguyen-Duc, Ingrid Sundb\u00f8, and Tayana Conte. 2020. Software engineering for artificial intelligence and machine learning software: A systematic literature review. arXiv preprint arXiv:2011.03751 (2020).","journal-title":"arXiv preprint arXiv:2011.03751"},{"key":"e_1_3_3_149_2","volume-title":"Software Engineering: Report on a Conference Sponsored by the NATO Science Committee, Garmisch, Germany, 7th to 11th October 1968","author":"Naur Peter","year":"1969","unstructured":"Peter Naur, Brian Randell, Friedrich Ludwig Bauer, and NATO Science Committee. (Eds.). 1969. Software Engineering: Report on a Conference Sponsored by the NATO Science Committee, Garmisch, Germany, 7th to 11th October 1968. Scientific Affairs Division, NATO."},{"key":"e_1_3_3_150_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2018.00060"},{"key":"e_1_3_3_151_2","first-page":"126","article-title":"Towards accountable AI: Hybrid human-machine analyses for characterizing system failure","author":"Nushi Besmira","year":"2018","unstructured":"Besmira Nushi, Ece Kamar, and Eric Horvitz. 2018. Towards accountable AI: Hybrid human-machine analyses for characterizing system failure. arXivHcomp (2018), 126\u2013135. arxiv:1809.07424.","journal-title":"arXiv"},{"key":"e_1_3_3_152_2","article-title":"Tensorfuzz: Debugging neural networks with coverage-guided fuzzing","author":"Odena Augustus","year":"2018","unstructured":"Augustus Odena and Ian Goodfellow. 2018. Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv (2018).","journal-title":"arXiv"},{"key":"e_1_3_3_153_2","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2014.76"},{"key":"e_1_3_3_154_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2020.2993662"},{"key":"e_1_3_3_155_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00130012"},{"key":"e_1_3_3_156_2","doi-asserted-by":"publisher","DOI":"10.1145\/1866218.1866222"},{"key":"e_1_3_3_157_2","doi-asserted-by":"publisher","DOI":"10.18293\/SEKE2019-094"},{"key":"e_1_3_3_158_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.106241"},{"key":"e_1_3_3_159_2","doi-asserted-by":"publisher","DOI":"10.14236\/ewic\/EASE2008.8"},{"key":"e_1_3_3_160_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.03.007"},{"key":"e_1_3_3_161_2","first-page":"243","article-title":"An abstraction-refinement approach to verification of artificial neural networks","volume":"616","author":"Pulina Luca","year":"2010","unstructured":"Luca Pulina and Armando Tacchella. 2010. An abstraction-refinement approach to verification of artificial neural networks. CEUR Workshop Proceedings 616 (2010), 243\u2013257.","journal-title":"CEUR Workshop Proceedings"},{"key":"e_1_3_3_162_2","doi-asserted-by":"publisher","DOI":"10.1109\/REW.2019.00049"},{"key":"e_1_3_3_163_2","first-page":"1","article-title":"Machine learning software engineering in practice: An industrial case study","author":"Rahman Saidur","year":"2019","unstructured":"Saidur Rahman, Emilio River, Foutse Khomh, Yann Gal Guhneuc, and Bernd Lehnert. 2019. Machine learning software engineering in practice: An industrial case study. arXiv (2019), 1\u201321. arxiv:1906.07154.","journal-title":"arXiv"},{"key":"e_1_3_3_164_2","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372873"},{"key":"e_1_3_3_165_2","unstructured":"Paul Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de Fran\u00e7a Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Mart\u00ednez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarim\u00e4ki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2021. Empirical Standards for Software Engineering Research. arxiv:2010.03525 [cs.SE]."},{"key":"e_1_3_3_166_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_3_167_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09881-0"},{"key":"e_1_3_3_168_2","doi-asserted-by":"publisher","DOI":"10.24193\/subbi.2019.1.05"},{"key":"e_1_3_3_169_2","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2018.00016"},{"key":"e_1_3_3_170_2","volume-title":"Artificial Intelligence: A Modern Approach (Fourth edition)","author":"Russell Stuart J.","year":"2021","unstructured":"Stuart J. Russell and Peter Norvig. 2021. Artificial Intelligence: A Modern Approach (Fourth edition). Pearson, Hoboken."},{"key":"e_1_3_3_171_2","article-title":"Using machine learning safely in automotive software: An assessment and adaption of software process requirements in ISO 26262","author":"Salay Rick","year":"2018","unstructured":"Rick Salay and Krzysztof Czarnecki. 2018. Using machine learning safely in automotive software: An assessment and adaption of software process requirements in ISO 26262. arXiv (2018). arxiv:1808.01614.","journal-title":"arXiv"},{"key":"e_1_3_3_172_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-26250-1_23"},{"key":"e_1_3_3_173_2","article-title":"An analysis of ISO 26262: Using machine learning safely in automotive software","author":"Salay Rick","year":"2017","unstructured":"Rick Salay, Rodrigo Queiroz, and Krzysztof Czarnecki. 2017. An analysis of ISO 26262: Using machine learning safely in automotive software. arXiv (2017). arxiv:1709.02435.","journal-title":"arXiv"},{"key":"e_1_3_3_174_2","first-page":"1","article-title":"Engineering reliable deep learning systems","volume":"3","author":"Santhanam P.","year":"2019","unstructured":"P. Santhanam, Eitan Farchi, and Victor Pankratius. 2019. Engineering reliable deep learning systems. arXiv 3 (2019), 1\u20138. arxiv:1910.12582.","journal-title":"arXiv"},{"key":"e_1_3_3_175_2","doi-asserted-by":"publisher","DOI":"10.1109\/DASC43569.2019.9081649"},{"key":"e_1_3_3_176_2","article-title":"DeepSaucer: Unified environment for verifying deep neural networks","author":"Sato Naoto","year":"2018","unstructured":"Naoto Sato, Hironobu Kuruma, Masanori Kaneko, Yuichiroh Nakagawa, Hideto Ogawa, Thai Son Hoang, and Michael Butler. 2018. DeepSaucer: Unified environment for verifying deep neural networks. arXiv (2018). arxiv:1811.03752.","journal-title":"arXiv"},{"key":"e_1_3_3_177_2","first-page":"2067","article-title":"Trial without error: Towards safe reinforcement learning via human intervention","volume":"3","author":"Saunders William","year":"2018","unstructured":"William Saunders, Andreas Stuhlm\u00fcller, Girish Sastry, and Owain Evans. 2018. Trial without error: Towards safe reinforcement learning via human intervention. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 3 (2018), 2067\u20132069. arxiv:1707.05173.","journal-title":"Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS"},{"key":"e_1_3_3_178_2","first-page":"5","article-title":"On challenges in machine learning model management","author":"Schelter Sebastian","year":"2018","unstructured":"Sebastian Schelter, Felix Biessmann, Tim Januschowski, David Salinas, Stephan Seufert, and Gyuri Szarvas. 2018. On challenges in machine learning model management. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering (2018), 5\u201313. http:\/\/sites.computer.org\/debull\/A18dec\/p5.pdf.","journal-title":"Bulletin of the IEEE Computer Society Technical Committee on Data Engineering"},{"key":"e_1_3_3_179_2","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788628"},{"key":"e_1_3_3_180_2","first-page":"2503","article-title":"Hidden technical debt in machine learning systems","volume":"2015","author":"Sculley D.","year":"2015","unstructured":"D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean Fran\u00e7ois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems 2015-Jan (2015), 2503\u20132511.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_181_2","doi-asserted-by":"publisher","DOI":"10.1145\/3382494.3410681"},{"key":"e_1_3_3_182_2","article-title":"An empirical study of software architecture for machine learning","author":"Serban Alex","year":"2021","unstructured":"Alex Serban and Joost Visser. 2021. An empirical study of software architecture for machine learning. arXiv preprint arXiv:2105.12422 (2021).","journal-title":"arXiv preprint arXiv:2105.12422"},{"key":"e_1_3_3_183_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSA-C.2019.00051"},{"key":"e_1_3_3_184_2","first-page":"1","article-title":"Towards verified artificial intelligence","author":"Seshia Sanjit A.","year":"2016","unstructured":"Sanjit A. Seshia, Dorsa Sadigh, and S. Shankar Sastry. 2016. Towards verified artificial intelligence. arXiv (2016), 1\u201318. arxiv:1606.08514http:\/\/arxiv.org\/abs\/1606.08514.","journal-title":"arXiv"},{"key":"e_1_3_3_185_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-99229-7_39"},{"key":"e_1_3_3_186_2","first-page":"1","article-title":"On a formal model of safe and scalable self-driving cars","author":"Shalev-Shwartz Shai","year":"2017","unstructured":"Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. 2017. On a formal model of safe and scalable self-driving cars. arXiv (2017), 1\u201337. arxiv:1708.06374.","journal-title":"arXiv"},{"key":"e_1_3_3_187_2","doi-asserted-by":"publisher","DOI":"10.1007\/s13218-018-0559-3"},{"key":"e_1_3_3_188_2","article-title":"Machine teaching a new paradigm for building machine learning systems","author":"Simard Patrice Y.","year":"2017","unstructured":"Patrice Y. Simard, Saleema Amershi, David M. Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Christopher Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, and John Wernsing. 2017. Machine teaching a new paradigm for building machine learning systems. arXiv (2017). arxiv:1707.06742.","journal-title":"arXiv"},{"issue":"2","key":"e_1_3_3_189_2","article-title":"Towards testing of deep learning systems with training set reduction","author":"Spieker Helge","year":"2019","unstructured":"Helge Spieker and Arnaud Gotlieb. 2019. Towards testing of deep learning systems with training set reduction. arXiv2 (2019). arxiv:1901.04169.","journal-title":"arXiv"},{"key":"e_1_3_3_190_2","article-title":"MulDef: Multi-model-based defense against adversarial examples for neural networks","author":"Srisakaokul Siwakorn","year":"2018","unstructured":"Siwakorn Srisakaokul, Yuhao Zhang, Zexuan Zhong, Wei Yang, Tao Xie, and Bo Li. 2018. MulDef: Multi-model-based defense against adversarial examples for neural networks. arXiv (2018). arxiv:1809.00065.","journal-title":"arXiv"},{"key":"e_1_3_3_191_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC.2017.41"},{"key":"e_1_3_3_192_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00051"},{"key":"e_1_3_3_193_2","doi-asserted-by":"publisher","DOI":"10.1145\/3358233"},{"key":"e_1_3_3_194_2","first-page":"109","article-title":"Concolic testing for deep neural networks","author":"Sun Youcheng","year":"2018","unstructured":"Youcheng Sun, Min Wu, Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska, and Daniel Kroening. 2018. Concolic testing for deep neural networks. arXiv (2018), 109\u2013119.","journal-title":"arXiv"},{"key":"e_1_3_3_195_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9196932"},{"key":"e_1_3_3_196_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2012.22"},{"key":"e_1_3_3_197_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_3_198_2","volume-title":"25th USENIX Security Symposium (USENIX Security 16)","author":"Tram\u00e8r Florian","year":"2016","unstructured":"Florian Tram\u00e8r, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In 25th USENIX Security Symposium (USENIX Security 16). https:\/\/www.usenix.org\/conference\/usenixsecurity16\/technical-sessions\/presentation\/tramer."},{"key":"e_1_3_3_199_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178126.3187004"},{"key":"e_1_3_3_200_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIV.2019.2955903"},{"key":"e_1_3_3_201_2","first-page":"98","article-title":"Automated directed fairness testing","author":"Udeshi Sakshi","year":"2018","unstructured":"Sakshi Udeshi, Pryanshu Arora, and Sudipta Chattopadhyay. 2018. Automated directed fairness testing. arXiv (2018), 98\u2013108.","journal-title":"arXiv"},{"key":"e_1_3_3_202_2","doi-asserted-by":"publisher","DOI":"10.1145\/3076246.3076248"},{"key":"e_1_3_3_203_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITA.2016.7888195"},{"key":"e_1_3_3_204_2","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0051"},{"key":"e_1_3_3_205_2","doi-asserted-by":"publisher","DOI":"10.1145\/3160504.3160584"},{"key":"e_1_3_3_206_2","doi-asserted-by":"publisher","DOI":"10.1109\/REW.2019.00050"},{"key":"e_1_3_3_207_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2937083"},{"key":"e_1_3_3_208_2","first-page":"1","article-title":"Detecting adversarial samples for deep neural networks through mutation testing","author":"Wang Jingyi","year":"2018","unstructured":"Jingyi Wang, Jun Sun, Peixin Zhang, and Xinyu Wang. 2018. Detecting adversarial samples for deep neural networks through mutation testing. arXiv (2018), 1\u201310. arxiv:1805.05010.","journal-title":"arXiv"},{"key":"e_1_3_3_209_2","article-title":"Synergy between machine\/deep learning and software engineering: How far are we?","author":"Wang Simin","year":"2020","unstructured":"Simin Wang, Liguo Huang, Jidong Ge, Tengfei Zhang, Haitao Feng, Ming Li, He Zhang, and Vincent Ng. 2020. Synergy between machine\/deep learning and software engineering: How far are we? arXiv preprint arXiv:2008.05515 (2020).","journal-title":"arXiv preprint arXiv:2008.05515"},{"key":"e_1_3_3_210_2","article-title":"Efficient formal safety analysis of neural networks","author":"Wang Shiqi","year":"2018","unstructured":"Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Efficient formal safety analysis of neural networks. arXivNeurIPS (2018).","journal-title":"arXiv"},{"key":"e_1_3_3_211_2","article-title":"Formal security analysis of neural networks using symbolic intervals","author":"Wang Shiqi","year":"2018","unstructured":"Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Formal security analysis of neural networks using symbolic intervals. arXiv (2018).","journal-title":"arXiv"},{"key":"e_1_3_3_212_2","doi-asserted-by":"publisher","DOI":"10.1109\/IWESEP49350.2019.00017"},{"key":"e_1_3_3_213_2","doi-asserted-by":"publisher","DOI":"10.1145\/2601248.2601268"},{"key":"e_1_3_3_214_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2013.04.076"},{"key":"e_1_3_3_215_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3391496"},{"key":"e_1_3_3_216_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSREW.2017.46"},{"key":"e_1_3_3_217_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111029"},{"key":"e_1_3_3_218_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2019.00026"},{"key":"e_1_3_3_219_2","first-page":"3","volume-title":"Dependable Software Engineering. Theories, Tools, and Applications","author":"Xie Tao","year":"2018","unstructured":"Tao Xie. 2018. Intelligent software engineering: Synergy between AI and software engineering. In Dependable Software Engineering. Theories, Tools, and Applications, Xinyu Feng, Markus M\u00fcller-Olm, and Zijiang Yang (Eds.). Springer International Publishing, Cham, 3\u20137."},{"key":"e_1_3_3_220_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_3_3_221_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/800"},{"key":"e_1_3_3_222_2","first-page":"179","article-title":"Gray-box adversarial testing for control systems with machine learning component","author":"Yaghoubi Shakiba","year":"2018","unstructured":"Shakiba Yaghoubi and Georgios Fainekos. 2018. Gray-box adversarial testing for control systems with machine learning component. arXiv (2018), 179\u2013184.","journal-title":"arXiv"},{"key":"e_1_3_3_223_2","first-page":"406","article-title":"The Role of Design in Creating Machine-Learning-enhanced User Experience","volume":"17","author":"Yang Qian","year":"2017","unstructured":"Qian Yang. 2017. The Role of Design in Creating Machine-Learning-enhanced User Experience. AAAI Spring Symposium - Technical Report SS-17-01 - (2017), 406\u2013411.","journal-title":"AAAI Spring Symposium - Technical Report"},{"key":"e_1_3_3_224_2","first-page":"400","article-title":"Telemade: A testing framework for learning-based malware detection systems","author":"Yang Wei","year":"2018","unstructured":"Wei Yang and Tao Xie. 2018. Telemade: A testing framework for learning-based malware detection systems. Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence (2018), 400\u2013403.","journal-title":"Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence"},{"key":"e_1_3_3_225_2","first-page":"1","article-title":"End-to-end robustness for sensing-reasoning machine learning pipelines","author":"Yang Zhuolin","year":"2020","unstructured":"Zhuolin Yang, Zhikuan Zhao, Hengzhi Pei, Boxin Wang, Bojan Karlas, Ji Liu, Heng Guo, Bo Li, and Ce Zhang. 2020. End-to-end robustness for sensing-reasoning machine learning pipelines. arXiv (2020), 1\u201343. arxiv:2003.00120.","journal-title":"arXiv"},{"key":"e_1_3_3_226_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSA-C.2019.00055"},{"key":"e_1_3_3_227_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_3_228_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2019.00020"},{"key":"e_1_3_3_229_2","article-title":"Software engineering practice in the development of deep learning applications","author":"Zhang Xufan","year":"2019","unstructured":"Xufan Zhang, Yilin Yang, Yang Feng, and Zhenyu Chen. 2019. Software engineering practice in the development of deep learning applications. arXiv (2019). arxiv:1910.03156.","journal-title":"arXiv"},{"key":"e_1_3_3_230_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"},{"key":"e_1_3_3_231_2","article-title":"Packaging and sharing machine learning models via the Acumos AI open platform","author":"Zhao Shuai","year":"2018","unstructured":"Shuai Zhao, Manoop Talasila, Guy Jacobson, Cristian Borcea, Syed Anwar Aftab, and John F. Murray. 2018. Packaging and sharing machine learning models via the Acumos AI open platform. arXiv (2018).","journal-title":"arXiv"},{"key":"e_1_3_3_232_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS-C.2018.00017"},{"key":"e_1_3_3_233_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00131"},{"key":"e_1_3_3_234_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380422"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3487043","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3487043","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:47Z","timestamp":1750191527000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3487043"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4]]},"references-count":233,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,4,30]]}},"alternative-id":["10.1145\/3487043"],"URL":"https:\/\/doi.org\/10.1145\/3487043","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4]]},"assertion":[{"value":"2021-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}