{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T05:29:05Z","timestamp":1769750945996,"version":"3.49.0"},"reference-count":75,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,6,30]],"date-time":"2020-06-30T00:00:00Z","timestamp":1593475200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100006602","name":"Air Force Research Laboratory","doi-asserted-by":"publisher","award":["FA8750-16-2-0284 and FA8750-19-2-0002"],"award-info":[{"award-number":["FA8750-16-2-0284 and FA8750-19-2-0002"]}],"id":[{"id":"10.13039\/100006602","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["CNS-1305358 and DBI-0939454"],"award-info":[{"award-number":["CNS-1305358 and DBI-0939454"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Auton. Adapt. Syst."],"published-print":{"date-parts":[[2020,6,30]]},"abstract":"<jats:p>Data-driven Learning-enabled Systems are limited by the quality of available training data, particularly when trained offline. For systems that must operate in real-world environments, the space of possible conditions that can occur is vast and difficult to comprehensively predict at design time. Environmental uncertainty arises when run-time conditions diverge from design-time training conditions. To address this problem, automated methods can generate synthetic data to fill in gaps for training and test data coverage. We propose an evolution-based technique to assist developers with uncovering limitations in existing data when previously unseen environmental phenomena are introduced. This technique explores unique contexts for a given environmental condition, with an emphasis on diversity. Synthetic data generated by this technique may be used for two purposes: (1) to assess the robustness of a system to uncertain environmental factors and (2) to improve the system\u2019s robustness. This technique is demonstrated to outperform random and greedy methods for multiple adverse environmental conditions applied to image-processing Deep Neural Networks.<\/jats:p>","DOI":"10.1145\/3460959","type":"journal-article","created":{"date-parts":[[2021,5,30]],"date-time":"2021-05-30T02:07:46Z","timestamp":1622340466000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Enki"],"prefix":"10.1145","volume":"15","author":[{"given":"Michael Austin","family":"Langford","sequence":"first","affiliation":[{"name":"Michigan State University, East Lansing, MI"}]},{"given":"Betty H. C.","family":"Cheng","sequence":"additional","affiliation":[{"name":"Michigan State University, East Lansing, MI"}]}],"member":"320","published-online":{"date-parts":[[2021,5,29]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera.","author":"Arrieta Alejandro Barredo","year":"2020","unstructured":"Alejandro Barredo Arrieta , Natalia Diaz-Rodriguez , Javier Del Ser , Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020 . Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Info. Fusion 58 (June 2020), 82\u2013115. DOI:https:\/\/doi.org\/10.1016\/j.inffus.2019.12.012 Alejandro Barredo Arrieta, Natalia Diaz-Rodriguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Info. Fusion 58 (June 2020), 82\u2013115. DOI:https:\/\/doi.org\/10.1016\/j.inffus.2019.12.012"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.354"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the ACM\/IEEE International Conference on Model-driven Engineering Languages and Systems (MODELS\u201920)","author":"Bengio Yoshua","year":"2020","unstructured":"Yoshua Bengio . 2020 . Priors For Deep Learning of Semantic Representations . In Proceedings of the ACM\/IEEE International Conference on Model-driven Engineering Languages and Systems (MODELS\u201920) . Yoshua Bengio. 2020. Priors For Deep Learning of Semantic Representations. In Proceedings of the ACM\/IEEE International Conference on Model-driven Engineering Languages and Systems (MODELS\u201920)."},{"key":"e_1_2_1_4_1","volume-title":"Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13 (Feb","author":"Bergstra James","year":"2012","unstructured":"James Bergstra and Yoshua Bengio . 2012. Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13 (Feb . 2012 ), 281\u2013305. James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13 (Feb. 2012), 281\u2013305."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/SBST.2015.17"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2739482.2764716"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/3013545.3013548"},{"key":"e_1_2_1_8_1","series-title":"Lecture Notes in Computer Science (LNCS)","volume-title":"Proceedings of the International Conference on Computer Safety, Reliability, and Security","author":"Burton Simon","unstructured":"Simon Burton , Lydia Gauerhof , and Christian Heinzemann . 2017. Making the case for safety of machine learning in highly automated driving . In Proceedings of the International Conference on Computer Safety, Reliability, and Security . Lecture Notes in Computer Science (LNCS) , Vol. 10489 . Springer , Cham . Simon Burton, Lydia Gauerhof, and Christian Heinzemann. 2017. Making the case for safety of machine learning in highly automated driving. In Proceedings of the International Conference on Computer Safety, Reliability, and Security. Lecture Notes in Computer Science (LNCS), Vol. 10489. Springer, Cham."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1868328.1868344"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSCI.2016.7850178"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/RCIS.2013.6577674"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEAMS.2019.00012"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/3103620.3103631"},{"key":"e_1_2_1_14_1","volume-title":"Smith","author":"Eiben Agoston E.","year":"2015","unstructured":"Agoston E. Eiben and James E . Smith . 2015 . Introduction to Evolutionary Computing (2nd ed.). Springer-Verlag , Berlin, Heidelberg, DEU. Agoston E. Eiben and James E. Smith. 2015. Introduction to Evolutionary Computing (2nd ed.). Springer-Verlag, Berlin, Heidelberg, DEU."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00175"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025179"},{"key":"e_1_2_1_17_1","unstructured":"Hans Gaiser Maarten de Vries Valeriu Lacatusu V. Carpani Ashley Williamson Enrico Liscio Andreas Yann Henon jjiun Cristian Gratie Mihai Morariu Charles Ye Martin Zlocha Ben Weinstein Rodrigo Meira de Andrade Pedro Concei\u00e7\u00e3o Alexander Pacha hannesedvartsen Daniyal Shahrokhian Wudi Fang Mike Clark meagerYak Iver Jordal Max Van Sande Jin Etienne-Meunier Andrew Grigorev Guillaume Erhard Eduardo Ramos and Denis Dowling. 2019. fizyr\/keras-retinanet 0.5.1. Retrieved from https:\/\/github.com\/fizyr\/keras-retinanet\/releases. DOI:https:\/\/doi.org\/10.5281\/zenodo.3250670  Hans Gaiser Maarten de Vries Valeriu Lacatusu V. Carpani Ashley Williamson Enrico Liscio Andreas Yann Henon jjiun Cristian Gratie Mihai Morariu Charles Ye Martin Zlocha Ben Weinstein Rodrigo Meira de Andrade Pedro Concei\u00e7\u00e3o Alexander Pacha hannesedvartsen Daniyal Shahrokhian Wudi Fang Mike Clark meagerYak Iver Jordal Max Van Sande Jin Etienne-Meunier Andrew Grigorev Guillaume Erhard Eduardo Ramos and Denis Dowling. 2019. fizyr\/keras-retinanet 0.5.1. Retrieved from https:\/\/github.com\/fizyr\/keras-retinanet\/releases. DOI:https:\/\/doi.org\/10.5281\/zenodo.3250670"},{"key":"e_1_2_1_18_1","volume-title":"Nayar","author":"Garg Kshitiz","year":"2004","unstructured":"Kshitiz Garg and Shree K . Nayar . 2004 . Photometric Model of a Rain Drop. Technical Report. Columbia University . Retrieved from https:\/\/www1.cs.columbia.edu\/CAVE\/publications\/pdfs\/Garg_TR04.pdf. Kshitiz Garg and Shree K. Nayar. 2004. Photometric Model of a Rain Drop. Technical Report. Columbia University. Retrieved from https:\/\/www1.cs.columbia.edu\/CAVE\/publications\/pdfs\/Garg_TR04.pdf."},{"key":"e_1_2_1_19_1","unstructured":"Xavier Gastaldi. 2017. Shake-Shake regularization. Retrieved from https:\/\/arXiv:1705.07485.  Xavier Gastaldi. 2017. Shake-Shake regularization. Retrieved from https:\/\/arXiv:1705.07485."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2012.6248074"},{"key":"e_1_2_1_21_1","volume-title":"Deep Learning","author":"Goodfellow Ian","unstructured":"Ian Goodfellow , Yoshua Bengio , and Aaron Courville . 2016. Deep Learning . MIT , Cambridge, MA . Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT, Cambridge, MA."},{"key":"e_1_2_1_22_1","volume-title":"Advances in Neural Information Processing Systems.","author":"Goodfellow Ian","unstructured":"Ian Goodfellow , Jean Pouget-Abadie , Mehdi Mirza , Bing Xu , David Warde-Farley , Sherjil Ozair , Aaron Courville , and Yoshua Bengio . 2014. Generative adversarial nets . In Advances in Neural Information Processing Systems. Vol. 27 . Curran Associates , Red Hook, NY , 2672\u20132680. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. Vol. 27. Curran Associates, Red Hook, NY, 2672\u20132680."},{"key":"e_1_2_1_23_1","unstructured":"Ian J. Goodfellow Jonathons Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. Retrieved from https:\/\/arXiv:1412.6572.  Ian J. Goodfellow Jonathons Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. Retrieved from https:\/\/arXiv:1412.6572."},{"key":"e_1_2_1_25_1","volume-title":"This is how AI bias really happens\u2014And why it\u2019s so hard to fix. MIT Technology Review Tech policy\/AI Ethics (Feb","author":"Hao Karen","year":"2019","unstructured":"Karen Hao . 2019. This is how AI bias really happens\u2014And why it\u2019s so hard to fix. MIT Technology Review Tech policy\/AI Ethics (Feb . 2019 ), 1 page. Retrieved from https:\/\/www.technologyreview.com\/2019\/02\/04\/137602\/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix\/. Karen Hao. 2019. This is how AI bias really happens\u2014And why it\u2019s so hard to fix. MIT Technology Review Tech policy\/AI Ethics (Feb. 2019), 1 page. Retrieved from https:\/\/www.technologyreview.com\/2019\/02\/04\/137602\/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix\/."},{"key":"e_1_2_1_26_1","volume-title":"Article 11 (Dec.","author":"Harman Mark","year":"2012","unstructured":"Mark Harman , S. Afshin Mansouri , and Yuanyuan Zhang . 2012. Search-based software engineering: Trends, techniques, and applications. ACM Comput. Surv. 45, 1 , Article 11 (Dec. 2012 ), 61 pages pages. DOI:https:\/\/doi.org\/10.1145\/2379776.2379787 Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based software engineering: Trends, techniques, and applications. ACM Comput. Surv. 45, 1, Article 11 (Dec. 2012), 61 pages pages. DOI:https:\/\/doi.org\/10.1145\/2379776.2379787"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2830661"},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1109\/TKDE.2008.239","article-title":"Learning from imbalanced data","volume":"21","author":"He Haibo","year":"2009","unstructured":"Haibo He and Edwardo A. Garcia . 2009 . Learning from imbalanced data . IEEE Trans. Knowl. Data Eng. 21 , 9 (Sept. 2009), 1263\u20131284. DOI:https:\/\/doi.org\/10.1109\/TKDE.2008.239 Haibo He and Edwardo A. Garcia. 2009. Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21, 9 (Sept. 2009), 1263\u20131284. DOI:https:\/\/doi.org\/10.1109\/TKDE.2008.239","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_30_1","volume-title":"Computer Graphics: Principles and Practice","author":"Hughes John F.","year":"2013","unstructured":"John F. Hughes , Andries van Dam , Morgan McGuire , David F. Sklar , James D. Foley , Steven Feiner , and Kurt Akeley . 2013 . Computer Graphics: Principles and Practice . Addison-Wesley , Upper Saddle River, NJ. John F. Hughes, Andries van Dam, Morgan McGuire, David F. Sklar, James D. Foley, Steven Feiner, and Kurt Akeley. 2013. Computer Graphics: Principles and Practice. Addison-Wesley, Upper Saddle River, NJ."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 32nd International Conference on Machine Learning (ICML\u201915)","volume":"37","author":"Ioffe Sergey","year":"2015","unstructured":"Sergey Ioffe and Christian Szegedy . 2015 . Batch normalization: Accelerating deep network training by reducing internal covariate shift . In Proceedings of the 32nd International Conference on Machine Learning (ICML\u201915) , Vol. 37 . Microtome Publishing, Brookline, MA, 448\u2013456. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning (ICML\u201915), Vol. 37. Microtome Publishing, Brookline, MA, 448\u2013456."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-59496-5_337"},{"key":"e_1_2_1_33_1","unstructured":"Joel Janai Fatma G\u00fcney Aseem Behl and Andreas Geiger. 2017. Computer vision for autonomous vehicles: Problems datasets and state-of-the-art. Retrieved from https:\/\/arXiv:1704.05519.  Joel Janai Fatma G\u00fcney Aseem Behl and Andreas Geiger. 2017. Computer vision for autonomous vehicles: Problems datasets and state-of-the-art. Retrieved from https:\/\/arXiv:1704.05519."},{"key":"e_1_2_1_34_1","unstructured":"Jason Jo and Yoshua Bengio. 2017. Measuring the tendency of CNNs to learn surface statistical regularities. Retrieved from https:\/\/arXiv:1711.11561.  Jason Jo and Yoshua Bengio. 2017. Measuring the tendency of CNNs to learn surface statistical regularities. Retrieved from https:\/\/arXiv:1711.11561."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSYST.2014.2322503"},{"key":"e_1_2_1_36_1","volume-title":"Deep networks can resemble human feed-forward vision in invariant object recognition. Sci. Rep. 6, 32672","author":"Kheradpisheh Saeed Reza","year":"2016","unstructured":"Saeed Reza Kheradpisheh , Masoud Ghodrati , Mohammad Ganjtabesh , and Timoth\u00e9e Masquelier . 2016. Deep networks can resemble human feed-forward vision in invariant object recognition. Sci. Rep. 6, 32672 ( 2016 ), 24 pages. DOI:https:\/\/doi.org\/10.1038\/srep32672 Saeed Reza Kheradpisheh, Masoud Ghodrati, Mohammad Ganjtabesh, and Timoth\u00e9e Masquelier. 2016. Deep networks can resemble human feed-forward vision in invariant object recognition. Sci. Rep. 6, 32672 (2016), 24 pages. DOI:https:\/\/doi.org\/10.1038\/srep32672"},{"key":"e_1_2_1_37_1","volume-title":"Kingma and Jimmy Ba","author":"Diederik","year":"2014","unstructured":"Diederik P. Kingma and Jimmy Ba . 2014 . Adam : A method for stochastic optimization. Retrieved from https:\/\/arXiv:1412.6980. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Retrieved from https:\/\/arXiv:1412.6980."},{"key":"e_1_2_1_38_1","doi-asserted-by":"crossref","unstructured":"Wilhelm Kirch (Ed.). 2008. Pearson\u2019s Correlation Coefficient. Springer Netherlands Dordrecht NLD 1090\u20131091. DOI:https:\/\/doi.org\/10.1007\/978-1-4020-5614-7_2569  Wilhelm Kirch (Ed.). 2008. Pearson\u2019s Correlation Coefficient. Springer Netherlands Dordrecht NLD 1090\u20131091. DOI:https:\/\/doi.org\/10.1007\/978-1-4020-5614-7_2569","DOI":"10.1007\/978-1-4020-5614-7_2569"},{"key":"e_1_2_1_39_1","volume-title":"The dark secret at the heart of AI. MIT Technology Review Artificial intelligence\/Machine learning (Apr","author":"Knight Will","year":"2017","unstructured":"Will Knight . 2017. The dark secret at the heart of AI. MIT Technology Review Artificial intelligence\/Machine learning (Apr . 2017 ), 1. Retrieved from https:\/\/www.technologyreview.com\/2017\/04\/11\/5113\/the-dark-secret-at-the-heart-of-ai\/. Will Knight. 2017. The dark secret at the heart of AI. MIT Technology Review Artificial intelligence\/Machine learning (Apr. 2017), 1. Retrieved from https:\/\/www.technologyreview.com\/2017\/04\/11\/5113\/the-dark-secret-at-the-heart-of-ai\/."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3065386"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICAC.2016.18"},{"key":"e_1_2_1_43_1","unstructured":"Alexey Kurakin Ian J. Goodfellow and Sami Bengio. 2016. Adversarial machine learning at scale. Retrieved from https:\/\/arXiv:1611.01236.  Alexey Kurakin Ian J. Goodfellow and Sami Bengio. 2016. Adversarial machine learning at scale. Retrieved from https:\/\/arXiv:1611.01236."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC\u201919)","author":"Michael","year":"2019","unstructured":"Michael A. Langford and Betty H. C. Cheng. 2019. Enhancing learning-enabled software systems to address environmental uncertainty . In Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC\u201919) . IEEE, Piscataway, NJ, 115\u2013124. DOI:https:\/\/doi.org\/10.1109\/ICAC. 2019 .00023 Michael A. Langford and Betty H. C. Cheng. 2019. Enhancing learning-enabled software systems to address environmental uncertainty. In Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC\u201919). IEEE, Piscataway, NJ, 115\u2013124. DOI:https:\/\/doi.org\/10.1109\/ICAC.2019.00023"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS\u201919)","author":"Langford Michael A.","year":"2019","unstructured":"Michael A. Langford , Glen A. Simon , Philip K. McKinley , and Betty H. C. Cheng . 2019. Applying evolution and novelty search to enhance the resilience of autonomous systems . In Proceedings of the 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS\u201919) . ACM, New York, NY, 63\u201369. DOI:https:\/\/doi.org\/10.1109\/SEAMS. 2019 .00017 Michael A. Langford, Glen A. Simon, Philip K. McKinley, and Betty H. C. Cheng. 2019. Applying evolution and novelty search to enhance the resilience of autonomous systems. In Proceedings of the 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS\u201919). ACM, New York, NY, 63\u201369. DOI:https:\/\/doi.org\/10.1109\/SEAMS.2019.00017"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1162\/EVCO_a_00025"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.324"},{"key":"e_1_2_1_50_1","unstructured":"Fredrik Lundh Alex Clark and Contributors. 2015. Pillow (PIL Fork) 3.0x. Read the Docs. Retrieved from https:\/\/pillow.readthedocs.io\/en\/3.0.x\/about.html.  Fredrik Lundh Alex Clark and Contributors. 2015. Pillow (PIL Fork) 3.0x. Read the Docs. Retrieved from https:\/\/pillow.readthedocs.io\/en\/3.0.x\/about.html."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931054"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2011.100"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML\u201910)","author":"Nair Vinod","unstructured":"Vinod Nair and Geoffrey E. Hinton . 2010. Rectified linear units improve restricted Boltzmann machines . In Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML\u201910) . Omnipress, Madison, WI, 807\u2013814. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML\u201910). Omnipress, Madison, WI, 807\u2013814."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298640"},{"key":"e_1_2_1_56_1","unstructured":"Nvidia Corporation. 2017. prepare_kitti_data.py. Retrieved from https:\/\/github.com\/NVIDIA\/DIGITS\/blob\/master\/examples\/object-detection\/prepare_kitti_data.py.  Nvidia Corporation. 2017. prepare_kitti_data.py. Retrieved from https:\/\/github.com\/NVIDIA\/DIGITS\/blob\/master\/examples\/object-detection\/prepare_kitti_data.py."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the 26th IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201911)","author":"Ramirez Andres J.","year":"2011","unstructured":"Andres J. Ramirez , Adam C. Jensen , Betty H. C. Cheng , and David B. Knoester . 2011. Automatically exploring how uncertainty impacts behavior of dynamically adaptive systems . In Proceedings of the 26th IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201911) . IEEE, Piscataway, NJ, 568\u2013571. DOI:https:\/\/doi.org\/10.1109\/ASE. 2011 .6100127 Andres J. Ramirez, Adam C. Jensen, Betty H. C. Cheng, and David B. Knoester. 2011. Automatically exploring how uncertainty impacts behavior of dynamically adaptive systems. In Proceedings of the 26th IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201911). IEEE, Piscataway, NJ, 568\u2013571. DOI:https:\/\/doi.org\/10.1109\/ASE.2011.6100127"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1007\/10721187_1"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2017.XIII.064"},{"key":"e_1_2_1_61_1","volume-title":"Russell and Peter Norvig","author":"Stuart","year":"2010","unstructured":"Stuart J. Russell and Peter Norvig . 2010 . Artificial Intelligence : A Modern Approach (3 ed.). Pearson Education, Upper Saddle River, NJ. Stuart J. Russell and Peter Norvig. 2010. Artificial Intelligence: A Modern Approach (3 ed.). Pearson Education, Upper Saddle River, NJ."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.2352\/ISSN.2470-1173.2017.19.AVM-023"},{"key":"e_1_2_1_64_1","volume-title":"Application of Neural Networks in High Assurance Systems: A Survey. Studies in Computational Intelligence (SCI)","author":"Schumann Johann","unstructured":"Johann Schumann , Pramod Gupta , and Yan Liu . 2010. Application of Neural Networks in High Assurance Systems: A Survey. Studies in Computational Intelligence (SCI) , Vol. 268 . Springer , Berlin . Johann Schumann, Pramod Gupta, and Yan Liu. 2010. Application of Neural Networks in High Assurance Systems: A Survey. Studies in Computational Intelligence (SCI), Vol. 268. Springer, Berlin."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00030"},{"key":"e_1_2_1_66_1","volume-title":"Proceedings of the the International Conference on Information Quality (ICIQ\u201906)","author":"Sessions Valerie","year":"2006","unstructured":"Valerie Sessions and Marco Valtorta . 2006 . The effects of data quality on machine learning algorithms . In Proceedings of the the International Conference on Information Quality (ICIQ\u201906) . MIT, Cambridge, MA, 485\u2013498. Valerie Sessions and Marco Valtorta. 2006. The effects of data quality on machine learning algorithms. In Proceedings of the the International Conference on Information Quality (ICIQ\u201906). MIT, Cambridge, MA, 485\u2013498."},{"key":"e_1_2_1_68_1","volume-title":"Information Theory: A Tutorial Introduction","author":"Stone James V.","year":"2015","unstructured":"James V. Stone . 2015 . Information Theory: A Tutorial Introduction ( 1 st ed.). Sebtel Press . James V. Stone. 2015. Information Theory: A Tutorial Introduction (1st ed.). Sebtel Press.","edition":"1"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00051"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_2_1_72_1","unstructured":"Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow and Rob Fergus. 2013. Intriguing properties of neural networks. Retrieved from https:\/\/arXiv:1312.6199.  Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow and Rob Fergus. 2013. Intriguing properties of neural networks. Retrieved from https:\/\/arXiv:1312.6199."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the 55th Annual Design Automation Conference (DAC\u201918)","author":"Tuncali Cumhur Erkan","year":"1959","unstructured":"Cumhur Erkan Tuncali , James Kapinski , Hisahiro Ito , and Jyotirmoy V. Deshmukh . 2018. Reasoning about safety of learning-enabled components in autonomous cyber-physical systems . In Proceedings of the 55th Annual Design Automation Conference (DAC\u201918) . ACM, New York, NY, Article 30, 6 pages. DOI:https:\/\/doi.org\/10.1145\/3 1959 70.3199852 Cumhur Erkan Tuncali, James Kapinski, Hisahiro Ito, and Jyotirmoy V. Deshmukh. 2018. Reasoning about safety of learning-enabled components in autonomous cyber-physical systems. In Proceedings of the 55th Annual Design Automation Conference (DAC\u201918). ACM, New York, NY, Article 30, 6 pages. DOI:https:\/\/doi.org\/10.1145\/3195970.3199852"},{"key":"e_1_2_1_79_1","volume-title":"Probability & Statistics For Engineers & Scientists","author":"Walpole Ronald E.","unstructured":"Ronald E. Walpole , Raymond H. Myers , Sharon L. Myers , and Keying Ye. 2012. Probability & Statistics For Engineers & Scientists ( 9 th ed.). Prentice Hall , Boston, MA . Ronald E. Walpole, Raymond H. Myers, Sharon L. Myers, and Keying Ye. 2012. Probability & Statistics For Engineers & Scientists (9th ed.). Prentice Hall, Boston, MA.","edition":"9"},{"key":"e_1_2_1_80_1","article-title":"Model-driven deep-learning","volume":"5","author":"Xu Zongben","year":"2017","unstructured":"Zongben Xu and Jian Sun . 2017 . Model-driven deep-learning . Natl. Sci. Rev. 5 , 1 (08 2017), 22\u201324. DOI:https:\/\/doi.org\/10.1093\/nsr\/nwx099 Zongben Xu and Jian Sun. 2017. Model-driven deep-learning. Natl. Sci. Rev. 5, 1 (08 2017), 22\u201324. DOI:https:\/\/doi.org\/10.1093\/nsr\/nwx099","journal-title":"Natl. Sci. Rev."},{"key":"e_1_2_1_81_1","unstructured":"Jason Yosinski Jeff Clune Anh Nguyen Thomas Fuchs and Hod Lipson. 2015. Understanding neural networks through deep visualization. Retrieved from https:\/\/arXiv:1506.06579.  Jason Yosinski Jeff Clune Anh Nguyen Thomas Fuchs and Hod Lipson. 2015. Understanding neural networks through deep visualization. Retrieved from https:\/\/arXiv:1506.06579."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.5555\/3367471.3367625"},{"key":"e_1_2_1_83_1","doi-asserted-by":"crossref","unstructured":"Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. Retrieved from https:\/\/arXiv:1605.07146.  Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. Retrieved from https:\/\/arXiv:1605.07146.","DOI":"10.5244\/C.30.87"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_2_1_85_1","volume-title":"Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917)","author":"Zhu Jun-Yan","unstructured":"Jun-Yan Zhu , Taesung Park , Phillip Isola , and Alexei A. Efros . 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks . In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917) . IEEE, Piscataway, NJ, 2242\u20132251. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917). IEEE, Piscataway, NJ, 2242\u20132251."}],"container-title":["ACM Transactions on Autonomous and Adaptive Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3460959","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3460959","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3460959","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:48:22Z","timestamp":1750193302000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3460959"}},"subtitle":["A Diversity-driven Approach to Test and Train Robust Learning-enabled Systems"],"short-title":[],"issued":{"date-parts":[[2020,6,30]]},"references-count":75,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6,30]]}},"alternative-id":["10.1145\/3460959"],"URL":"https:\/\/doi.org\/10.1145\/3460959","relation":{},"ISSN":["1556-4665","1556-4703"],"issn-type":[{"value":"1556-4665","type":"print"},{"value":"1556-4703","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,30]]},"assertion":[{"value":"2020-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-05-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}