{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,24]],"date-time":"2026-01-24T15:03:15Z","timestamp":1769266995410,"version":"3.49.0"},"reference-count":122,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,7,6]],"date-time":"2022-07-06T00:00:00Z","timestamp":1657065600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["CMMI-1653339\/1834710, IIS-1849280, and IIS-1849304"],"award-info":[{"award-number":["CMMI-1653339\/1834710, IIS-1849280, and IIS-1849304"]}]},{"name":"Manufacturing Futures Initiative at Carnegie Mellon University"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Model. Comput. Simul."],"published-print":{"date-parts":[[2022,7,31]]},"abstract":"<jats:p>We study rare-event simulation for a class of problems where the target hitting sets of interest are defined via modern machine learning tools such as neural networks and random forests. This problem is motivated from fast emerging studies on the safety evaluation of intelligent systems, robustness quantification of learning models, and other potential applications to large-scale simulation in which machine learning tools can be used to approximate complex rare-event set boundaries. We investigate an importance sampling scheme that integrates the dominating point machinery in large deviations and sequential mixed integer programming to locate the underlying dominating points. Our approach works for a range of neural network architectures including fully connected layers, rectified linear units, normalization, pooling and convolutional layers, and random forests built from standard decision trees. We provide efficiency guarantees and numerical demonstration of our approach using a classification model in the UCI Machine Learning Repository.<\/jats:p>","DOI":"10.1145\/3519385","type":"journal-article","created":{"date-parts":[[2022,3,7]],"date-time":"2022-03-07T20:34:42Z","timestamp":1646685282000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["Rare-event Simulation for Neural Network and Random Forest Predictors"],"prefix":"10.1145","volume":"32","author":[{"given":"Yuanlu","family":"Bai","sequence":"first","affiliation":[{"name":"Columbia University, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1284-2128","authenticated-orcid":false,"given":"Zhiyuan","family":"Huang","sequence":"additional","affiliation":[{"name":"Tongji University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3193-563X","authenticated-orcid":false,"given":"Henry","family":"Lam","sequence":"additional","affiliation":[{"name":"Columbia University, USA"}]},{"given":"Ding","family":"Zhao","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,7,6]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1214\/11-AAP792"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1287\/opre.1060.0291"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3167969"},{"key":"e_1_3_2_5_2","first-page":"595","volume-title":"International Conference on Artificial Intelligence and Statistics","author":"Arief Mansur","year":"2021","unstructured":"Mansur Arief, Zhiyuan Huang, Guru Koushik Senthil Kumar, Yuanlu Bai, Shengyi He, Wenhao Ding, Henry Lam, and Ding Zhao. 2021. Deep probabilistic accelerated evaluation: A robust certifiable rare-event simulation methodology for black-box safety-critical systems. In International Conference on Artificial Intelligence and Statistics. PMLR, 595\u2013603."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/0304-4149(85)90211-X"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1142\/7431"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.2143\/AST.27.2.542054"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-69033-9"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0001867800001099"},{"key":"e_1_3_2_11_2","unstructured":"Arthur Asuncion and David Newman. 2007. UCI Machine Learning Repository. https:\/\/archive.ics.uci.edu\/ml\/index.php."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-74208-1_24"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.5555\/548834"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1214\/08-AAP558"},{"key":"e_1_3_2_15_2","first-page":"1351","article-title":"Efficient rare-event simulation for the maximum of heavy-tailed random walks","author":"Blanchet Jose","year":"2008","unstructured":"Jose Blanchet and Peter Glynn. 2008. Efficient rare-event simulation for the maximum of heavy-tailed random walks. Ann. Appl. Probab. 18, 4 (2008), 1351\u20131378.","journal-title":"Ann. Appl. Probab."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11134-009-9154-5"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.sorms.2011.09.002"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1287\/moor.2014.0654"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.spa.2012.05.002"},{"key":"e_1_3_2_20_2","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1002\/9780470745403.ch5","volume-title":"Rare Event Simulation Using Monte Carlo Methods","author":"Blanchet J.","year":"2009","unstructured":"J. Blanchet and M. Mandjes. 2009. Rare event simulation for queues. In Rare Event Simulation Using Monte Carlo Methods. Wiley Publishing, 87\u2013124."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1239\/aap\/1231340166"},{"key":"e_1_3_2_22_2","unstructured":"National Transportation Safety Board. 2018. Preliminary Report Highway hwy18mh010. Retrieved from https:\/\/www.ntsb.gov\/investigations\/AccidentReports\/Reports\/HWY18MH010-prelim.pdf."},{"key":"e_1_3_2_23_2","unstructured":"National Transportation Safety Board. 2019. Collision between Car Operating with Partial Driving Automation and Truck-tractor Semitrailer Delray Beach Florida March 1 2019. Retrieved from https:\/\/www.ntsb.gov\/investigations\/AccidentReports\/Reports\/HAB2001.pdf."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.nima.2003.08.157"},{"key":"e_1_3_2_25_2","first-page":"646","volume-title":"Winter Simulation Conference","author":"Boer P. T. De","year":"2000","unstructured":"P. T. De Boer, V. F. Nicola, and R. Y. Rubinstein. 2000. Adaptive importance sampling simulation of queueing networks. In Winter Simulation Conference. IEEE Press, 646\u2013655."},{"key":"e_1_3_2_26_2","first-page":"1","volume-title":"Mixed Integer Nonlinear Programming","author":"Bonami Pierre","year":"2012","unstructured":"Pierre Bonami, Mustafa Kilin\u00e7, and Jeff Linderoth. 2012. Algorithms and software for convex mixed integer nonlinear programs. In Mixed Integer Nonlinear Programming. Springer, 1\u201339."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.4310\/CIS.2003.v3.n4.a3"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1287\/ijoc.2019.0936"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11222-011-9308-2"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.3384148"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_3_2_32_2","volume-title":"Introduction to Rare Event Simulation","author":"Bucklew James","year":"2013","unstructured":"James Bucklew. 2013. Introduction to Rare Event Simulation. Springer Science & Business Media, New York."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11222-011-9275-7"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1287\/moor.2018.0950"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.312"},{"key":"e_1_3_2_36_2","first-page":"1856","volume-title":"IEEE Intelligent Vehicles Symposium","author":"Chen Zhilu","year":"2017","unstructured":"Zhilu Chen and Xinming Huang. 2017. End-to-end learning for lane keeping of self-driving cars. In IEEE Intelligent Vehicles Symposium. IEEE, 1856\u20131860."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/31.7600"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoap\/1015961169"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10479-005-5724-z"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.spa.2008.02.017"},{"key":"e_1_3_2_41_2","article-title":"Efficient black-box importance sampling for VaR and CVaR estimation","author":"Deo Anand","year":"2021","unstructured":"Anand Deo and Karthyek Murthy. 2021. Efficient black-box importance sampling for VaR and CVaR estimation. arXiv preprint arXiv:2106.10236 (2021).","journal-title":"arXiv preprint arXiv:2106.10236"},{"key":"e_1_3_2_42_2","first-page":"391","article-title":"A Markov chain perspective on adaptive Monte Carlo algorithms","volume":"9","author":"Desai P. Y.","year":"2001","unstructured":"P. Y. Desai and P. W. Glynn.2001. A Markov chain perspective on adaptive Monte Carlo algorithms. Proc. 2001 Winter Simul. Conf. 9 (2001), 391\u2013412.","journal-title":"Proc. 2001 Winter Simul. Conf."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1239\/aap\/1118858638"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10817-018-09509-5"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/1243991.1243995"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1287\/moor.1090.0389"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1137\/110842545"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-71618-1_80"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.7249\/RR2662"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/TII.2018.2822828"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1016\/0893-6080(94)E0045-M"},{"key":"e_1_3_2_52_2","volume-title":"Monte Carlo Methods in Financial Engineering","author":"Glasserman Paul","year":"2013","unstructured":"Paul Glasserman. 2013. Monte Carlo Methods in Financial Engineering. Vol. 53. Springer Science & Business Media, New York."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/9.736061"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1287\/opre.47.4.585"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1287\/opre.1080.0558"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.1050.0415"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoap\/1034801251"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.5555\/3217764.3217770"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.5555\/3086952"},{"key":"e_1_3_2_60_2","article-title":"Explaining and harnessing adversarial examples","author":"Goodfellow Ian J.","year":"2014","unstructured":"Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014).","journal-title":"arXiv preprint arXiv:1412.6572"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1239\/jap\/1409932671"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0010-4655(02)00205-9"},{"key":"e_1_3_2_63_2","first-page":"841","volume-title":"AMIA Annual Symposium Proceedings","volume":"2013","author":"Hagen Matthew S.","year":"2013","unstructured":"Matthew S. Hagen, Jeffrey K. Jopling, Timothy G. Buchman, and Eva K. Lee. 2013. Priority queuing models for hospital intensive care units and impacts to severe case patients. In AMIA Annual Symposium Proceedings, Vol. 2013. American Medical Informatics Association, 841."},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF02517804"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7_15"},{"key":"e_1_3_2_66_2","unstructured":"Hera Y. He and Art B. Owen. 2014. Optimal mixture weights in multiple importance sampling. (Nov. 2014). Retrieved from http:\/\/arxiv.org\/abs\/1411.3954."},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/203091.203094"},{"key":"e_1_3_2_68_2","unstructured":"Harsha Honnappa Raghu Pasupathy and Prateek Jaiswal. 2018. Dominating Points of Gaussian Extremes. arxiv:1810.12132 [math.PR]."},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2017.2766172"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2018.8632173"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2018.8632355"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/2133390.2133392"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0927-0507(06)13011-X"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tra.2016.09.010"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/90.251894"},{"key":"e_1_3_2_76_2","volume-title":"Next-generation Airborne Collision Avoidance System","author":"Kochenderfer Mykel J.","year":"2012","unstructured":"Mykel J. Kochenderfer, Jessica E. Holland, and James P. Chryssanthacopoulos. 2012. Next-generation Airborne Collision Avoidance System. Technical Report. Massachusetts Institute of Technology-Lincoln Laboratory, Lexington, MA."},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoap\/1029962748"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/MITS.2016.2583491"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-5316(99)00036-X"},{"key":"e_1_3_2_80_2","article-title":"Adversarial examples in the physical world","author":"Kurakin Alexey","year":"2016","unstructured":"Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 (2016).","journal-title":"arXiv preprint arXiv:1607.02533"},{"key":"e_1_3_2_81_2","article-title":"Adversarial machine learning at scale","author":"Kurakin Alexey","year":"2016","unstructured":"Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236 (2016).","journal-title":"arXiv preprint arXiv:1611.01236"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2012.2191542"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/1667072.1667078"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1002\/9780470745403.ch3"},{"key":"e_1_3_2_85_2","article-title":"Optimization of tree ensembles","author":"Mi\u0161ic Velibor V.","year":"2017","unstructured":"Velibor V. Mi\u0161ic. 2017. Optimization of tree ensembles. arXiv preprint arXiv:1705.10883 (2017).","journal-title":"arXiv preprint arXiv:1705.10883"},{"key":"e_1_3_2_86_2","first-page":"739","volume-title":"Conference on Advances in Neural Information Processing Systems","author":"Muller Urs","year":"2006","unstructured":"Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp, and Yann L. Cun. 2006. Off-road obstacle avoidance through end-to-end learning. In Conference on Advances in Neural Information Processing Systems. 739\u2013746."},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1287\/13-SSY114"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1109\/12.260634"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1109\/24.974122"},{"key":"e_1_3_2_90_2","first-page":"9827","volume-title":"Conference on Advances in Neural Information Processing Systems","author":"O\u2019Kelly Matthew","year":"2018","unstructured":"Matthew O\u2019Kelly, Aman Sinha, Hongseok Namkoong, Russ Tedrake, and John C. Duchi. 2018. Scalable end-to-end autonomous vehicle testing via rare-event simulation. In Conference on Advances in Neural Information Processing Systems. 9827\u20139838."},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1214\/18-EJS1527"},{"key":"e_1_3_2_92_2","first-page":"1666","article-title":"A large deviations perspective on the effiency of multilevel splitting","author":"Zajic P. Glasserman, P. Heidelberger, P. Shahabuddin, and T.","year":"1998","unstructured":"P. Glasserman, P. Heidelberger, P. Shahabuddin, and T. Zajic. 1998. A large deviations perspective on the effiency of multilevel splitting. IEEE Trans. Autom. Contr. 43, 12 (1998), 1666\u20131679.","journal-title":"IEEE Trans. Autom. Contr."},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1109\/9.8649"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01053"},{"key":"e_1_3_2_95_2","first-page":"1","article-title":"Rare-event simulation via cross-entropy and importance sampling","author":"Rubinstein R. Y.","year":"1999","unstructured":"R. Y. Rubinstein. 1999. Rare-event simulation via cross-entropy and importance sampling. In 2nd Workshop on Rare Event Simulation (RESIM\u201999). 1\u201317.","journal-title":"2nd Workshop on Rare Event Simulation (RESIM\u201999)"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2016.2621185"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2008.10.028"},{"key":"e_1_3_2_98_2","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1002\/9780470745403.ch6","volume-title":"Rare Event Simulation Using Monte Carlo Methods","author":"Rubino G.","year":"2009","unstructured":"G. Rubino and B. Tuffin. 2009. Markovian models for dependability analysis. In Rare Event Simulation Using Monte Carlo Methods. Wiley Publishing, 125\u2013144."},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-4321-0"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0377-2217(96)00385-2"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.5555\/3172929"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1109\/9.106154"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1109\/18.54903"},{"key":"e_1_3_2_104_2","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1002\/9780470745403.ch11","volume-title":"Rare Event Simulation Using Monte Carlo Methods","author":"Sandmann W.","year":"2009","unstructured":"W. Sandmann. 2009. Rare event simulation methodologies in systems biology. In Rare Event Simulation Using Monte Carlo Methods. Wiley Publishing, 243\u2013266."},{"key":"e_1_3_2_105_2","first-page":"1767","volume-title":"16th Symposium on Computational Statistics","author":"Savick\u1ef3 Petr","year":"2004","unstructured":"Petr Savick\u1ef3 and Emil Kotrc. 2004. Experimental study of leaf confidences for random forest. In 16th Symposium on Computational Statistics. 1767\u20131774."},{"key":"e_1_3_2_106_2","first-page":"2083","article-title":"Active estimation of f-measures","volume":"23","author":"Sawade Christoph","year":"2010","unstructured":"Christoph Sawade, Niels Landwehr, and Tobias Scheffer. 2010. Active estimation of f-measures. Adv. Neural Inf. Process. Syst. 23 (2010), 2083\u20132091.","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_107_2","first-page":"673","article-title":"Importance sampling in the Monte Carlo study of sequential tests","author":"Siegmund David","year":"1976","unstructured":"David Siegmund. 1976. Importance sampling in the Monte Carlo study of sequential tests. Ann. Statist. 4, 4 (1976), 673\u2013684.","journal-title":"Ann. Statist."},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aaw1975"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2002.1172912"},{"key":"e_1_3_2_110_2","article-title":"Verifying neural networks with mixed integer programming","author":"Tjeng Vincent","year":"2017","unstructured":"Vincent Tjeng and Russ Tedrake. 2017. Verifying neural networks with mixed integer programming. arXiv preprint arXiv:1711.07356 (2017).","journal-title":"arXiv preprint arXiv:1711.07356"},{"key":"e_1_3_2_111_2","doi-asserted-by":"publisher","DOI":"10.1109\/QEST.2004.1348030"},{"key":"e_1_3_2_112_2","article-title":"Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures","author":"Uesato Jonathan","year":"2018","unstructured":"Jonathan Uesato, Ananya Kumar, Csaba Szepesvari, Tom Erez, Avraham Ruderman, Keith Anderson, Nicolas Heess, Pushmeet Kohli, et\u00a0al. 2018. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. arXiv preprint arXiv:1812.01647 (2018).","journal-title":"arXiv preprint arXiv:1812.01647"},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.trc.2018.02.012"},{"key":"e_1_3_2_114_2","first-page":"1","volume-title":"Joint IBM\/University of Newcastle upon Tyne Seminar on Data Base Systems Seminar","author":"Rijsbergen C. Van","year":"1979","unstructured":"C. Van Rijsbergen. 1979. Information retrieval: Theory and practice. In Joint IBM\/University of Newcastle upon Tyne Seminar on Data Base Systems Seminar. 1\u201314."},{"key":"e_1_3_2_115_2","doi-asserted-by":"publisher","DOI":"10.1002\/cpa.21428"},{"key":"e_1_3_2_116_2","doi-asserted-by":"publisher","DOI":"10.1007\/0-306-48332-7_65"},{"key":"e_1_3_2_117_2","article-title":"Statistically robust neural network classification","author":"Wang Benjie","year":"2019","unstructured":"Benjie Wang, Stefan Webb, and Tom Rainforth. 2019. Statistically robust neural network classification. arXiv preprint arXiv:1912.04884 (2019).","journal-title":"arXiv preprint arXiv:1912.04884"},{"key":"e_1_3_2_118_2","article-title":"A statistical approach to assessing neural network robustness","author":"Webb Stefan","year":"2018","unstructured":"Stefan Webb, Tom Rainforth, Yee Whye Teh, and M. Pawan Kumar. 2018. A statistical approach to assessing neural network robustness. arXiv preprint arXiv:1811.07209 (2018).","journal-title":"arXiv preprint arXiv:1811.07209"},{"key":"e_1_3_2_119_2","article-title":"PROVEN: Certifying robustness of neural networks with a probabilistic approach","author":"Weng Tsui-Wei","year":"2018","unstructured":"Tsui-Wei Weng, Pin-Yu Chen, Lam M. Nguyen, Mark S. Squillante, Ivan Oseledets, and Luca Daniel. 2018. PROVEN: Certifying robustness of neural networks with a probabilistic approach. arXiv preprint arXiv:1812.08329 (2018).","journal-title":"arXiv preprint arXiv:1812.08329"},{"key":"e_1_3_2_120_2","first-page":"1523","volume-title":"Conference on Advances in Neural Information Processing Systems","author":"Wu Jianxin","year":"2004","unstructured":"Jianxin Wu, James M. Rehg, and Matthew D. Mullin. 2004. Learning a rare event detection cascade by direct feature selection. In Conference on Advances in Neural Information Processing Systems. 1523\u20131530."},{"key":"e_1_3_2_121_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2003.811769"},{"key":"e_1_3_2_122_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2017.2701846"},{"key":"e_1_3_2_123_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2016.2582208"}],"container-title":["ACM Transactions on Modeling and Computer Simulation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3519385","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3519385","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:05Z","timestamp":1750186805000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3519385"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,6]]},"references-count":122,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,7,31]]}},"alternative-id":["10.1145\/3519385"],"URL":"https:\/\/doi.org\/10.1145\/3519385","relation":{},"ISSN":["1049-3301","1558-1195"],"issn-type":[{"value":"1049-3301","type":"print"},{"value":"1558-1195","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,6]]},"assertion":[{"value":"2020-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-07-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}