{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T11:16:58Z","timestamp":1778498218740,"version":"3.51.4"},"reference-count":71,"publisher":"Springer Science and Business Media LLC","issue":"10","license":[{"start":{"date-parts":[[2022,2,2]],"date-time":"2022-02-02T00:00:00Z","timestamp":1643760000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,2,2]],"date-time":"2022-02-02T00:00:00Z","timestamp":1643760000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DGE-1106400"],"award-info":[{"award-number":["DGE-1106400"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1545126"],"award-info":[{"award-number":["CNS-1545126"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1646208"],"award-info":[{"award-number":["CNS-1646208"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1739816"],"award-info":[{"award-number":["CNS-1739816"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CCF-1837132"],"award-info":[{"award-number":["CCF-1837132"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["FA8750-16-C0043"],"award-info":[{"award-number":["FA8750-16-C0043"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["FA8750-20-C-0156"],"award-info":[{"award-number":["FA8750-20-C-0156"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Berkeley Deep Drive"},{"DOI":"10.13039\/100000028","name":"Semiconductor Research Corporation","doi-asserted-by":"publisher","award":["TerraSwarm"],"award-info":[{"award-number":["TerraSwarm"]}],"id":[{"id":"10.13039\/100000028","id-type":"DOI","asserted-by":"publisher"}]},{"name":"iCyPhy"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We propose a new probabilistic programming language for the design and analysis of cyber-physical systems, especially those based on machine learning. We consider several problems arising in the design process, including training a system to be robust to rare events, testing its performance under different conditions, and debugging failures. We show how a probabilistic programming language can help address these problems by specifying distributions encoding interesting types of inputs, then sampling these to generate specialized training and test data. More generally, such languages can be used to write environment models, an essential prerequisite to any formal analysis. In this paper, we focus on systems such as autonomous cars and robots, whose environment at any point in time is a <jats:italic>scene<\/jats:italic>, a configuration of physical objects and agents. We design a domain-specific language, <jats:sc>Scenic<\/jats:sc>, for describing <jats:italic>scenarios<\/jats:italic> that are distributions over scenes and the behaviors of their agents over time. <jats:sc>Scenic<\/jats:sc> combines concise, readable syntax for spatiotemporal relationships with the ability to declaratively impose hard and soft constraints over the scenario. We develop specialized techniques for sampling from the resulting distribution, taking advantage of the structure provided by <jats:sc>Scenic<\/jats:sc>\u2019s domain-specific syntax. Finally, we apply <jats:sc>Scenic<\/jats:sc> in multiple case studies for training, testing, and debugging neural networks for perception both as standalone components and within the context of a full cyber-physical system.\n<\/jats:p>","DOI":"10.1007\/s10994-021-06120-5","type":"journal-article","created":{"date-parts":[[2022,2,2]],"date-time":"2022-02-02T22:03:32Z","timestamp":1643839412000},"page":"3805-3849","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":73,"title":["Scenic: a language for scenario specification and data generation"],"prefix":"10.1007","volume":"112","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9992-9965","authenticated-orcid":false,"given":"Daniel J.","family":"Fremont","sequence":"first","affiliation":[]},{"given":"Edward","family":"Kim","sequence":"additional","affiliation":[]},{"given":"Tommaso","family":"Dreossi","sequence":"additional","affiliation":[]},{"given":"Shromona","family":"Ghosh","sequence":"additional","affiliation":[]},{"given":"Xiangyu","family":"Yue","sequence":"additional","affiliation":[]},{"given":"Alberto L.","family":"Sangiovanni-Vincentelli","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6190-8707","authenticated-orcid":false,"given":"Sanjit A.","family":"Seshia","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,2,2]]},"reference":[{"key":"6120_CR1","unstructured":"Amodei, D., Olah, C., Steinhardt, J., Christiano, P. F., Schulman, J., & Man\u00e9, D. (2016). Concrete problems in AI safety. CoRR. arXiv:abs\/1606.06565"},{"key":"6120_CR2","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1214\/09-SS054","volume":"4","author":"S Arlot","year":"2010","unstructured":"Arlot, S., & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40\u201379. https:\/\/doi.org\/10.1214\/09-SS054","journal-title":"Statistics Surveys"},{"key":"6120_CR3","unstructured":"Azad, A. S., Kim, E., Wu, Q., Lee, K., Stoica, I., Abbeel, P., Seshia, S. A. (2021). Scenic4rl: Programmatic modeling and generation of reinforcement learning environments. CoRR. arXiv:abs\/2106.10365"},{"key":"6120_CR4","unstructured":"Baidu. (2020). Apollo. https:\/\/apollo.auto\/"},{"key":"6120_CR5","doi-asserted-by":"crossref","unstructured":"Broy, M., Jonsson, B., Katoen, J. P., Leucker, M., & Pretschner, A. (2005). Model-based testing of reactive systems: Advanced lectures (lecture notes in computer science). Springer.","DOI":"10.1007\/b137241"},{"key":"6120_CR6","doi-asserted-by":"crossref","unstructured":"Claret, G., Rajamani, S. K., Nori, A. V., Gordon, A. D., & Borgstr\u00f6m, J. (2013). Bayesian inference using data flow analysis. In Proceedings of the 2013 9th joint meeting on foundations of software engineering (pp. 92\u2013102). ACM.","DOI":"10.1145\/2491411.2491423"},{"key":"6120_CR7","unstructured":"Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017). CARLA: An open urban driving simulator. In Conference on robot learning, CoRL (pp. 1\u201316)."},{"key":"6120_CR8","doi-asserted-by":"publisher","unstructured":"Dreossi, T., Donz\u00e9, A., & Seshia, S. A. (2017). Compositional falsification of cyber-physical systems with machine learning components. In NASA formal methods, NFM (pp. 357\u2013372). https:\/\/doi.org\/10.1007\/978-3-319-57288-8_26","DOI":"10.1007\/978-3-319-57288-8_26"},{"key":"6120_CR9","doi-asserted-by":"publisher","unstructured":"Dreossi, T., Fremont, D. J., Ghosh, S., Kim, E., Ravanbakhsh, H., Vazquez-Chanlatte, M., & Seshia, S. A. (2019). VerifAI: A toolkit for the formal design and analysis of artificial intelligence-based systems. In I. Dillig, & S. Tasiran (Eds.), Computer aided verification\u201431st international conference, CAV 2019, New York City, NY, USA, July 15\u201318, 2019, proceedings, part I, lecture notes in computer science (Vol. 11561, pp. 432\u2013442). Springer. https:\/\/doi.org\/10.1007\/978-3-030-25540-4_25","DOI":"10.1007\/978-3-030-25540-4_25"},{"key":"6120_CR10","doi-asserted-by":"publisher","unstructured":"Dreossi, T., Ghosh, S., Yue, X., Keutzer, K., Sangiovanni-Vincentelli, A. L., & Seshia, S. A. (2018). Counterexample-guided data augmentation. In J. Lang (Ed.), Proceedings of the 27th international joint conference on artificial intelligence, IJCAI 2018, July 13\u201319, 2018, Stockholm, Sweden (pp. 2071\u20132078). ijcai.org. https:\/\/doi.org\/10.24963\/ijcai.2018\/286","DOI":"10.24963\/ijcai.2018\/286"},{"key":"6120_CR11","doi-asserted-by":"crossref","unstructured":"Dreossi, T., Jha, S., & Seshia, S. A. (2018). Semantic adversarial deep learning. In 30th international conference on computer aided verification (CAV).","DOI":"10.1007\/978-3-319-96145-3_1"},{"key":"6120_CR12","unstructured":"DuPont. (2012). Global automotive color popularity report. https:\/\/web.archive.org\/web\/20130818022236\/http:\/\/www2.dupont.com\/Media_Center\/en_US\/color_popularity\/Images_2012\/DuPont2012ColorPopularity.pdf"},{"key":"6120_CR13","doi-asserted-by":"publisher","unstructured":"Elmas, T., Burnim, J., Necula, G., & Sen, K. (2013). CONCURRIT: A domain specific language for reproducing concurrency bugs. In: Proceedings of the 34th ACM SIGPLAN conference on programming language design and implementation, PLDI \u201913 (pp. 153\u2013164). Association for Computing Machinery. https:\/\/doi.org\/10.1145\/2491956.2462162","DOI":"10.1145\/2491956.2462162"},{"key":"6120_CR14","unstructured":"Filipowicz, A., Liu, J., & Kornhauser, A. (2017). Learning to recognize distance to stop signs using the virtual world of grand theft auto 5. Tech. rep., Princeton University."},{"key":"6120_CR15","doi-asserted-by":"crossref","unstructured":"Fisher, M., Ritchie, D., Savva, M., Funkhouser, T., & Hanrahan, P. (2012). Example-based synthesis of 3d object arrangements. In ACM SIGGRAPH 2012, SIGGRAPH Asia \u201912.","DOI":"10.1145\/2366145.2366154"},{"key":"6120_CR16","unstructured":"Foretellix. (2020). Measurable scenario description language. https:\/\/www.foretellix.com\/wp-content\/uploads\/2020\/07\/M-SDL_LRM_OS.pdf"},{"key":"6120_CR17","unstructured":"Fremont, D., Yue, X., Dreossi, T., Ghosh, S., Sangiovanni-Vincentelli, A. L., & Seshia, S. A. (2018). Scenic: Language-based scene generation. Tech. Rep. UCB\/EECS-2018-8, EECS Department, University of California. http:\/\/www2.eecs.berkeley.edu\/Pubs\/TechRpts\/2018\/EECS-2018-8.html"},{"key":"6120_CR18","unstructured":"Fremont, D. J. (2019). Algorithmic improvisation. Ph.D. thesis, University of California. https:\/\/escholarship.org\/uc\/item\/3812m6wx"},{"key":"6120_CR19","doi-asserted-by":"crossref","unstructured":"Fremont, D. J., Chiu, J., Margineantu, D. D., Osipychev, D., & Seshia, S. A. (2020). Formal analysis and redesign of a neural network-based aircraft taxiing system with VerifAI. In 32nd international conference on computer aided verification (CAV).","DOI":"10.1007\/978-3-030-53288-8_6"},{"key":"6120_CR20","unstructured":"Fremont, D. J., Donz\u00e9, A., Seshia, S. A., & Wessel, D. (2015). Control improvisation. In 35th IARCS annual conference on foundation of software technology and theoretical computer science (FSTTCS), LIPIcs (Vol.\u00a045, pp. 463\u2013474)."},{"key":"6120_CR21","doi-asserted-by":"publisher","unstructured":"Fremont, D. J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A. L., & Seshia, S. A. (2019). Scenic: A language for scenario specification and scene generation. In K. S. McKinley, & K.\u00a0Fisher (Eds.), Proceedings of the 40th ACM SIGPLAN conference on programming language design and implementation (PLDI) (pp. 63\u201378). ACM. https:\/\/doi.org\/10.1145\/3314221.3314633","DOI":"10.1145\/3314221.3314633"},{"key":"6120_CR22","doi-asserted-by":"crossref","unstructured":"Fremont, D. J., Kim, E., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A. L., & Seshia, S. A. (2020). Scenic: A language for scenario specification and data generation. https:\/\/arxiv.org\/abs\/2010.06580","DOI":"10.1145\/3314221.3314633"},{"key":"6120_CR23","doi-asserted-by":"crossref","unstructured":"Fremont, D. J., Kim, E., Pant, Y. V., Seshia, S. A., Acharya, A., Bruso, X., Wells, P., Lemke, S., Lu, Q., & Mehta, S. (2020). Formal scenario-based testing of autonomous vehicles: From simulation to the real world. In 2020 IEEE intelligent transportation systems conference, ITSC 2020 (pp. 913\u2013920). IEEE. arxiv:2003.07739","DOI":"10.1109\/ITSC45102.2020.9294368"},{"key":"6120_CR24","doi-asserted-by":"publisher","unstructured":"Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? the Kitti vision benchmark suite. In Computer vision and pattern recognition, CVPR (pp. 3354\u20133361). https:\/\/doi.org\/10.1109\/CVPR.2012.6248074","DOI":"10.1109\/CVPR.2012.6248074"},{"key":"6120_CR25","unstructured":"Goldberg, A., & Robson, D. (1983). Smalltalk-80: The language and its implementation. Addison-Wesley."},{"key":"6120_CR26","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672\u20132680)."},{"key":"6120_CR27","unstructured":"Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014). Explaining and harnessing adversarial examples. CoRR. arXiv:1412.6572"},{"key":"6120_CR28","unstructured":"Goodman, N., Mansinghka, V. K., Roy, D., Bonawitz, K., Tenenbaum, J. B. (2008). Church: A universal language for generative models. In Uncertainty in artificial intelligence 24 (UAI) (pp. 220\u2013229)."},{"key":"6120_CR29","unstructured":"Goodman, N. D., Stuhlm\u00fcller, A. (2014). The design and implementation of probabilistic programming languages. Retrieved July 11, 2018, from http:\/\/dippl.org"},{"key":"6120_CR30","doi-asserted-by":"crossref","unstructured":"Gordon, A. D., Henzinger, T. A., Nori, A. V., & Rajamani, S. K. (2014). Probabilistic programming. In FOSE 2014 (pp. 167\u2013181). ACM.","DOI":"10.1145\/2593882.2593900"},{"key":"6120_CR31","doi-asserted-by":"publisher","unstructured":"Gupta, A., Vedaldi, A., & Zisserman, A. (2016). Synthetic data for text localisation in natural images. In Computer vision and pattern recognition, CVPR (pp. 2315\u20132324). https:\/\/doi.org\/10.1109\/CVPR.2016.254","DOI":"10.1109\/CVPR.2016.254"},{"key":"6120_CR32","unstructured":"Jaderberg, M., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014). Synthetic data and artificial neural networks for natural scene text recognition. CoRR. arXiv:abs\/1406.2227"},{"key":"6120_CR33","doi-asserted-by":"crossref","unstructured":"Jiang, C., Qi, S., Zhu, Y., Huang, S., Lin, J., Yu, L. F., Terzopoulos, D., & Zhu, S. C. (2018). Configurable 3d scene synthesis and 2d image rendering with per-pixel ground truth using stochastic grammars. International Journal of Computer Vision, 1\u201322.","DOI":"10.1007\/s11263-018-1103-5"},{"key":"6120_CR34","doi-asserted-by":"publisher","unstructured":"Johnson-Roberson, M., Barto, C., Mehta, R., Sridhar, S. N., Rosaen, K., & Vasudevan, R. (2017). Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? In International conference on robotics and automation, ICRA (pp. 746\u2013753). https:\/\/doi.org\/10.1109\/ICRA.2017.7989092","DOI":"10.1109\/ICRA.2017.7989092"},{"key":"6120_CR35","unstructured":"Jung, A. (2018). imgaug. https:\/\/github.com\/aleju\/imgaug"},{"key":"6120_CR36","doi-asserted-by":"publisher","unstructured":"Kim, E., Gopinath, D., Pasareanu, C. S., & Seshia, S. A. (2020). A programmatic and semantic approach to explaining and debugging neural network based object detectors. In 2020 IEEE\/CVF conference on computer vision and pattern recognition, CVPR 2020 (pp. 11125\u201311134). IEEE. https:\/\/doi.org\/10.1109\/CVPR42600.2020.01114","DOI":"10.1109\/CVPR42600.2020.01114"},{"issue":"4","key":"6120_CR37","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1007\/BF01995674","volume":"2","author":"R Koymans","year":"1990","unstructured":"Koymans, R. (1990). Specifying real-time properties with metric temporal logic. Real-Time Systems, 2(4), 255\u2013299.","journal-title":"Real-Time Systems"},{"key":"6120_CR38","doi-asserted-by":"crossref","unstructured":"Kulkarni, T., Kohli, P., Tenenbaum, J. B., & Mansinghka, V. K. (2015). Picture: A probabilistic programming language for scene perception. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 4390\u20134399).","DOI":"10.1109\/CVPR.2015.7299068"},{"key":"6120_CR39","unstructured":"Laminar Research. (2019). X-plane 11. https:\/\/www.x-plane.com\/"},{"key":"6120_CR40","doi-asserted-by":"crossref","unstructured":"Liang, X., Hu, Z., Zhang, H., Gan, C., & Xing, E. P. (2017). Recurrent topic-transition GAN for visual paragraph generation. ArXiv preprint. arXiv:1703.07022","DOI":"10.1109\/ICCV.2017.364"},{"key":"6120_CR41","doi-asserted-by":"publisher","unstructured":"Liebelt, J., & Schmid, C. (2010). Multi-view object class detection with a 3d geometric model. In Computer vision and pattern recognition, CVPR (pp. 1688\u20131695). https:\/\/doi.org\/10.1109\/CVPR.2010.5539836","DOI":"10.1109\/CVPR.2010.5539836"},{"key":"6120_CR42","unstructured":"Majumdar, R., Mathur, A. S., Pirron, M., Stegner, L., & Zufferey, D. (2019). Paracosm: A language and tool for testing autonomous driving systems. CoRR. arxiv:1902.01084"},{"key":"6120_CR43","unstructured":"Marchesi, M. (2017). Megapixel size image creation using generative adversarial networks. ArXiv preprint (2017). arXiv:1706.00082"},{"issue":"1","key":"6120_CR44","doi-asserted-by":"publisher","first-page":"39","DOI":"10.5772\/5618","volume":"1","author":"O Michel","year":"2004","unstructured":"Michel, O. (2004). Webots: Professional mobile robot simulation. International Journal of Advanced Robotic Systems, 1(1), 39\u201342.","journal-title":"International Journal of Advanced Robotic Systems"},{"key":"6120_CR45","unstructured":"Milch, B., Marthi, B., & Russell, S. (2004). Blog: Relational modeling with unknown objects. In ICML 2004 workshop on statistical relational learning and its connections to other fields (pp. 67\u201373)."},{"key":"6120_CR46","doi-asserted-by":"publisher","unstructured":"Moosavi-Dezfooli, S., Fawzi, A., & Frossard, P. (2016). Deepfool: A simple and accurate method to fool deep neural networks. In Computer Vision and Pattern Recognition, CVPR (pp. 2574\u20132582). https:\/\/doi.org\/10.1109\/CVPR.2016.282","DOI":"10.1109\/CVPR.2016.282"},{"issue":"3","key":"6120_CR47","doi-asserted-by":"publisher","first-page":"614","DOI":"10.1145\/1141911.1141931","volume":"25","author":"P M\u00fcller","year":"2006","unstructured":"M\u00fcller, P., Wonka, P., Haegler, S., Ulmer, A., & Gool, L. V. (2006). Procedural modeling of buildings. ACM Transactions Graphics, 25(3), 614\u2013623. https:\/\/doi.org\/10.1145\/1141911.1141931","journal-title":"ACM Transactions Graphics"},{"key":"6120_CR48","unstructured":"Naveh, Y., Rimon, M., Jaeger, I., Katz, Y., Vinov, M., Marcus, E., & Shurek, G. (2006). Constraint-based random stimuli generation for hardware verification. In Proc. of AAAI (pp. 1720\u20131727)."},{"key":"6120_CR49","doi-asserted-by":"publisher","unstructured":"Nguyen, A. M., Yosinski, J., & Clune, J. (2015). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Computer vision and pattern recognition, CVPR (pp. 427\u2013436). https:\/\/doi.org\/10.1109\/CVPR.2015.7298640","DOI":"10.1109\/CVPR.2015.7298640"},{"key":"6120_CR50","doi-asserted-by":"crossref","unstructured":"Nori, A. V., Hur, C. K., Rajamani, S. K., & Samuel, S. (2014). R2: An efficient MCMC sampler for probabilistic programs. In AAAI (pp. 2476\u20132482).","DOI":"10.1609\/aaai.v28i1.9060"},{"key":"6120_CR51","doi-asserted-by":"publisher","unstructured":"Pei, K., Cao, Y., Yang, J., & Jana, S. (2017). Deepxplore: Automated whitebox testing of deep learning systems. In Symposium on operating systems principles, SOSP (pp. 1\u201318). https:\/\/doi.org\/10.1145\/3132747.3132785","DOI":"10.1145\/3132747.3132785"},{"key":"6120_CR52","unstructured":"Ritchie, D. (2014). Quicksand: A lightweight embedding of probabilistic programming for procedural modeling and design. In 3rd NIPS workshop on probabilistic programming. https:\/\/dritchie.github.io\/pdf\/qs.pdf"},{"key":"6120_CR53","unstructured":"Ritchie, D. (2016). Probabilistic programming for procedural modeling and design. Ph.D. thesis, Stanford University. https:\/\/purl.stanford.edu\/vh730bw6700"},{"key":"6120_CR54","unstructured":"Rockstar Games. (2015). Grand theft auto v. Windows PC version. https:\/\/www.rockstargames.com\/games\/info\/V"},{"key":"6120_CR55","doi-asserted-by":"crossref","unstructured":"Rong, G., Shin, B. H., Tabatabaee, H., Lu, Q., Lemke, S., Mo\u017eeiko, M., Boise, E., Uhm, G., Gerow, M., Mehta, S., Agafonov, E., Kim, T. H., Sterner, E., Ushiroda, K., Reyes, M., Zelenkovsky, D., Kim, S. (2020). LGSVL simulator: A high fidelity simulator for autonomous driving. arxiv:2005.03778","DOI":"10.1109\/ITSC45102.2020.9294422"},{"key":"6120_CR56","doi-asserted-by":"publisher","unstructured":"Ros, G., Sellart, L., Materzynska, J., V\u00e1zquez, D., & L\u00f3pez, A. M. (2016). The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Computer vision and pattern recognition, CVPR (pp. 3234\u20133243). https:\/\/doi.org\/10.1109\/CVPR.2016.352","DOI":"10.1109\/CVPR.2016.352"},{"key":"6120_CR57","doi-asserted-by":"publisher","unstructured":"Rubinstein, R. Y., & Kroese, D. P. (2004). The cross-entropy method: A unified approach to combinatorial optimization, Monte-Carlo simulation, and machine learning. Springer. https:\/\/doi.org\/10.1007\/978-1-4757-4321-0","DOI":"10.1007\/978-1-4757-4321-0"},{"key":"6120_CR58","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1609\/aimag.v36i4.2577","volume":"36","author":"S Russell","year":"2015","unstructured":"Russell, S., Dietterich, T., Horvitz, E., Selman, B., Rossi, F., Hassabis, D., Legg, S., Suleyman, M., George, D., & Phoenix, S. (2015). Letter to the editor: Research priorities for robust and beneficial artificial intelligence: An open letter. AI Magazine, 36, 4.","journal-title":"AI Magazine"},{"key":"6120_CR59","doi-asserted-by":"crossref","unstructured":"Saheb-Djahromi, N. (1978). Probabilistic LCF. In Mathematical foundations of computer science (pp. 442\u2013451). Springer.","DOI":"10.1007\/3-540-08921-7_92"},{"key":"6120_CR60","unstructured":"Seshia, S. A., Sadigh, D., & Sastry, S. S. (2016). Towards verified artificial intelligence. ArXiv e-prints."},{"key":"6120_CR61","doi-asserted-by":"publisher","unstructured":"Stark, M., Goesele, M., & Schiele, B. (2010). Back to the future: Learning shape models from 3d CAD data. In British machine vision conference, BMVC (pp. 1\u201311). https:\/\/doi.org\/10.5244\/C.24.106","DOI":"10.5244\/C.24.106"},{"key":"6120_CR62","unstructured":"Sutton, M., Greene, A., & Amini, P. (2007). Fuzzing: Brute force vulnerability discovery. Addison-Wesley."},{"key":"6120_CR63","unstructured":"Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2014). Intriguing properties of neural networks. In International conference on learning representations (ICLR)."},{"key":"6120_CR64","unstructured":"Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., & Fergus, R. (2013). Intriguing properties of neural networks. CoRR. arxiv:1312.6199"},{"key":"6120_CR65","unstructured":"Thorn, E., Kimmel, S., & Chaka, M. (2018). A framework for automated driving system testable cases and scenarios. Tech. Rep. DOT HS 812 623, National Highway Traffic Safety Administration. https:\/\/www.nhtsa.gov\/sites\/nhtsa.dot.gov\/files\/documents\/13882-automateddrivingsystems_092618_v1a_tag.pdf"},{"key":"6120_CR66","doi-asserted-by":"publisher","unstructured":"Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. In International conference on intelligent robots and systems, IROS (pp. 23\u201330). https:\/\/doi.org\/10.1109\/IROS.2017.8202133","DOI":"10.1109\/IROS.2017.8202133"},{"issue":"4","key":"6120_CR67","doi-asserted-by":"publisher","first-page":"797","DOI":"10.1109\/TPAMI.2013.163","volume":"36","author":"D Vazquez","year":"2014","unstructured":"Vazquez, D., Lopez, A. M., Marin, J., Ponsa, D., & Geronimo, D. (2014). Virtual and real world adaptation for pedestrian detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(4), 797\u2013809.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"6120_CR68","doi-asserted-by":"crossref","unstructured":"Wong, S. C., Gatt, A., Stamatescu, V., & McDonnell, M. D. (2016). Understanding data augmentation for classification: when to warp? In Digital image computing: Techniques and applications (DICTA), 2016 international conference on (pp. 1\u20136). IEEE.","DOI":"10.1109\/DICTA.2016.7797091"},{"key":"6120_CR69","unstructured":"Wood, F., Meent, J. W., & Mansinghka, V. (2014). A new approach to probabilistic programming inference. In Artificial intelligence and statistics (pp. 1024\u20131032)."},{"key":"6120_CR70","doi-asserted-by":"publisher","unstructured":"Wu, B., Iandola, F. N., Jin, P. H., & Keutzer, K. (2017). Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. In Conference on computer vision and pattern recognition workshops, CVPR workshops (pp. 446\u2013454). https:\/\/doi.org\/10.1109\/CVPRW.2017.60","DOI":"10.1109\/CVPRW.2017.60"},{"key":"6120_CR71","unstructured":"Xu, Y., Jia, R., Mou, L., Li, G., Chen, Y., Lu, Y., & Jin, Z. (2016). Improved relation classification by deep recurrent neural networks with data augmentation. ArXiv preprint. arXiv:1601.03651"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06120-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-021-06120-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06120-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,7]],"date-time":"2023-09-07T22:04:46Z","timestamp":1694124286000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-021-06120-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,2]]},"references-count":71,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2023,10]]}},"alternative-id":["6120"],"URL":"https:\/\/doi.org\/10.1007\/s10994-021-06120-5","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,2]]},"assertion":[{"value":"22 September 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 October 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 October 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 February 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"DJF is a guest editor of the special issue in which this article appears; he was recused from all matters related to this paper. Otherwise, the authors have no conflicts of interest to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"The <scp>Scenic<\/scp> implementation is open-source; its code, as well as code for all <scp>Scenic<\/scp> programs used in this paper, can be found at .","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Code availability"}}]}}