{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T23:38:12Z","timestamp":1775259492564,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":184,"publisher":"ACM","license":[{"start":{"date-parts":[[2018,10,26]],"date-time":"2018-10-26T00:00:00Z","timestamp":1540512000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CCF-1703637,CCF-1629431,CCF-1421503"],"award-info":[{"award-number":["CCF-1703637,CCF-1629431,CCF-1421503"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2018,10,26]]},"DOI":"10.1145\/3236024.3236057","type":"proceedings-article","created":{"date-parts":[[2018,10,29]],"date-time":"2018-10-29T08:06:00Z","timestamp":1540800360000},"page":"574-586","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":41,"title":["Testing probabilistic programming systems"],"prefix":"10.1145","author":[{"given":"Saikat","family":"Dutta","sequence":"first","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Owolabi","family":"Legunsen","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zixin","family":"Huang","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sasa","family":"Misailovic","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,10,26]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Eric Atkinson and Michael Carbin. 2016. Eric Atkinson and Michael Carbin. 2016."},{"key":"e_1_3_2_1_2_1","volume-title":"NIPS Workshop on Machine Learning Systems.","author":"Towards","unstructured":"Towards correct-by-construction probabilistic inference . In NIPS Workshop on Machine Learning Systems. Towards correct-by-construction probabilistic inference. In NIPS Workshop on Machine Learning Systems."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1882291.1882308"},{"key":"e_1_3_2_1_4_1","unstructured":"Antoine Balestrat. 2018. CCG - random C Code Generator. https:\/\/github.com\/ Mrktn\/ccg. Antoine Balestrat. 2018. CCG - random C Code Generator. https:\/\/github.com\/ Mrktn\/ccg."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.59"},{"key":"e_1_3_2_1_6_1","unstructured":"Moritz Beller Radjino Bholanath Shane McIntosh and Andy Zaidman. 2016. Moritz Beller Radjino Bholanath Shane McIntosh and Andy Zaidman. 2016."},{"key":"e_1_3_2_1_7_1","unstructured":"Analyzing the state of static analysis: A large-scale evaluation in open source software. In SANER. Analyzing the state of static analysis: A large-scale evaluation in open source software. In SANER."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1646353.1646374"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786832"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Johannes Borgstr\u00f6m Andrew D Gordon Michael Greenberg James Margetson and Jurgen Van Gael. 2011. Measure transformer semantics for Bayesian machine learning. In ESOP. Johannes Borgstr\u00f6m Andrew D Gordon Michael Greenberg James Margetson and Jurgen Van Gael. 2011. Measure transformer semantics for Bayesian machine learning. In ESOP.","DOI":"10.1007\/978-3-642-19718-5_5"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Bob Carpenter. 2017. Bob Carpenter. 2017.","DOI":"10.1111\/rssa.12319"},{"key":"e_1_3_2_1_12_1","unstructured":"https:\/\/github.com\/standev\/stan\/issues\/603. https:\/\/github.com\/standev\/stan\/issues\/603."},{"key":"e_1_3_2_1_13_1","unstructured":"Bob Carpenter Andrew Gelman Matt Hoffman Daniel Lee Ben Goodrich Michael Betancourt Michael A Brubaker Jiqiang Guo Peter Li Allen Riddell etal 2016. Bob Carpenter Andrew Gelman Matt Hoffman Daniel Lee Ben Goodrich Michael Betancourt Michael A Brubaker Jiqiang Guo Peter Li Allen Riddell et al. 2016."},{"key":"e_1_3_2_1_14_1","first-page":"2","volume":"20","author":"A","year":"2016","unstructured":"Stan: A probabilistic programming language. JSTATSOFT 20 , 2 ( 2016 ). Stan: A probabilistic programming language. JSTATSOFT 20, 2 (2016).","journal-title":"JSTATSOFT"},{"key":"e_1_3_2_1_15_1","unstructured":"Tsong Y Chen Shing C Cheung and Siu Ming Yiu. 1998. Tsong Y Chen Shing C Cheung and Siu Ming Yiu. 1998."},{"key":"e_1_3_2_1_17_1","unstructured":"Tsong Yueh Chen Jianqiang Feng and TH Tse. 2002. Metamorphic testing of programs on partial differential equations: a case study. In COMPSAC. Tsong Yueh Chen Jianqiang Feng and TH Tse. 2002. Metamorphic testing of programs on partial differential equations: a case study. In COMPSAC."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491956.2462173"},{"key":"e_1_3_2_1_19_1","unstructured":"Yuting Chen Ting Su Chengnian Sun Zhendong Su and Jianjun Zhao. 2016. Yuting Chen Ting Su Chengnian Sun Zhendong Su and Jianjun Zhao. 2016."},{"key":"e_1_3_2_1_20_1","unstructured":"Coverage-directed differential testing of JVM implementations. In PLDI. Coverage-directed differential testing of JVM implementations. In PLDI."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2010.5609723"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2491423"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(90)90060-D"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1287624.1287651"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/3155562.3155627"},{"key":"e_1_3_2_1_26_1","unstructured":"Domain Error in arguments 2018. Domain Error in arguments 2018."},{"key":"e_1_3_2_1_27_1","unstructured":"https:\/\/github.com\/uber\/pyro\/issues\/875. https:\/\/github.com\/uber\/pyro\/issues\/875."},{"key":"e_1_3_2_1_28_1","unstructured":"Edward Commit 002a27e 2016. Edward Commit 002a27e 2016."},{"key":"e_1_3_2_1_29_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/commit\/ 002a27e. https:\/\/github.com\/bleilab\/edward\/commit\/ 002a27e."},{"key":"e_1_3_2_1_30_1","unstructured":"Edward Commit 10118db 2016. Edward Commit 10118db 2016."},{"key":"e_1_3_2_1_31_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/commit\/ 10118db. https:\/\/github.com\/bleilab\/edward\/commit\/ 10118db."},{"key":"e_1_3_2_1_32_1","unstructured":"Edward Commit 3616d41 2016. Edward Commit 3616d41 2016."},{"key":"e_1_3_2_1_33_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/commit\/ 3616d41. https:\/\/github.com\/bleilab\/edward\/commit\/ 3616d41."},{"key":"e_1_3_2_1_34_1","unstructured":"Edward commit 3616d41 2016. Edward commit 3616d41 2016."},{"key":"e_1_3_2_1_35_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/commit\/ 3616d41. https:\/\/github.com\/bleilab\/edward\/commit\/ 3616d41."},{"key":"e_1_3_2_1_36_1","unstructured":"Edward commit 43d8a39 2016. Edward commit 43d8a39 2016."},{"key":"e_1_3_2_1_37_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/commit\/ 43d8a39. https:\/\/github.com\/bleilab\/edward\/commit\/ 43d8a39."},{"key":"e_1_3_2_1_38_1","unstructured":"Edward Commit 79f4193 2017. Edward Commit 79f4193 2017."},{"key":"e_1_3_2_1_39_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/commit\/ 79f4193. https:\/\/github.com\/bleilab\/edward\/commit\/ 79f4193."},{"key":"e_1_3_2_1_40_1","unstructured":"Edward commit 79f4193 2017. Edward commit 79f4193 2017."},{"key":"e_1_3_2_1_41_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/commit\/ 79f4193. https:\/\/github.com\/bleilab\/edward\/commit\/ 79f4193."},{"key":"e_1_3_2_1_42_1","unstructured":"Edward Commit 972a9d9 2017. Edward Commit 972a9d9 2017."},{"key":"e_1_3_2_1_43_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/commit\/ 972a9d9. https:\/\/github.com\/bleilab\/edward\/commit\/ 972a9d9."},{"key":"e_1_3_2_1_44_1","unstructured":"Edward Commit c9afcb1f 2017. Edward Commit c9afcb1f 2017."},{"key":"e_1_3_2_1_45_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/commit\/ c9afcb1. https:\/\/github.com\/bleilab\/edward\/commit\/ c9afcb1."},{"key":"e_1_3_2_1_46_1","unstructured":"Edward Commit fe01657 2016. Edward Commit fe01657 2016."},{"key":"e_1_3_2_1_47_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/commit\/ fe01657. https:\/\/github.com\/bleilab\/edward\/commit\/ fe01657."},{"key":"e_1_3_2_1_48_1","unstructured":"EdwardWebPage 2018. Edward. http:\/\/edwardlib.org. EdwardWebPage 2018. Edward. http:\/\/edwardlib.org."},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"crossref","unstructured":"Antonio Filieri Corina S P\u0103s\u0103reanu and Willem Visser. 2013. Reliability analysis in symbolic pathfinder. In ICSE. Antonio Filieri Corina S P\u0103s\u0103reanu and Willem Visser. 2013. Reliability analysis in symbolic pathfinder. In ICSE.","DOI":"10.1109\/ICSE.2013.6606608"},{"key":"e_1_3_2_1_50_1","unstructured":"Timon Gehr Sasa Misailovic and Martin Vechev. 2016. Timon Gehr Sasa Misailovic and Martin Vechev. 2016."},{"key":"e_1_3_2_1_51_1","unstructured":"PSI : Exact Symbolic Inference for Probabilistic Programs. In CAV. PSI: Exact Symbolic Inference for Probabilistic Programs. In CAV."},{"key":"e_1_3_2_1_52_1","unstructured":"Jaco Geldenhuys Matthew B Dwyer and Willem Visser. 2012. Jaco Geldenhuys Matthew B Dwyer and Willem Visser. 2012."},{"key":"e_1_3_2_1_53_1","unstructured":"Probabilistic symbolic execution. In ISSTA. Probabilistic symbolic execution. In ISSTA."},{"key":"e_1_3_2_1_54_1","volume-title":"Stan A Probabilistic Programming Language for Bayesian Inference and Optimization. Journal of Educational and Behavioral Statistics","author":"Gelman Andrew","year":"2015","unstructured":"Andrew Gelman , Daniel Lee , and Jiqiang Guo . 2015. Stan A Probabilistic Programming Language for Bayesian Inference and Optimization. Journal of Educational and Behavioral Statistics ( 2015 ). Andrew Gelman, Daniel Lee, and Jiqiang Guo. 2015. Stan A Probabilistic Programming Language for Bayesian Inference and Optimization. Journal of Educational and Behavioral Statistics (2015)."},{"key":"e_1_3_2_1_55_1","volume-title":"A language and program for complex Bayesian modelling. The Statistician","author":"Gilks Wally R","year":"1994","unstructured":"Wally R Gilks , Andrew Thomas , and David J Spiegelhalter . 1994. A language and program for complex Bayesian modelling. The Statistician ( 1994 ). Wally R Gilks, Andrew Thomas, and David J Spiegelhalter. 1994. A language and program for complex Bayesian modelling. The Statistician (1994)."},{"key":"e_1_3_2_1_56_1","unstructured":"Milos Gligoric Tihomir Gvero Vilas Jagannath Sarfraz Khurshid Viktor Kuncak and Darko Marinov. 2010. Milos Gligoric Tihomir Gvero Vilas Jagannath Sarfraz Khurshid Viktor Kuncak and Darko Marinov. 2010."},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1806799.1806835"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1375581.1375607"},{"key":"e_1_3_2_1_59_1","unstructured":"Patrice Godefroid Michael Y Levin David A Molnar etal 2008. Automated whitebox fuzz testing.. In NDSS. Patrice Godefroid Michael Y Levin David A Molnar et al. 2008. Automated whitebox fuzz testing.. In NDSS."},{"key":"e_1_3_2_1_60_1","volume-title":"Church: a language for generative models. arXiv preprint arXiv:1206.3255","author":"Goodman Noah","year":"2012","unstructured":"Noah Goodman , Vikash Mansinghka , Daniel M Roy , Keith Bonawitz , and Joshua B Tenenbaum . 2012. Church: a language for generative models. arXiv preprint arXiv:1206.3255 ( 2012 ). Noah Goodman, Vikash Mansinghka, Daniel M Roy, Keith Bonawitz, and Joshua B Tenenbaum. 2012. Church: a language for generative models. arXiv preprint arXiv:1206.3255 (2012)."},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593882.2593900"},{"key":"e_1_3_2_1_62_1","unstructured":"Arnaud Gotlieb and Bernard Botella. 2003. Automated metamorphic testing. In COMPSAC. Arnaud Gotlieb and Bernard Botella. 2003. Automated metamorphic testing. In COMPSAC."},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2007.68"},{"key":"e_1_3_2_1_64_1","unstructured":"Roger B Grosse Siddharth Ancha and Daniel M Roy. 2016. Roger B Grosse Siddharth Ancha and Daniel M Roy. 2016."},{"key":"e_1_3_2_1_65_1","unstructured":"Measuring the reliability of MCMC inference with bidirectional Monte Carlo. In NIPS. Measuring the reliability of MCMC inference with bidirectional Monte Carlo. In NIPS."},{"key":"e_1_3_2_1_66_1","doi-asserted-by":"crossref","unstructured":"Farah Hariri August Shi Owolabi Legunsen Milos Gligoric Sarfraz Khurshid and Sasa Misailovic. 2018. Approximate Transformations as Mutation Operators. In ICST. Farah Hariri August Shi Owolabi Legunsen Milos Gligoric Sarfraz Khurshid and Sasa Misailovic. 2018. Approximate Transformations as Mutation Operators. In ICST.","DOI":"10.1109\/ICST.2018.00036"},{"key":"e_1_3_2_1_67_1","unstructured":"Shawn Hershey Jeff Bernstein Bill Bradley Andrew Schweitzer Noah Stein Theo Weber and Ben Vigoda. 2012. Shawn Hershey Jeff Bernstein Bill Bradley Andrew Schweitzer Noah Stein Theo Weber and Ben Vigoda. 2012."},{"key":"e_1_3_2_1_68_1","volume-title":"towards a full language, compiler and hardware stack. arXiv preprint arXiv:1212.2991","author":"Accelerating","year":"2012","unstructured":"Accelerating inference : towards a full language, compiler and hardware stack. arXiv preprint arXiv:1212.2991 ( 2012 ). Accelerating inference: towards a full language, compiler and hardware stack. arXiv preprint arXiv:1212.2991 (2012)."},{"key":"e_1_3_2_1_69_1","unstructured":"Christian Holler Kim Herzig and Andreas Zeller. 2012. Christian Holler Kim Herzig and Andreas Zeller. 2012."},{"key":"e_1_3_2_1_70_1","unstructured":"Fuzzing with Code Fragments. In USENIX Security. Fuzzing with Code Fragments. In USENIX Security."},{"key":"e_1_3_2_1_71_1","unstructured":"Daniel Jackson and Craig A Damon. 1996. Daniel Jackson and Craig A Damon. 1996."},{"key":"e_1_3_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.538605"},{"key":"e_1_3_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-00593-0_12"},{"key":"e_1_3_2_1_74_1","volume-title":"Joost-Pieter Katoen, and Lukas Westhofen.","author":"Jansen Nils","year":"2016","unstructured":"Nils Jansen , Christian Dehnert , Benjamin Lucien Kaminski , Joost-Pieter Katoen, and Lukas Westhofen. 2016 . Bounded Model Checking for Probabilistic Programs. ATVA ( 2016). Nils Jansen, Christian Dehnert, Benjamin Lucien Kaminski, Joost-Pieter Katoen, and Lukas Westhofen. 2016. Bounded Model Checking for Probabilistic Programs. ATVA (2016)."},{"key":"e_1_3_2_1_75_1","volume-title":"Blei","author":"Kucukelbir Alp","year":"2015","unstructured":"Alp Kucukelbir , Rajesh Ranganath , Andrew Gelman , and David M . Blei . 2015 . Alp Kucukelbir, Rajesh Ranganath, Andrew Gelman, and David M. Blei. 2015."},{"key":"e_1_3_2_1_76_1","unstructured":"Automatic Variational Inference in Stan. In NIPS. Automatic Variational Inference in Stan. In NIPS."},{"key":"e_1_3_2_1_77_1","unstructured":"Vu Le Mehrdad Afshari and Zhendong Su. 2014. Vu Le Mehrdad Afshari and Zhendong Su. 2014."},{"key":"e_1_3_2_1_78_1","unstructured":"Compiler Validation via Equivalence Modulo Inputs. In PLDI. Compiler Validation via Equivalence Modulo Inputs. In PLDI."},{"key":"e_1_3_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/2814270.2814319"},{"key":"e_1_3_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/2771783.2771785"},{"key":"e_1_3_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390832.1390837"},{"key":"e_1_3_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/2737924.2737986"},{"key":"e_1_3_2_1_83_1","doi-asserted-by":"crossref","unstructured":"Daniel Liew Daniel Schemmel Cristian Cadar Alastair F Donaldson Rafael Z\u00e4hl and Klaus Wehrle. 2017. Floating-point symbolic execution: A case study in N-version programming. In ASE. Daniel Liew Daniel Schemmel Cristian Cadar Alastair F Donaldson Rafael Z\u00e4hl and Klaus Wehrle. 2017. Floating-point symbolic execution: A case study in N-version programming. In ASE.","DOI":"10.1109\/ASE.2017.8115670"},{"key":"e_1_3_2_1_84_1","unstructured":"Huai Liu Fei-Ching Kuo Dave Towey and Tsong Yueh Chen. 2014. Huai Liu Fei-Ching Kuo Dave Towey and Tsong Yueh Chen. 2014."},{"key":"e_1_3_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.46"},{"key":"e_1_3_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2643011"},{"key":"e_1_3_2_1_87_1","unstructured":"Rupak Majumdar and Ru-Gang Xu. 2007. Rupak Majumdar and Ru-Gang Xu. 2007."},{"key":"e_1_3_2_1_88_1","unstructured":"Directed Test Generation Using Symbolic Grammars. In ASE. Directed Test Generation Using Symbolic Grammars. In ASE."},{"key":"e_1_3_2_1_89_1","volume-title":"Venture: a higherorder probabilistic programming platform with programmable inference. arXiv preprint 1404.0099","author":"Mansinghka Vikash","year":"2014","unstructured":"Vikash Mansinghka , Daniel Selsam , and Yura Perov . 2014. Venture: a higherorder probabilistic programming platform with programmable inference. arXiv preprint 1404.0099 ( 2014 ). Vikash Mansinghka, Daniel Selsam, and Yura Perov. 2014. Venture: a higherorder probabilistic programming platform with programmable inference. arXiv preprint 1404.0099 (2014)."},{"key":"e_1_3_2_1_90_1","unstructured":"Piotr Mardziel Stephen Magill Michael Hicks and Mudhakar Srivatsa. 2013. Piotr Mardziel Stephen Magill Michael Hicks and Mudhakar Srivatsa. 2013."},{"key":"e_1_3_2_1_91_1","first-page":"4","volume":"21","author":"Dynamic","year":"2013","unstructured":"Dynamic enforcement of knowledge-based security policies using probabilistic abstract interpretation. JCS 21 , 4 ( 2013 ). Dynamic enforcement of knowledge-based security policies using probabilistic abstract interpretation. JCS 21, 4 (2013).","journal-title":"JCS"},{"key":"e_1_3_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1109\/52.56422"},{"key":"e_1_3_2_1_93_1","unstructured":"William M McKeeman. 1998. William M McKeeman. 1998."},{"key":"e_1_3_2_1_94_1","first-page":"1","volume":"10","author":"Differential","year":"1998","unstructured":"Differential testing for software. DEC Digital Technical Journal 10 , 1 ( 1998 ). Differential testing for software. DEC Digital Technical Journal 10, 1 (1998).","journal-title":"DEC Digital Technical Journal"},{"key":"e_1_3_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1145\/96267.96279"},{"key":"e_1_3_2_1_96_1","doi-asserted-by":"crossref","unstructured":"David Monniaux. 2000. Abstract interpretation of probabilistic semantics. In SAS. David Monniaux. 2000. Abstract interpretation of probabilistic semantics. In SAS.","DOI":"10.1007\/978-3-540-45099-3_17"},{"key":"e_1_3_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1145\/229542.229547"},{"key":"e_1_3_2_1_98_1","unstructured":"Christian Murphy and Gail E Kaiser. 2010. Christian Murphy and Gail E Kaiser. 2010."},{"key":"e_1_3_2_1_99_1","volume-title":"Technical Report. CS Department","author":"Improving","unstructured":"Improving the dependability of machine learning applications. Technical Report. CS Department , Columbia University . Improving the dependability of machine learning applications. Technical Report. CS Department, Columbia University."},{"key":"e_1_3_2_1_100_1","unstructured":"Christian Murphy Gail E Kaiser Lifeng Hu and Leon Wu. 2008. Properties of Machine Learning Applications for Use in Metamorphic Testing.. In SEKE. Christian Murphy Gail E Kaiser Lifeng Hu and Leon Wu. 2008. Properties of Machine Learning Applications for Use in Metamorphic Testing.. In SEKE."},{"key":"e_1_3_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/3088525.3088564"},{"key":"e_1_3_2_1_102_1","doi-asserted-by":"crossref","unstructured":"Aditya V Nori Chung-Kil Hur Sriram K Rajamani and Selva Samuel. 2014. R2: An efficient MCMC sampler for probabilistic programs. In AAAI. Aditya V Nori Chung-Kil Hur Sriram K Rajamani and Selva Samuel. 2014. R2: An efficient MCMC sampler for probabilistic programs. In AAAI.","DOI":"10.1609\/aaai.v28i1.9060"},{"key":"e_1_3_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2005.55"},{"key":"e_1_3_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.62"},{"key":"e_1_3_2_1_105_1","volume-title":"A sentence generator for testing parsers. BIT Numerical Mathematics 12, 3","author":"Purdom Paul","year":"1972","unstructured":"Paul Purdom . 1972. A sentence generator for testing parsers. BIT Numerical Mathematics 12, 3 ( 1972 ). Paul Purdom. 1972. A sentence generator for testing parsers. BIT Numerical Mathematics 12, 3 (1972)."},{"key":"e_1_3_2_1_106_1","unstructured":"Pyro Commit 0f1d27e 2017. Pyro Commit 0f1d27e 2017."},{"key":"e_1_3_2_1_107_1","unstructured":"https:\/\/github.com\/uber\/pyro\/commit\/0f1d27e. https:\/\/github.com\/uber\/pyro\/commit\/0f1d27e."},{"key":"e_1_3_2_1_108_1","unstructured":"Pyro commit 3671b06 2017. Pyro commit 3671b06 2017."},{"key":"e_1_3_2_1_109_1","unstructured":"https:\/\/github.com\/uber\/pyro\/commit\/3671b06. https:\/\/github.com\/uber\/pyro\/commit\/3671b06."},{"key":"e_1_3_2_1_110_1","unstructured":"Pyro Commit 6e26f5e 2017. Pyro Commit 6e26f5e 2017."},{"key":"e_1_3_2_1_111_1","unstructured":"https:\/\/github.com\/uber\/pyro\/commit\/6e26f5e. https:\/\/github.com\/uber\/pyro\/commit\/6e26f5e."},{"key":"e_1_3_2_1_112_1","unstructured":"Pyro Commit 7b6cf58 2017. Pyro Commit 7b6cf58 2017."},{"key":"e_1_3_2_1_113_1","unstructured":"https:\/\/github.com\/uber\/pyro\/commit\/7b6cf58. https:\/\/github.com\/uber\/pyro\/commit\/7b6cf58."},{"key":"e_1_3_2_1_114_1","unstructured":"Pyro commit 7b6cf58 2017. Pyro commit 7b6cf58 2017."},{"key":"e_1_3_2_1_115_1","unstructured":"https:\/\/github.com\/uber\/pyro\/commit\/7b6cf58. https:\/\/github.com\/uber\/pyro\/commit\/7b6cf58."},{"key":"e_1_3_2_1_116_1","unstructured":"Pyro Commit 8c14f36 2017. Pyro Commit 8c14f36 2017."},{"key":"e_1_3_2_1_117_1","unstructured":"https:\/\/github.com\/uber\/pyro\/commit\/8c14f36. https:\/\/github.com\/uber\/pyro\/commit\/8c14f36."},{"key":"e_1_3_2_1_118_1","unstructured":"Pyro Commit b94f06a 2017. Pyro Commit b94f06a 2017."},{"key":"e_1_3_2_1_119_1","unstructured":"https:\/\/github.com\/uber\/pyro\/commit\/b94f06a. https:\/\/github.com\/uber\/pyro\/commit\/b94f06a."},{"key":"e_1_3_2_1_120_1","unstructured":"Pyro Commit f5a51fe 2017. Pyro Commit f5a51fe 2017."},{"key":"e_1_3_2_1_121_1","unstructured":"https:\/\/github.com\/uber\/pyro\/commit\/f5a51f. https:\/\/github.com\/uber\/pyro\/commit\/f5a51f."},{"key":"e_1_3_2_1_122_1","first-page":"303","year":"2017","unstructured":"Pyro Issue 303 2017 . Pyro Issue 303 2017.","journal-title":"Pyro Issue"},{"key":"e_1_3_2_1_123_1","unstructured":"https:\/\/github.com\/uber\/pyro\/issues\/303. https:\/\/github.com\/uber\/pyro\/issues\/303."},{"key":"e_1_3_2_1_124_1","unstructured":"PyroWebPage 2018. Pyro. http:\/\/pyro.ai. PyroWebPage 2018. Pyro. http:\/\/pyro.ai."},{"key":"e_1_3_2_1_125_1","unstructured":"PyTorchWebPage 2018. PyTorch. http:\/\/pytorch.org. PyTorchWebPage 2018. PyTorch. http:\/\/pytorch.org."},{"key":"e_1_3_2_1_126_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180207"},{"key":"e_1_3_2_1_127_1","volume-title":"James Demmel, William Kahan, Koushik Sen, David H Bailey, Costin Iancu, and David Hough.","author":"Rubio-Gonz\u00e1lez Cindy","year":"2013","unstructured":"Cindy Rubio-Gonz\u00e1lez , Cuong Nguyen , Hong Diep Nguyen , James Demmel, William Kahan, Koushik Sen, David H Bailey, Costin Iancu, and David Hough. 2013 . Precimonious : Tuning assistant for floating-point precision. In SC. Cindy Rubio-Gonz\u00e1lez, Cuong Nguyen, Hong Diep Nguyen, James Demmel, William Kahan, Koushik Sen, David H Bailey, Costin Iancu, and David Hough. 2013. Precimonious: Tuning assistant for floating-point precision. In SC."},{"key":"e_1_3_2_1_128_1","unstructured":"Sriram Sankaranarayanan Aleksandar Chakarov and Sumit Gulwani. 2013. Sriram Sankaranarayanan Aleksandar Chakarov and Sumit Gulwani. 2013."},{"key":"e_1_3_2_1_129_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499370.2462179"},{"key":"e_1_3_2_1_130_1","unstructured":"Sergio Segura Gordon Fraser Ana B Sanchez and Antonio Ruiz-Cort\u00e9s. 2016. Sergio Segura Gordon Fraser Ana B Sanchez and Antonio Ruiz-Cort\u00e9s. 2016."},{"key":"e_1_3_2_1_131_1","volume-title":"IEEE Transactions on software engineering 42, 9","author":"A","year":"2016","unstructured":"A survey on metamorphic testing. IEEE Transactions on software engineering 42, 9 ( 2016 ), 805\u2013824. A survey on metamorphic testing. IEEE Transactions on software engineering 42, 9 (2016), 805\u2013824."},{"key":"e_1_3_2_1_132_1","unstructured":"Daniel Selsam Percy Liang and David L Dill. 2017. Daniel Selsam Percy Liang and David L Dill. 2017."},{"key":"e_1_3_2_1_133_1","volume-title":"November 4\u20139","author":"Bug-Free Machine Developing","year":"2018","unstructured":"Developing Bug-Free Machine Learning Systems With Formal Mathematics . arXiv preprint Testing Probabilistic Programming Systems ESEC\/FSE \u201918 , November 4\u20139 , 2018 , Lake Buena Vista, FL , USA arXiv:1706.08605 (2017). Developing Bug-Free Machine Learning Systems With Formal Mathematics. arXiv preprint Testing Probabilistic Programming Systems ESEC\/FSE \u201918, November 4\u20139, 2018, Lake Buena Vista, FL, USA arXiv:1706.08605 (2017)."},{"key":"e_1_3_2_1_134_1","unstructured":"SGLD produces Nan in output 2018. SGLD produces Nan in output 2018."},{"key":"e_1_3_2_1_135_1","unstructured":"https:\/\/github.com\/bleilab\/edward\/issues\/ 859. https:\/\/github.com\/bleilab\/edward\/issues\/ 859."},{"key":"e_1_3_2_1_136_1","doi-asserted-by":"crossref","unstructured":"Guoqiang Shu Yating Hsu and David Lee. 2008. Guoqiang Shu Yating Hsu and David Lee. 2008.","DOI":"10.1515\/SEM.2008.057"},{"key":"e_1_3_2_1_137_1","unstructured":"Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning. In FORTE. Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning. In FORTE."},{"key":"e_1_3_2_1_138_1","doi-asserted-by":"publisher","DOI":"10.1145\/331960.331965"},{"key":"e_1_3_2_1_139_1","unstructured":"Siwakorn Srisakaokul Zhengkai Wu Angello Astorga Oreoluwa Alebiosu and Tao Xie. 2018. Multiple-Implementation Testing of Supervised Learning Software. In EDSMLS. Siwakorn Srisakaokul Zhengkai Wu Angello Astorga Oreoluwa Alebiosu and Tao Xie. 2018. Multiple-Implementation Testing of Supervised Learning Software. In EDSMLS."},{"key":"e_1_3_2_1_140_1","unstructured":"Stan 2018. http:\/\/mcstan.org. Stan 2018. http:\/\/mcstan.org."},{"key":"e_1_3_2_1_141_1","unstructured":"Stan Best Practices 2018. Stan Best Practices 2018."},{"key":"e_1_3_2_1_142_1","unstructured":"https:\/\/github.com\/standev\/stan\/wiki\/ Stan-Best-Practices. https:\/\/github.com\/standev\/stan\/wiki\/ Stan-Best-Practices."},{"key":"e_1_3_2_1_143_1","unstructured":"Stan Commit 04fcb74 2013. Stan Commit 04fcb74 2013."},{"key":"e_1_3_2_1_144_1","unstructured":"https:\/\/github.com\/standev\/stan\/commit\/04fcb74. https:\/\/github.com\/standev\/stan\/commit\/04fcb74."},{"key":"e_1_3_2_1_145_1","unstructured":"Stan Commit 2fc94d4 2017. Stan Commit 2fc94d4 2017."},{"key":"e_1_3_2_1_146_1","unstructured":"https:\/\/github.com\/standev\/stan\/commit\/2fc94d4. https:\/\/github.com\/standev\/stan\/commit\/2fc94d4."},{"key":"e_1_3_2_1_147_1","unstructured":"Stan Commit 40c8224 2016. Stan Commit 40c8224 2016."},{"key":"e_1_3_2_1_148_1","unstructured":"https:\/\/github.com\/standev\/stan\/commit\/40c8224. https:\/\/github.com\/standev\/stan\/commit\/40c8224."},{"key":"e_1_3_2_1_149_1","unstructured":"Stan commit 45a82fd 2015. Stan commit 45a82fd 2015."},{"key":"e_1_3_2_1_150_1","unstructured":"https:\/\/github.com\/standev\/stan\/commit\/45a82fd. https:\/\/github.com\/standev\/stan\/commit\/45a82fd."},{"key":"e_1_3_2_1_151_1","first-page":"5224850","year":"2015","unstructured":"Stan Commit 5224850 2015 . Stan Commit 5224850 2015.","journal-title":"Stan Commit"},{"key":"e_1_3_2_1_152_1","unstructured":"https:\/\/github.com\/standev\/stan\/commit\/5224850. https:\/\/github.com\/standev\/stan\/commit\/5224850."},{"key":"e_1_3_2_1_153_1","unstructured":"Stan Commit 5845db97 2016. Stan Commit 5845db97 2016."},{"key":"e_1_3_2_1_154_1","unstructured":"https:\/\/github.com\/standev\/stan\/commit\/ 5845db9. https:\/\/github.com\/standev\/stan\/commit\/ 5845db9."},{"key":"e_1_3_2_1_155_1","unstructured":"Stan Commit 7a98bd2 2016. Stan Commit 7a98bd2 2016."},{"key":"e_1_3_2_1_156_1","unstructured":"https:\/\/github.com\/standev\/stan\/commit\/7a98bd2. https:\/\/github.com\/standev\/stan\/commit\/7a98bd2."},{"key":"e_1_3_2_1_157_1","unstructured":"Stan Commit 99289c85 2015. Stan Commit 99289c85 2015."},{"key":"e_1_3_2_1_158_1","unstructured":"https:\/\/github.com\/standev\/stan\/commit\/ 99289c85. https:\/\/github.com\/standev\/stan\/commit\/ 99289c85."},{"key":"e_1_3_2_1_159_1","unstructured":"Stan Commit ae423b2 2017. Stan Commit ae423b2 2017."},{"key":"e_1_3_2_1_160_1","unstructured":"https:\/\/github.com\/standev\/stan\/commit\/ae423b2. https:\/\/github.com\/standev\/stan\/commit\/ae423b2."},{"key":"e_1_3_2_1_161_1","unstructured":"Stan Commit b8d5086 2013. Stan Commit b8d5086 2013."},{"key":"e_1_3_2_1_162_1","unstructured":"https:\/\/github.com\/standev\/stan\/commit\/b8d5086. https:\/\/github.com\/standev\/stan\/commit\/b8d5086."},{"key":"e_1_3_2_1_163_1","unstructured":"Stan Issue #2178 2017. Stan Issue #2178 2017."},{"key":"e_1_3_2_1_164_1","unstructured":"https:\/\/github.com\/standev\/stan\/issues\/2178. https:\/\/github.com\/standev\/stan\/issues\/2178."},{"key":"e_1_3_2_1_165_1","unstructured":"Stan Language Manual. Appendix D. 2018. http:\/\/mcstan.org\/users\/ documentation\/index.html. Stan Language Manual. Appendix D. 2018. http:\/\/mcstan.org\/users\/ documentation\/index.html."},{"key":"e_1_3_2_1_166_1","unstructured":"Stan Language Manual. Chapter 3.1. 2018. http:\/\/mcstan.org\/users\/ documentation\/index.html. Stan Language Manual. Chapter 3.1. 2018. http:\/\/mcstan.org\/users\/ documentation\/index.html."},{"key":"e_1_3_2_1_167_1","unstructured":"Stan Pedantic Mode 2018. Stan Pedantic Mode 2018."},{"key":"e_1_3_2_1_168_1","unstructured":"https:\/\/github.com\/standev\/stan\/wiki\/ Stan-Parser-Pedantic-Mode. https:\/\/github.com\/standev\/stan\/wiki\/ Stan-Parser-Pedantic-Mode."},{"key":"e_1_3_2_1_169_1","unstructured":"Stan throws error with empty array 2018. Stan throws error with empty array 2018."},{"key":"e_1_3_2_1_170_1","unstructured":"https:\/\/github.com\/standev\/pystan\/ issues\/437. https:\/\/github.com\/standev\/pystan\/ issues\/437."},{"key":"e_1_3_2_1_171_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884879"},{"key":"e_1_3_2_1_172_1","unstructured":"Michael Sutton Adam Greene and Pedram Amini. 2007. Michael Sutton Adam Greene and Pedram Amini. 2007."},{"key":"e_1_3_2_1_173_1","volume-title":"Brute Force Vulnerability Discovery","author":"Fuzzing","unstructured":"Fuzzing : Brute Force Vulnerability Discovery . Addison-Wesley Professional . Fuzzing: Brute Force Vulnerability Discovery. Addison-Wesley Professional."},{"key":"e_1_3_2_1_174_1","unstructured":"Symmetric Mean Absolute Percentage Error. Armstrong 1985. http:\/\/www. forecastingprinciples.com\/files\/LRFCh13b.pdf. Symmetric Mean Absolute Percentage Error. Armstrong 1985. http:\/\/www. forecastingprinciples.com\/files\/LRFCh13b.pdf."},{"key":"e_1_3_2_1_175_1","unstructured":"Kunal Taneja Nuo Li Madhuri R. Marri Tao Xie and Nikolai Tillmann. 2010. Kunal Taneja Nuo Li Madhuri R. Marri Tao Xie and Nikolai Tillmann. 2010."},{"key":"e_1_3_2_1_176_1","unstructured":"MiTV: Multiple-implementation Testing of User-input Validators for Web Applications. In ASE. MiTV: Multiple-implementation Testing of User-input Validators for Web Applications. In ASE."},{"key":"e_1_3_2_1_177_1","unstructured":"TensorFlowWebPage 2018. TensorFlow. https:\/\/www.tensorflow.org. TensorFlowWebPage 2018. TensorFlow. https:\/\/www.tensorflow.org."},{"key":"e_1_3_2_1_178_1","volume-title":"Blei","author":"Tran Dustin","year":"2017","unstructured":"Dustin Tran , Matthew D. Hoffman , Rif A. Saurous , Eugene Brevdo , Kevin Murphy , and David M . Blei . 2017 . Deep probabilistic programming. In ICLR. Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M. Blei. 2017. Deep probabilistic programming. In ICLR."},{"key":"e_1_3_2_1_179_1","volume-title":"Blei","author":"Tran Dustin","year":"2016","unstructured":"Dustin Tran , Alp Kucukelbir , Adji B. Dieng , Maja Rudolph , Dawen Liang , and David M . Blei . 2016 . Edward : A library for probabilistic modeling, inference, and criticism. arXiv (2016). Dustin Tran, Alp Kucukelbir, Adji B. Dieng, Maja Rudolph, Dawen Liang, and David M. Blei. 2016. Edward: A library for probabilistic modeling, inference, and criticism. arXiv (2016)."},{"key":"e_1_3_2_1_180_1","volume-title":"Jan Willem van de Meent, and Vikash Mansinghka","author":"Wood Frank","year":"2014","unstructured":"Frank Wood , Jan Willem van de Meent, and Vikash Mansinghka . 2014 . A new approach to probabilistic programming inference. In AISTATS. Frank Wood, Jan Willem van de Meent, and Vikash Mansinghka. 2014. A new approach to probabilistic programming inference. In AISTATS."},{"key":"e_1_3_2_1_181_1","unstructured":"Tao Xie Kunal Taneja Shreyas Kale and Darko Marinov. 2007. Tao Xie Kunal Taneja Shreyas Kale and Darko Marinov. 2007."},{"key":"e_1_3_2_1_182_1","unstructured":"Towards a framework for differential unit testing of object-oriented programs. In AST. Towards a framework for differential unit testing of object-oriented programs. In AST."},{"key":"e_1_3_2_1_183_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"e_1_3_2_1_184_1","doi-asserted-by":"publisher","DOI":"10.1145\/1993498.1993532"},{"key":"e_1_3_2_1_185_1","doi-asserted-by":"publisher","DOI":"10.1145\/3062341.3062379"}],"event":{"name":"ESEC\/FSE '18: 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Lake Buena Vista FL USA","acronym":"ESEC\/FSE '18","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3236024.3236057","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3236024.3236057","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3236024.3236057","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T22:16:49Z","timestamp":1775254609000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3236024.3236057"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,10,26]]},"references-count":184,"alternative-id":["10.1145\/3236024.3236057","10.1145\/3236024"],"URL":"https:\/\/doi.org\/10.1145\/3236024.3236057","relation":{},"subject":[],"published":{"date-parts":[[2018,10,26]]},"assertion":[{"value":"2018-10-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}