{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T03:01:41Z","timestamp":1783393301277,"version":"3.54.6"},"reference-count":73,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA","license":[{"start":{"date-parts":[[2020,11,13]],"date-time":"2020-11-13T00:00:00Z","timestamp":1605225600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2020,11,13]]},"abstract":"<jats:p>\n            Concurrency bugs are notoriously hard to detect and reproduce. Controlled concurrency testing (CCT) techniques aim to offer a solution, where a\n            <jats:italic>scheduler<\/jats:italic>\n            explores the space of possible interleavings of a concurrent program looking for bugs. Since the set of possible interleavings is typically very large, these schedulers employ heuristics that prioritize the search to \u201cinteresting\u201d subspaces. However, current heuristics are typically tuned to specific bug patterns, which limits their effectiveness in practice.\n          <\/jats:p>\n          <jats:p>In this paper, we present QL, a learning-based CCT framework where the likelihood of an action being selected by the scheduler is influenced by earlier explorations. We leverage the classical Q-learning algorithm to explore the space of possible interleavings, allowing the exploration to adapt to the program under test, unlike previous techniques. We have implemented and evaluated QL on a set of microbenchmarks, complex protocols, as well as production cloud services. In our experiments, we found QL to consistently outperform the state-of-the-art in CCT.<\/jats:p>","DOI":"10.1145\/3428298","type":"journal-article","created":{"date-parts":[[2020,11,24]],"date-time":"2020-11-24T23:40:14Z","timestamp":1606261214000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["Learning-based controlled concurrency testing"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9040-0053","authenticated-orcid":false,"given":"Suvam","family":"Mukherjee","sequence":"first","affiliation":[{"name":"Microsoft Research, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7582-4520","authenticated-orcid":false,"given":"Pantazis","family":"Deligiannis","sequence":"additional","affiliation":[{"name":"Microsoft Research, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5720-013X","authenticated-orcid":false,"given":"Arpita","family":"Biswas","sequence":"additional","affiliation":[{"name":"IISc Bangalore, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akash","family":"Lal","sequence":"additional","affiliation":[{"name":"Microsoft Research, India"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,11,13]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Akka Raft. 2015. Leader election bug in Akka Raft implementation. https:\/\/github.com\/ktoso\/akka-raft\/issues\/45.  Akka Raft. 2015. Leader election bug in Akka Raft implementation. https:\/\/github.com\/ktoso\/akka-raft\/issues\/45."},{"key":"e_1_2_2_2_1","unstructured":"Amazon. 2012. Summary of the AWS service event in the US East Region. http:\/\/aws.amazon.com\/message\/67457\/.  Amazon. 2012. Summary of the AWS service event in the US East Region. http:\/\/aws.amazon.com\/message\/67457\/."},{"key":"e_1_2_2_3_1","volume-title":"16th International Conference, CAV 2004, Boston, MA, USA, July 13-17, 2004, Proceedings. 484-487","author":"Andrews Tony","year":"2004","unstructured":"Tony Andrews , Shaz Qadeer , Sriram K. Rajamani , Jakob Rehof , and Yichen Xie . 2004 . Zing: A Model Checker for Concurrent Software. In Computer Aided Verification , 16th International Conference, CAV 2004, Boston, MA, USA, July 13-17, 2004, Proceedings. 484-487 . Tony Andrews, Shaz Qadeer, Sriram K. Rajamani, Jakob Rehof, and Yichen Xie. 2004. Zing: A Model Checker for Concurrent Software. In Computer Aided Verification, 16th International Conference, CAV 2004, Boston, MA, USA, July 13-17, 2004, Proceedings. 484-487."},{"key":"e_1_2_2_4_1","volume-title":"Connectionist Models","author":"Barto Andrew G","unstructured":"Andrew G Barto and Satinder Pal Singh . 1991. On the computational economics of reinforcement learning . In Connectionist Models . Elsevier , 35-44. Andrew G Barto and Satinder Pal Singh. 1991. On the computational economics of reinforcement learning. In Connectionist Models. Elsevier, 35-44."},{"key":"e_1_2_2_5_1","volume-title":"Proceedings of the 20th International Joint Conference on Artifical Intelligence. Morgan Kaufmann Publishers Inc., 2274-2279","author":"Baskiotis Nicolas","year":"2007","unstructured":"Nicolas Baskiotis , Mich\u00e8le Sebag , Marie-Claude Gaudel , and Sandrine Gouraud . 2007 . A machine learning approach for statistical software testing . In Proceedings of the 20th International Joint Conference on Artifical Intelligence. Morgan Kaufmann Publishers Inc., 2274-2279 . Nicolas Baskiotis, Mich\u00e8le Sebag, Marie-Claude Gaudel, and Sandrine Gouraud. 2007. A machine learning approach for statistical software testing. In Proceedings of the 20th International Joint Conference on Artifical Intelligence. Morgan Kaufmann Publishers Inc., 2274-2279."},{"key":"e_1_2_2_6_1","doi-asserted-by":"crossref","unstructured":"Richard Bellman et al. 1954. The theory of dynamic programming. Bull. Amer. Math. Soc. 60 6 ( 1954 ) 503-515.  Richard Bellman et al. 1954. The theory of dynamic programming. Bull. Amer. Math. Soc. 60 6 ( 1954 ) 503-515.","DOI":"10.1090\/S0002-9904-1954-09848-8"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-17502-3_9"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/SPW.2018.00026"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1736020.1736040"},{"key":"e_1_2_2_10_1","volume-title":"Thirty-Second AAAI Conference on Artificial Intelligence.","author":"Cai Qingpeng","year":"2018","unstructured":"Qingpeng Cai , Aris Filos-Ratsikas , Pingzhong Tang , and Yiwei Zhang . 2018 . Reinforcement mechanism design for fraudulent behaviour in e-commerce . In Thirty-Second AAAI Conference on Artificial Intelligence. Qingpeng Cai, Aris Filos-Ratsikas, Pingzhong Tang, and Yiwei Zhang. 2018. Reinforcement mechanism design for fraudulent behaviour in e-commerce. In Thirty-Second AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_2_11_1","unstructured":"Tom Cargill. 2009. Extreme Programming Challenge Fourteen. http:\/\/wiki.c2.com\/ ?ExtremeProgrammingChallengeFourteen.  Tom Cargill. 2009. Extreme Programming Challenge Fourteen. http:\/\/wiki.c2.com\/ ?ExtremeProgrammingChallengeFourteen."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3158119"},{"key":"e_1_2_2_13_1","first-page":"410","volume-title":"NUSMV: A New Symbolic Model Checker. STTT 2, 4 ( 2000 )","author":"Cimatti Alessandro","year":"2000","unstructured":"Alessandro Cimatti , Edmund M. Clarke , Fausto Giunchiglia , and Marco Roveri . 2000 . NUSMV: A New Symbolic Model Checker. STTT 2, 4 ( 2000 ) , 410 - 425 . Alessandro Cimatti, Edmund M. Clarke, Fausto Giunchiglia, and Marco Roveri. 2000. NUSMV: A New Symbolic Model Checker. STTT 2, 4 ( 2000 ), 410-425."},{"key":"e_1_2_2_14_1","volume-title":"8th International Conference, CAV ' 96, New Brunswick, NJ, USA, July 31-August 3, 1996, Proceedings. 419-427","author":"Clarke Edmund M.","year":"1996","unstructured":"Edmund M. Clarke , Kenneth L. McMillan , S\u00e9rgio Vale Aguiar Campos , and Vasiliki Hartonas-Garmhausen . 1996 . Symbolic Model Checking. In Computer Aided Verification , 8th International Conference, CAV ' 96, New Brunswick, NJ, USA, July 31-August 3, 1996, Proceedings. 419-427 . Edmund M. Clarke, Kenneth L. McMillan, S\u00e9rgio Vale Aguiar Campos, and Vasiliki Hartonas-Garmhausen. 1996. Symbolic Model Checking. In Computer Aided Verification, 8th International Conference, CAV ' 96, New Brunswick, NJ, USA, July 31-August 3, 1996, Proceedings. 419-427."},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2737924.2737996"},{"key":"e_1_2_2_16_1","volume-title":"Building Reliable Cloud Services Using P# (Experience Report ). ArXiv abs\/","author":"Deligiannis Pantazis","year":"2002","unstructured":"Pantazis Deligiannis , Narayanan Ganapathy , Akash Lal , and Shaz Qadeer . 2020. Building Reliable Cloud Services Using P# (Experience Report ). ArXiv abs\/ 2002 .04903 ( 2020 ). Pantazis Deligiannis, Narayanan Ganapathy, Akash Lal, and Shaz Qadeer. 2020. Building Reliable Cloud Services Using P# (Experience Report ). ArXiv abs\/ 2002.04903 ( 2020 )."},{"key":"e_1_2_2_17_1","volume-title":"14th USENIX Conference on File and Storage Technologies, FAST 2016","author":"Deligiannis Pantazis","year":"2016","unstructured":"Pantazis Deligiannis , Matt McCutchen , Paul Thomson , Shuo Chen , Alastair F. Donaldson , John Erickson , Cheng Huang , Akash Lal , Rashmi Mudduluru , Shaz Qadeer , and Wolfram Schulte . 2016 . Uncovering Bugs in Distributed Storage Systems during Testing (Not in Production!) . In 14th USENIX Conference on File and Storage Technologies, FAST 2016 , Santa Clara, CA, USA , February 22-25, 2016., Angela Demke Brown and Florentina I. Popovici (Eds.). USENIX Association, 249-262. https:\/\/www.usenix.org\/conference\/fast16\/technical-sessions\/presentation\/deligiannis Pantazis Deligiannis, Matt McCutchen, Paul Thomson, Shuo Chen, Alastair F. Donaldson, John Erickson, Cheng Huang, Akash Lal, Rashmi Mudduluru, Shaz Qadeer, and Wolfram Schulte. 2016. Uncovering Bugs in Distributed Storage Systems during Testing (Not in Production!). In 14th USENIX Conference on File and Storage Technologies, FAST 2016, Santa Clara, CA, USA, February 22-25, 2016., Angela Demke Brown and Florentina I. Popovici (Eds.). USENIX Association, 249-262. https:\/\/www.usenix.org\/conference\/fast16\/technical-sessions\/presentation\/deligiannis"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786861"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1926385.1926432"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2003.1250664"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/INCISCOS.2018.00050"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1040305.1040315"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10703-005-1489-x"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2017.8115618"},{"key":"e_1_2_2_25_1","volume-title":"Proceedings of the 5th Symposium on Reliability in Distributed Software and Database Systems. IEEE, 3-12","author":"Gray Jim","year":"1986","unstructured":"Jim Gray . 1986 . Why do computers stop and what can be done about it? . In Proceedings of the 5th Symposium on Reliability in Distributed Software and Database Systems. IEEE, 3-12 . Jim Gray. 1986. Why do computers stop and what can be done about it?. In Proceedings of the 5th Symposium on Reliability in Distributed Software and Database Systems. IEEE, 3-12."},{"key":"e_1_2_2_26_1","first-page":"277","article-title":"Reinforcement learning in a nutshell","author":"Heidrich-Meisner Verena","year":"2007","unstructured":"Verena Heidrich-Meisner , Martin Lauer , Christian Igel , and Martin A Riedmiller . 2007 . Reinforcement learning in a nutshell .. In ESANN. Citeseer , 277 - 288 . Verena Heidrich-Meisner, Martin Lauer, Christian Igel, and Martin A Riedmiller. 2007. Reinforcement learning in a nutshell.. In ESANN. Citeseer, 277-288.","journal-title":"ESANN. Citeseer"},{"key":"e_1_2_2_27_1","volume-title":"The SPIN Model Checker: Primer and Reference Manual","author":"Holzmann Gerard","unstructured":"Gerard Holzmann . 2011. The SPIN Model Checker: Primer and Reference Manual ( 1 st ed.). Addison-Wesley Professional . Gerard Holzmann. 2011. The SPIN Model Checker: Primer and Reference Manual (1st ed.). Addison-Wesley Professional.","edition":"1"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2737924.2737975"},{"key":"e_1_2_2_29_1","volume-title":"Speeding Up Maximal Causality Reduction with Static Dependency Analysis. In 31st European Conference on Object-Oriented Programming, ECOOP 2017","author":"Huang Shiyou","year":"2017","unstructured":"Shiyou Huang and Jef Huang . 2017 . Speeding Up Maximal Causality Reduction with Static Dependency Analysis. In 31st European Conference on Object-Oriented Programming, ECOOP 2017 , June 19-23, 2017, Barcelona, Spain. 16 : 1-16 : 22. Shiyou Huang and Jef Huang. 2017. Speeding Up Maximal Causality Reduction with Static Dependency Analysis. In 31st European Conference on Object-Oriented Programming, ECOOP 2017, June 19-23, 2017, Barcelona, Spain. 16 : 1-16 : 22."},{"key":"e_1_2_2_30_1","article-title":"A Scalable Reinforcement Learning Algorithm for Scheduling Railway Lines","volume":"20","author":"Khadilkar Harshad","year":"2018","unstructured":"Harshad Khadilkar . 2018 . A Scalable Reinforcement Learning Algorithm for Scheduling Railway Lines . IEEE Transactions on Intelligent Transportation Systems 20 , 2 ( 2018 ), 727-736. Harshad Khadilkar. 2018. A Scalable Reinforcement Learning Algorithm for Scheduling Railway Lines. IEEE Transactions on Intelligent Transportation Systems 20, 2 ( 2018 ), 727-736.","journal-title":"IEEE Transactions on Intelligent Transportation Systems"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364913495721"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2012.6252823"},{"key":"e_1_2_2_33_1","first-page":"399","volume-title":"SAMC: Semantic-Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems. In 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI '14","author":"Leesatapornwongsa Tanakorn","year":"2014","unstructured":"Tanakorn Leesatapornwongsa , Mingzhe Hao , Pallavi Joshi , Jefrey F. Lukman , and Haryadi S. Gunawi . 2014 . SAMC: Semantic-Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems. In 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI '14 , Broomfield, CO, USA , October 6-8, 2014 . 399 - 414 . Tanakorn Leesatapornwongsa, Mingzhe Hao, Pallavi Joshi, Jefrey F. Lukman, and Haryadi S. Gunawi. 2014. SAMC: Semantic-Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems. In 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI '14, Broomfield, CO, USA, October 6-8, 2014. 399-414."},{"key":"e_1_2_2_34_1","article-title":"End-to-end training of deep visuomotor policies","volume":"17","author":"Levine Sergey","year":"2016","unstructured":"Sergey Levine , Chelsea Finn , Trevor Darrell , and Pieter Abbeel . 2016 . End-to-end training of deep visuomotor policies . The Journal of Machine Learning Research 17 , 1 ( 2016 ), 1334-1373. Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. 2016. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research 17, 1 ( 2016 ), 1334-1373.","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCI.2019.2901089"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/361227.361234"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2012.88"},{"key":"e_1_2_2_38_1","first-page":"279","volume-title":"Proceedings of an Advanced Course","author":"Mazurkiewicz Antoni W.","year":"1986","unstructured":"Antoni W. Mazurkiewicz . 1986 . Trace Theory. In Petri Nets: Central Models and Their Properties, Advances in Petri Nets 1986, Part II , Proceedings of an Advanced Course , Bad Honnef, Germany , 8-19 September 1986. 279 - 324 . Antoni W. Mazurkiewicz. 1986. Trace Theory. In Petri Nets: Central Models and Their Properties, Advances in Petri Nets 1986, Part II, Proceedings of an Advanced Course, Bad Honnef, Germany, 8-19 September 1986. 279-324."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.23919\/FMCAD.2017.8102245"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1250734.1250785"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1375581.1375625"},{"key":"e_1_2_2_43_1","volume-title":"Piramanayagam Arumuga Nainar, and Iulian Neamtiu","author":"Musuvathi Madanlal","year":"2008","unstructured":"Madanlal Musuvathi , Shaz Qadeer , Thomas Ball , G\u00e9rard Basler , Piramanayagam Arumuga Nainar, and Iulian Neamtiu . 2008 . Finding and Reproducing Heisenbugs in Concurrent Programs. In 8th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2008, December 8-10, 2008, San Diego, California, USA, Proceedings, Richard Draves and Robbert van Renesse (Eds.). USENIX Association , 267-280. http:\/\/www.usenix.org\/events\/osdi08\/tech\/full_papers\/musuvathi\/ musuvathi.pdf Madanlal Musuvathi, Shaz Qadeer, Thomas Ball, G\u00e9rard Basler, Piramanayagam Arumuga Nainar, and Iulian Neamtiu. 2008. Finding and Reproducing Heisenbugs in Concurrent Programs. In 8th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2008, December 8-10, 2008, San Diego, California, USA, Proceedings, Richard Draves and Robbert van Renesse (Eds.). USENIX Association, 267-280. http:\/\/www.usenix.org\/events\/osdi08\/tech\/full_papers\/musuvathi\/ musuvathi.pdf"},{"key":"e_1_2_2_44_1","doi-asserted-by":"crossref","unstructured":"Emre O Neftci and Bruno B Averbeck. 2019. Reinforcement learning in artificial and biological systems. Nature Machine Intelligence 1 3 ( 2019 ) 133-143.  Emre O Neftci and Bruno B Averbeck. 2019. Reinforcement learning in artificial and biological systems. Nature Machine Intelligence 1 3 ( 2019 ) 133-143.","DOI":"10.1038\/s42256-019-0025-4"},{"key":"e_1_2_2_45_1","first-page":"9827","article-title":"Scalable end-to-end autonomous vehicle testing via rare-event simulation","author":"O'Kelly Matthew","year":"2018","unstructured":"Matthew O'Kelly , Aman Sinha , Hongseok Namkoong , Russ Tedrake , and John C Duchi . 2018 . Scalable end-to-end autonomous vehicle testing via rare-event simulation . In Advances in Neural Information Processing Systems. 9827 - 9838 . Matthew O'Kelly, Aman Sinha, Hongseok Namkoong, Russ Tedrake, and John C Duchi. 2018. Scalable end-to-end autonomous vehicle testing via rare-event simulation. In Advances in Neural Information Processing Systems. 9827-9838.","journal-title":"Advances in Neural Information Processing Systems."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/2643634.2643666"},{"key":"e_1_2_2_47_1","volume-title":"Mitra Tabaei Befrouei, and Georg Weissenbacher","author":"Ozkan Burcu Kulahcioglu","year":"2018","unstructured":"Burcu Kulahcioglu Ozkan , Rupak Majumdar , Filip Niksic , Mitra Tabaei Befrouei, and Georg Weissenbacher . 2018 . Randomized testing of distributed systems with probabilistic guarantees. PACMPL 2, OOPSLA ( 2018 ), 160 : 1-160 : 28. Burcu Kulahcioglu Ozkan, Rupak Majumdar, Filip Niksic, Mitra Tabaei Befrouei, and Georg Weissenbacher. 2018. Randomized testing of distributed systems with probabilistic guarantees. PACMPL 2, OOPSLA ( 2018 ), 160 : 1-160 : 28."},{"key":"e_1_2_2_48_1","unstructured":"P# Team. 2019. P# : A framework for rapid development of reliable asynchronous software. https:\/\/github.com\/p-org\/PSharp.  P# Team. 2019. P# : A framework for rapid development of reliable asynchronous software. https:\/\/github.com\/p-org\/PSharp."},{"key":"e_1_2_2_49_1","volume-title":"Greybox fuzzing as a contextual bandits problem. CoRR abs\/","author":"Patil Ketan","year":"1806","unstructured":"Ketan Patil and Aditya Kanade . 2018. Greybox fuzzing as a contextual bandits problem. CoRR abs\/ 1806 .03806 ( 2018 ). arXiv: 1806.03806 http:\/\/arxiv.org\/abs\/ 1806.03806 Ketan Patil and Aditya Kanade. 2018. Greybox fuzzing as a contextual bandits problem. CoRR abs\/ 1806.03806 ( 2018 ). arXiv: 1806.03806 http:\/\/arxiv.org\/abs\/ 1806.03806"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00114731"},{"key":"e_1_2_2_51_1","volume-title":"On-line Q-learning using connectionist systems","author":"Rummery Gavin A","unstructured":"Gavin A Rummery and Mahesan Niranjan . 1994. On-line Q-learning using connectionist systems . University of Cambridge , Department of Engineering. Gavin A Rummery and Mahesan Niranjan. 1994. On-line Q-learning using connectionist systems. University of Cambridge, Department of Engineering."},{"key":"e_1_2_2_52_1","volume-title":"Artificial intelligence: a modern approach","author":"Russell Stuart Jonathan","unstructured":"Stuart Jonathan Russell , Peter Norvig , John F Canny , Jitendra M Malik , and Douglas D Edwards . 2003. Artificial intelligence: a modern approach . Vol. 2 . Prentice hall Upper Saddle River . Stuart Jonathan Russell, Peter Norvig, John F Canny, Jitendra M Malik, and Douglas D Edwards. 2003. Artificial intelligence: a modern approach. Vol. 2. Prentice hall Upper Saddle River."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/21.21595"},{"key":"e_1_2_2_54_1","volume-title":"NEUZZ: Eficient Fuzzing with Neural Program Learning. CoRR abs\/","author":"She Dongdong","year":"2018","unstructured":"Dongdong She , Kexin Pei , Dave Epstein , Junfeng Yang , Baishakhi Ray , and Suman Jana . 2018 . NEUZZ: Eficient Fuzzing with Neural Program Learning. CoRR abs\/ 1807.05620 ( 2018 ). arXiv: 1807.05620 http:\/\/arxiv.org\/abs\/ 1807.05620 Dongdong She, Kexin Pei, Dave Epstein, Junfeng Yang, Baishakhi Ray, and Suman Jana. 2018. NEUZZ: Eficient Fuzzing with Neural Program Learning. CoRR abs\/ 1807.05620 ( 2018 ). arXiv: 1807.05620 http:\/\/arxiv.org\/abs\/ 1807.05620"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33014902"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature16961"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-22306-8_14"},{"key":"e_1_2_2_58_1","volume-title":"Reinforcement learning: An introduction","author":"Sutton Richard S","unstructured":"Richard S Sutton and Andrew G Barto . 1998. Reinforcement learning: An introduction . MIT press . Richard S Sutton and Andrew G Barto. 1998. Reinforcement learning: An introduction. MIT press."},{"key":"e_1_2_2_59_1","doi-asserted-by":"crossref","unstructured":"Csaba Szepesv\u00e1ri. 2010. Algorithms for reinforcement learning. Synthesis lectures on artificial intelligence and machine learning 4 1 ( 2010 ) 1-103.  Csaba Szepesv\u00e1ri. 2010. Algorithms for reinforcement learning. Synthesis lectures on artificial intelligence and machine learning 4 1 ( 2010 ) 1-103.","DOI":"10.2200\/S00268ED1V01Y201005AIM009"},{"key":"e_1_2_2_61_1","volume-title":"Advances in Neural Information Processing Systems 4, [NIPS Conference","author":"Tesauro Gerald","year":"1991","unstructured":"Gerald Tesauro . 1991. Practical Issues in Temporal Diference Learning . In Advances in Neural Information Processing Systems 4, [NIPS Conference , Denver, Colorado, USA , December 2-5, 1991 ], John E. Moody, Stephen Jose Hanson, and Richard Lippmann (Eds.). Morgan Kaufmann , 259-266. http:\/\/papers.nips.cc\/paper\/465-practical-issues-in-temporal-diference-learning Gerald Tesauro. 1991. Practical Issues in Temporal Diference Learning. In Advances in Neural Information Processing Systems 4, [NIPS Conference, Denver, Colorado, USA, December 2-5, 1991 ], John E. Moody, Stephen Jose Hanson, and Richard Lippmann (Eds.). Morgan Kaufmann, 259-266. http:\/\/papers.nips.cc\/paper\/465-practical-issues-in-temporal-diference-learning"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858651"},{"key":"e_1_2_2_63_1","unstructured":"Ben Treynor. 2014. GoogleBlog-Today's outage for several Google services. http:\/\/googleblog.blogspot.com\/ 2014 \/01\/todaysoutage-for-several-google.html.  Ben Treynor. 2014. GoogleBlog-Today's outage for several Google services. http:\/\/googleblog.blogspot.com\/ 2014 \/01\/todaysoutage-for-several-google.html."},{"key":"e_1_2_2_64_1","volume-title":"Formal Approaches to Software Testing and Runtime Verification, Klaus Havelund, Manuel N\u00fa\u00f1ez, Grigore Ro\u015fu","author":"Veanes Margus","unstructured":"Margus Veanes , Pritam Roy , and Colin Campbell . 2006. Online Testing with Reinforcement Learning . In Formal Approaches to Software Testing and Runtime Verification, Klaus Havelund, Manuel N\u00fa\u00f1ez, Grigore Ro\u015fu , and Burkhart Wolf (Eds.). Springer Berlin Heidelberg , Berlin, Heidelberg , 240-253. Margus Veanes, Pritam Roy, and Colin Campbell. 2006. Online Testing with Reinforcement Learning. In Formal Approaches to Software Testing and Runtime Verification, Klaus Havelund, Manuel N\u00fa\u00f1ez, Grigore Ro\u015fu, and Burkhart Wolf (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 240-253."},{"key":"e_1_2_2_65_1","unstructured":"Dmitry Vyukov. 2010. Bug with a context switch bound 5. https:\/\/social.msdn.microsoft.com\/Forums\/en-US\/ 91c1971c-519f4ad2-816d-149e6b2fd916\/bug-with-a-context-switch-bound-5?forum=chess.  Dmitry Vyukov. 2010. Bug with a context switch bound 5. https:\/\/social.msdn.microsoft.com\/Forums\/en-US\/ 91c1971c-519f4ad2-816d-149e6b2fd916\/bug-with-a-context-switch-bound-5?forum=chess."},{"key":"e_1_2_2_67_1","doi-asserted-by":"crossref","unstructured":"Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8 3-4 ( 1992 ) 279-292.  Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8 3-4 ( 1992 ) 279-292.","DOI":"10.1023\/A:1022676722315"},{"key":"e_1_2_2_68_1","unstructured":"Christopher John Cornish Hellaby Watkins. 1989b. Learning from delayed rewards. ( 1989 ).  Christopher John Cornish Hellaby Watkins. 1989b. Learning from delayed rewards. ( 1989 )."},{"key":"e_1_2_2_69_1","unstructured":"Hillel Wayne. 2018. Augmenting Agile with Formal Methods. https:\/\/www.hillelwayne.com\/post\/augmenting-agile\/.  Hillel Wayne. 2018. Augmenting Agile with Formal Methods. https:\/\/www.hillelwayne.com\/post\/augmenting-agile\/."},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220096"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-200-7.50075-1"},{"key":"e_1_2_2_73_1","volume-title":"Proceedings of the 6th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2009","author":"Yang Junfeng","year":"2009","unstructured":"Junfeng Yang , Tisheng Chen , Ming Wu , Zhilei Xu , Xuezheng Liu , Haoxiang Lin , Mao Yang , Fan Long , Lintao Zhang , and Lidong Zhou . 2009 . MODIST: Transparent Model Checking of Unmodified Distributed Systems . In Proceedings of the 6th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2009 , April 22-24, 2009, Boston, MA, USA. 213-228. Junfeng Yang, Tisheng Chen, Ming Wu, Zhilei Xu, Xuezheng Liu, Haoxiang Lin, Mao Yang, Fan Long, Lintao Zhang, and Lidong Zhou. 2009. MODIST: Transparent Model Checking of Unmodified Distributed Systems. In Proceedings of the 6th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2009, April 22-24, 2009, Boston, MA, USA. 213-228."},{"key":"e_1_2_2_74_1","first-page":"603","volume-title":"NIPS 2003","author":"Zheng Alice X.","year":"2003","unstructured":"Alice X. Zheng , Michael I. Jordan , Ben Liblit , and Alexander Aiken . 2003 . Statistical Debugging of Sampled Programs. In Advances in Neural Information Processing Systems 16 [Neural Information Processing Systems , NIPS 2003 , December 8-13, 2003, Vancouver and Whistler, British Columbia, Canada], Sebastian Thrun, Lawrence K. Saul, and Bernhard Sch\u00f6lkopf (Eds.). MIT Press , 603 - 610 . http:\/\/papers.nips.cc\/paper\/2371-statistical-debugging-of-sampled-programs Alice X. Zheng, Michael I. Jordan, Ben Liblit, and Alexander Aiken. 2003. Statistical Debugging of Sampled Programs. In Advances in Neural Information Processing Systems 16 [Neural Information Processing Systems, NIPS 2003, December 8-13, 2003, Vancouver and Whistler, British Columbia, Canada], Sebastian Thrun, Lawrence K. Saul, and Bernhard Sch\u00f6lkopf (Eds.). MIT Press, 603-610. http:\/\/papers.nips.cc\/paper\/2371-statistical-debugging-of-sampled-programs"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143983"},{"key":"e_1_2_2_76_1","doi-asserted-by":"crossref","unstructured":"Zhenpeng Zhou Xiaocheng Li and Richard N Zare. 2017. Optimizing chemical reactions with deep reinforcement learning. ACS central science 3 12 ( 2017 ) 1337-1344.  Zhenpeng Zhou Xiaocheng Li and Richard N Zare. 2017. Optimizing chemical reactions with deep reinforcement learning. ACS central science 3 12 ( 2017 ) 1337-1344.","DOI":"10.1021\/acscentsci.7b00492"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3428298","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3428298","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:02:58Z","timestamp":1750197778000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3428298"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,13]]},"references-count":73,"journal-issue":{"issue":"OOPSLA","published-print":{"date-parts":[[2020,11,13]]}},"alternative-id":["10.1145\/3428298"],"URL":"https:\/\/doi.org\/10.1145\/3428298","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,13]]},"assertion":[{"value":"2020-11-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}