{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T18:50:37Z","timestamp":1773341437830,"version":"3.50.1"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA","license":[{"start":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T00:00:00Z","timestamp":1570665600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100007297","name":"Office of Naval Research","doi-asserted-by":"publisher","award":["N00014-17-1-2699"],"award-info":[{"award-number":["N00014-17-1-2699"]}],"id":[{"id":"10.13039\/100007297","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2019,10,10]]},"abstract":"<jats:p>As machine learning systems are increasingly used to make real world legal and financial decisions, it is of paramount importance that we develop algorithms to verify that these systems do not discriminate against minorities. We design a scalable algorithm for verifying fairness specifications. Our algorithm obtains strong correctness guarantees based on adaptive concentration inequalities; such inequalities enable our algorithm to adaptively take samples until it has enough data to make a decision. We implement our algorithm in a tool called VeriFair, and show that it scales to large machine learning models, including a deep recurrent neural network that is more than five orders of magnitude larger than the largest previously-verified neural network. While our technique only gives probabilistic guarantees due to the use of random samples, we show that we can choose the probability of error to be extremely small.<\/jats:p>","DOI":"10.1145\/3360544","type":"journal-article","created":{"date-parts":[[2019,10,11]],"date-time":"2019-10-11T14:53:33Z","timestamp":1570805613000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":39,"title":["Probabilistic verification of fairness properties via concentration"],"prefix":"10.1145","volume":"3","author":[{"given":"Osbert","family":"Bastani","sequence":"first","affiliation":[{"name":"University of Pennsylvania, USA"}]},{"given":"Xin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology, USA"}]},{"given":"Armando","family":"Solar-Lezama","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology, USA"}]}],"member":"320","published-online":{"date-parts":[[2019,10,10]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"crossref","unstructured":"Aws Albarghouthi Loris D\u2019Antoni Samuel Drews and Aditya V Nori. 2017. FairSquare: probabilistic verification of program fairness. In OOPSLA.  Aws Albarghouthi Loris D\u2019Antoni Samuel Drews and Aditya V Nori. 2017. FairSquare: probabilistic verification of program fairness. In OOPSLA.","DOI":"10.1145\/3133904"},{"key":"e_1_2_2_2_1","first-page":"671","article-title":"Big data\u2019s disparate impact","volume":"104","author":"Barocas Solon","year":"2016","unstructured":"Solon Barocas and Andrew D Selbst . 2016 . Big data\u2019s disparate impact . Cal. L. Rev. 104 (2016), 671 . Solon Barocas and Andrew D Selbst. 2016. Big data\u2019s disparate impact. Cal. L. Rev. 104 (2016), 671.","journal-title":"Cal. L. Rev."},{"key":"e_1_2_2_3_1","unstructured":"Osbert Bastani Yani Ioannou Leonidas Lampropoulos Dimitrios Vytiniotis Aditya Nori and Antonio Criminisi. 2016. Measuring neural net robustness with constraints. In Advances in neural information processing systems. 2613\u20132621.  Osbert Bastani Yani Ioannou Leonidas Lampropoulos Dimitrios Vytiniotis Aditya Nori and Antonio Criminisi. 2016. Measuring neural net robustness with constraints. In Advances in neural information processing systems. 2613\u20132621."},{"key":"e_1_2_2_4_1","volume-title":"Adverse impact and test validation: A practitioner\u2019s guide to valid and defensible employment testing","author":"Biddle Dan","unstructured":"Dan Biddle . 2006. Adverse impact and test validation: A practitioner\u2019s guide to valid and defensible employment testing . Gower Publishing, Ltd. Dan Biddle. 2006. Adverse impact and test validation: A practitioner\u2019s guide to valid and defensible employment testing. Gower Publishing, Ltd."},{"key":"e_1_2_2_5_1","doi-asserted-by":"crossref","unstructured":"Toon Calders Faisal Kamiran and Mykola Pechenizkiy. 2009. Building classifiers with independency constraints. In ICDMW. 13\u201318.  Toon Calders Faisal Kamiran and Mykola Pechenizkiy. 2009. Building classifiers with independency constraints. In ICDMW. 13\u201318.","DOI":"10.1109\/ICDMW.2009.83"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-010-0190-x"},{"key":"e_1_2_2_7_1","volume-title":"Karthikeyan Natesan Ramamurthy, and Kush R Varshney","author":"Calmon Flavio","year":"2017","unstructured":"Flavio Calmon , Dennis Wei , Bhanukiran Vinzamuri , Karthikeyan Natesan Ramamurthy, and Kush R Varshney . 2017 . Optimized Pre-Processing for Discrimination Prevention. In Advances in Neural Information Processing Systems . 3995\u2013 4004. Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. 2017. Optimized Pre-Processing for Discrimination Prevention. In Advances in Neural Information Processing Systems. 3995\u2013 4004."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291125.3309617"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2491423"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-24372-1_1"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098095"},{"key":"e_1_2_2_12_1","volume-title":"Transparent Data Mining for Big and Small Data","author":"Datta Anupam","unstructured":"Anupam Datta , Shayak Sen , and Yair Zick . 2017. Algorithmic transparency via quantitative input influence . In Transparent Data Mining for Big and Small Data . Springer , 71\u201394. Anupam Datta, Shayak Sen, and Yair Zick. 2017. Algorithmic transparency via quantitative input influence. In Transparent Data Mining for Big and Small Data. Springer, 71\u201394."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_2_2_14_1","volume-title":"Decoupled Classifiers for Group-Fair and Efficient Machine Learning. In Conference on Fairness, Accountability and Transparency. 119\u2013133","author":"Dwork Cynthia","year":"2018","unstructured":"Cynthia Dwork , Nicole Immorlica , Adam Tauman Kalai , and Mark DM Leiserson . 2018 . Decoupled Classifiers for Group-Fair and Efficient Machine Learning. In Conference on Fairness, Accountability and Transparency. 119\u2013133 . Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Mark DM Leiserson. 2018. Decoupled Classifiers for Group-Fair and Efficient Machine Learning. In Conference on Fairness, Accountability and Transparency. 119\u2013133."},{"key":"e_1_2_2_15_1","volume-title":"Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 7639","author":"Esteva Andre","year":"2017","unstructured":"Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 7639 ( 2017 ), 115. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 7639 (2017), 115."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/2486788.2486870"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611974348.17"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2018.00058"},{"key":"e_1_2_2_21_1","volume-title":"Psi: Exact symbolic inference for probabilistic programs. In CAV.","author":"Gehr Timon","year":"2016","unstructured":"Timon Gehr , Sasa Misailovic , and Martin Vechev . 2016 . Psi: Exact symbolic inference for probabilistic programs. In CAV. Timon Gehr, Sasa Misailovic, and Martin Vechev. 2016. Psi: Exact symbolic inference for probabilistic programs. In CAV."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2338965.2336773"},{"key":"e_1_2_2_23_1","unstructured":"Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. In ICLR.  Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. In ICLR."},{"key":"e_1_2_2_24_1","unstructured":"Google. 2018. Recurrent Neural Networks for Drawing Classification. https:\/\/www.tensorflow.org\/versions\/master\/tutorials\/ recurrent_quickdraw . Accessed: 2018-04-15.  Google. 2018. Recurrent Neural Networks for Drawing Classification. https:\/\/www.tensorflow.org\/versions\/master\/tutorials\/ recurrent_quickdraw . Accessed: 2018-04-15."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593882.2593900"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-31980-1_18"},{"key":"e_1_2_2_27_1","volume-title":"A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477","author":"Ha David","year":"2017","unstructured":"David Ha and Douglas Eck . 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477 ( 2017 ). David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477 (2017)."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2012.72"},{"key":"e_1_2_2_29_1","unstructured":"Moritz Hardt Eric Price and Nathan Srebro. 2016. Equality of opportunity in supervised learning. In NIPS. 3315\u20133323.  Moritz Hardt Eric Price and Nathan Srebro. 2016. Equality of opportunity in supervised learning. In NIPS. 3315\u20133323."},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24622-0_8"},{"key":"e_1_2_2_31_1","volume-title":"Quantitative Evaluation of Systems, 2006. QEST 2006. Third International Conference on. IEEE, 129\u2013130","author":"Herault Thomas","year":"2006","unstructured":"Thomas Herault , Richard Lassaigne , and Sylvain Peyronnet . 2006 . APMC 3.0: Approximate verification of discrete and continuous time Markov chains . In Quantitative Evaluation of Systems, 2006. QEST 2006. Third International Conference on. IEEE, 129\u2013130 . Thomas Herault, Richard Lassaigne, and Sylvain Peyronnet. 2006. APMC 3.0: Approximate verification of discrete and continuous time Markov chains. In Quantitative Evaluation of Systems, 2006. QEST 2006. Third International Conference on. IEEE, 129\u2013130."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1963.10500830"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-63387-9_1"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3097992"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-63387-9_5"},{"key":"e_1_2_2_36_1","volume-title":"Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Sch\u00f6lkopf.","author":"Kilbertus Niki","year":"2017","unstructured":"Niki Kilbertus , Mateo Rojas Carulla , Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Sch\u00f6lkopf. 2017 . Avoiding discrimination through causal reasoning. In Advances in Neural Information Processing Systems . 656\u2013666. Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Sch\u00f6lkopf. 2017. Avoiding discrimination through causal reasoning. In Advances in Neural Information Processing Systems. 656\u2013666."},{"key":"e_1_2_2_37_1","doi-asserted-by":"crossref","unstructured":"Jon Kleinberg Sendhil Mullainathan and Manish Raghavan. 2017. Inherent trade-offs in the fair determination of risk scores. In ITCS.  Jon Kleinberg Sendhil Mullainathan and Manish Raghavan. 2017. Inherent trade-offs in the fair determination of risk scores. In ITCS.","DOI":"10.1145\/3219617.3219634"},{"key":"e_1_2_2_38_1","unstructured":"Matt J Kusner Joshua Loftus Chris Russell and Ricardo Silva. 2017. Counterfactual fairness. In Advances in Neural Information Processing Systems. 4069\u20134079.  Matt J Kusner Joshua Loftus Chris Russell and Ricardo Silva. 2017. Counterfactual fairness. In Advances in Neural Information Processing Systems. 4069\u20134079."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-46029-2_13"},{"key":"e_1_2_2_40_1","doi-asserted-by":"crossref","unstructured":"Himabindu Lakkaraju Jon Kleinberg Jure Leskovec Jens Ludwig and Sendhil Mullainathan. 2017. The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables. In KDD.  Himabindu Lakkaraju Jon Kleinberg Jure Leskovec Jens Ludwig and Sendhil Mullainathan. 2017. The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables. In KDD.","DOI":"10.1145\/3097983.3098066"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1090\/S0025-5718-1991-1079024-2"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-16612-9_11"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-45099-3_17"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/360204.360211"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45309-1_24"},{"key":"e_1_2_2_46_1","volume-title":"AAAI","volume":"2018","author":"Nabi Razieh","year":"2018","unstructured":"Razieh Nabi and Ilya Shpitser . 2018 . Fair inference on outcomes . In AAAI , Vol. 2018 . Razieh Nabi and Ilya Shpitser. 2018. Fair inference on outcomes. In AAAI, Vol. 2018."},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401959"},{"key":"e_1_2_2_48_1","volume-title":"Odds of winning $1 billion Mega Millions and Powerball: 1 in 88 quadrillion. CBS News","author":"Picchi Aimee","year":"2019","unstructured":"Aimee Picchi . 2019. Odds of winning $1 billion Mega Millions and Powerball: 1 in 88 quadrillion. CBS News ( 2019 ). https:\/\/www.cbsnews.com\/news\/odds-of-winning-1-billion-mega-millions-and-powerball-1-in-88-quadrillion Aimee Picchi. 2019. Odds of winning $1 billion Mega Millions and Powerball: 1 in 88 quadrillion. CBS News (2019). https:\/\/www.cbsnews.com\/news\/odds-of-winning-1-billion-mega-millions-and-powerball-1-in-88-quadrillion"},{"key":"e_1_2_2_49_1","unstructured":"Aditi Raghunathan Jacob Steinhardt and Percy Liang. 2018. Certified defenses against adversarial examples. In ICLR.  Aditi Raghunathan Jacob Steinhardt and Percy Liang. 2018. Certified defenses against adversarial examples. In ICLR."},{"key":"e_1_2_2_50_1","doi-asserted-by":"crossref","unstructured":"Adrian Sampson Pavel Panchekha Todd Mytkowicz Kathryn S McKinley Dan Grossman and Luis Ceze. 2014. Expressing and verifying probabilistic assertions. In PLDI.  Adrian Sampson Pavel Panchekha Todd Mytkowicz Kathryn S McKinley Dan Grossman and Luis Ceze. 2014. Expressing and verifying probabilistic assertions. In PLDI.","DOI":"10.1145\/2594291.2594294"},{"key":"e_1_2_2_51_1","doi-asserted-by":"crossref","unstructured":"Sriram Sankaranarayanan Aleksandar Chakarov and Sumit Gulwani. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In PLDI. 447\u2013458.  Sriram Sankaranarayanan Aleksandar Chakarov and Sumit Gulwani. 2013. Static analysis for probabilistic programs: inferring whole program properties from finitely many paths. In PLDI. 447\u2013458.","DOI":"10.1145\/2499370.2462179"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-27813-9_16"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/11513988_26"},{"key":"e_1_2_2_54_1","unstructured":"Mallory Simon. 2009. HP looking into claim webcams can\u2019t see black people. http:\/\/www.cnn.com\/2009\/TECH\/12\/22\/hp. webcams\/index.html  Mallory Simon. 2009. HP looking into claim webcams can\u2019t see black people. http:\/\/www.cnn.com\/2009\/TECH\/12\/22\/hp. webcams\/index.html"},{"key":"e_1_2_2_55_1","volume-title":"Verifying Neural Networks with Mixed Integer Programming. arXiv preprint arXiv:1711.07356","author":"Tjeng Vincent","year":"2017","unstructured":"Vincent Tjeng and Russ Tedrake . 2017. Verifying Neural Networks with Mixed Integer Programming. arXiv preprint arXiv:1711.07356 ( 2017 ). Vincent Tjeng and Russ Tedrake. 2017. Verifying Neural Networks with Mixed Integer Programming. arXiv preprint arXiv:1711.07356 (2017)."},{"key":"e_1_2_2_56_1","volume-title":"The complexity of computing the permanent. Theoretical computer science 8, 2","author":"Valiant Leslie G","year":"1979","unstructured":"Leslie G Valiant . 1979. The complexity of computing the permanent. Theoretical computer science 8, 2 ( 1979 ), 189\u2013201. Leslie G Valiant. 1979. The complexity of computing the permanent. Theoretical computer science 8, 2 (1979), 189\u2013201."},{"key":"e_1_2_2_57_1","volume-title":"Sequential tests of statistical hypotheses. The annals of mathematical statistics 16, 2","author":"Wald Abraham","year":"1945","unstructured":"Abraham Wald . 1945. Sequential tests of statistical hypotheses. The annals of mathematical statistics 16, 2 ( 1945 ), 117\u2013186. Abraham Wald. 1945. Sequential tests of statistical hypotheses. The annals of mathematical statistics 16, 2 (1945), 117\u2013186."},{"key":"e_1_2_2_58_1","volume-title":"Fairness with Dynamics. arXiv preprint arXiv:1901.08568","author":"Wen Min","year":"2019","unstructured":"Min Wen , Osbert Bastani , and Ufuk Topcu . 2019. Fairness with Dynamics. arXiv preprint arXiv:1901.08568 ( 2019 ). Min Wen, Osbert Bastani, and Ufuk Topcu. 2019. Fairness with Dynamics. arXiv preprint arXiv:1901.08568 (2019)."},{"key":"e_1_2_2_59_1","volume-title":"Proceedings of the AIPS-02 Workshop on Planning via Model Checking. Citeseer, 81\u201388","author":"Younes H\u00e5kan LS","year":"2002","unstructured":"H\u00e5kan LS Younes , David J Musliner , 2002 . Probabilistic plan verification through acceptance sampling . In Proceedings of the AIPS-02 Workshop on Planning via Model Checking. Citeseer, 81\u201388 . H\u00e5kan LS Younes, David J Musliner, et al. 2002. Probabilistic plan verification through acceptance sampling. In Proceedings of the AIPS-02 Workshop on Planning via Model Checking. Citeseer, 81\u201388."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45657-0_17"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ic.2006.05.002"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052660"},{"key":"e_1_2_2_64_1","first-page":"1375","article-title":"Understanding discrimination in the scored society","volume":"89","author":"Zarsky Tal Z","year":"2014","unstructured":"Tal Z Zarsky . 2014 . Understanding discrimination in the scored society . Wash. L. Rev. 89 (2014), 1375 . Tal Z Zarsky. 2014. Understanding discrimination in the scored society. Wash. L. Rev. 89 (2014), 1375.","journal-title":"Wash. L. Rev."},{"key":"e_1_2_2_65_1","volume-title":"International Conference on Machine Learning. 325\u2013333","author":"Zemel Rich","year":"2013","unstructured":"Rich Zemel , Yu Wu , Kevin Swersky , Toni Pitassi , and Cynthia Dwork . 2013 . Learning fair representations . In International Conference on Machine Learning. 325\u2013333 . Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International Conference on Machine Learning. 325\u2013333."},{"key":"e_1_2_2_66_1","unstructured":"Shengjia Zhao Enze Zhou Ashish Sabharwal and Stefano Ermon. 2016. Adaptive Concentration Inequalities for Sequential Decision Problems. In NIPS. 1343\u20131351.  Shengjia Zhao Enze Zhou Ashish Sabharwal and Stefano Ermon. 2016. Adaptive Concentration Inequalities for Sequential Decision Problems. In NIPS. 1343\u20131351."}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3360544","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3360544","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3360544","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:22:58Z","timestamp":1750202578000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3360544"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,10]]},"references-count":65,"journal-issue":{"issue":"OOPSLA","published-print":{"date-parts":[[2019,10,10]]}},"alternative-id":["10.1145\/3360544"],"URL":"https:\/\/doi.org\/10.1145\/3360544","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,10]]},"assertion":[{"value":"2019-10-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}