{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T01:54:14Z","timestamp":1769738054981,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":59,"publisher":"ACM","license":[{"start":{"date-parts":[[2018,10,26]],"date-time":"2018-10-26T00:00:00Z","timestamp":1540512000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CCF-1453474, IIS-1453543, CNS-1744471, CCF-1763423"],"award-info":[{"award-number":["CCF-1453474, IIS-1453543, CNS-1744471, CCF-1763423"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2018,10,26]]},"DOI":"10.1145\/3236024.3264590","type":"proceedings-article","created":{"date-parts":[[2018,10,29]],"date-time":"2018-10-29T12:06:00Z","timestamp":1540814760000},"page":"871-875","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":67,"title":["Themis: automatically testing software for discrimination"],"prefix":"10.1145","author":[{"given":"Rico","family":"Angell","sequence":"first","affiliation":[{"name":"University of Massachusetts at Amherst, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Brittany","family":"Johnson","sequence":"additional","affiliation":[{"name":"University of Massachusetts at Amherst, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuriy","family":"Brun","sequence":"additional","affiliation":[{"name":"University of Massachusetts at Amherst, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexandra","family":"Meliou","sequence":"additional","affiliation":[{"name":"University of Massachusetts at Amherst, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,10,26]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Iterative orthogonal feature projection for diagnosing bias in black-box models. CoRR, abs\/1611.04967","author":"Adebayo Julius","year":"2016","unstructured":"Julius Adebayo and Lalana Kagal . Iterative orthogonal feature projection for diagnosing bias in black-box models. CoRR, abs\/1611.04967 , 2016 . Julius Adebayo and Lalana Kagal. Iterative orthogonal feature projection for diagnosing bias in black-box models. CoRR, abs\/1611.04967, 2016."},{"key":"e_1_3_2_1_2_1","volume-title":"International Conference on Machine Learning (ICML)","author":"Agarwal Alekh","year":"2018","unstructured":"Alekh Agarwal , Alina Beygelzimer , Miroslav Dud\u00edk , John Langford , and Hanna Wallach . A reductions approach to fair classification . In International Conference on Machine Learning (ICML) , Stockholm, Sweden , 2018 . Alekh Agarwal, Alina Beygelzimer, Miroslav Dud\u00edk, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International Conference on Machine Learning (ICML), Stockholm, Sweden, 2018."},{"key":"e_1_3_2_1_3_1","volume-title":"ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA)","author":"Albarghouthi Aws","year":"2017","unstructured":"Aws Albarghouthi , Loris D\u2019Antoni , Samuel Drews , and Aditya Nori . FairSquare : Probabilistic verification for program fairness . In ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA) , 2017 . Aws Albarghouthi, Loris D\u2019Antoni, Samuel Drews, and Aditya Nori. FairSquare: Probabilistic verification for program fairness. In ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA), 2017."},{"key":"e_1_3_2_1_4_1","volume-title":"ProPublica","author":"Angwin Julia","year":"2016","unstructured":"Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . Machine bias . ProPublica , May 23, 2016 . Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, May 23, 2016."},{"key":"e_1_3_2_1_5_1","unstructured":"https:\/\/www.propublica.org\/article\/ machine-bias-risk-assessments-in-criminal-sentencing.  https:\/\/www.propublica.org\/article\/ machine-bias-risk-assessments-in-criminal-sentencing."},{"key":"e_1_3_2_1_6_1","first-page":"50","volume-title":"Software Engineering for Self-Adaptive Systems II","author":"Brun Yuriy","unstructured":"Yuriy Brun , Ron Desmarais , Kurt Geihs , Marin Litoiu , Antonia Lopes , Mary Shaw , and Mike Smit . A design space for adaptive systems . In Rog\u00e9rio de Lemos, Holger Giese, Hausi A. M\u00fcller, and Mary Shaw, editors, Software Engineering for Self-Adaptive Systems II , volume 7475 , pages 33\u2013 50 . Springer-Verlag, 2013. Yuriy Brun, Ron Desmarais, Kurt Geihs, Marin Litoiu, Antonia Lopes, Mary Shaw, and Mike Smit. A design space for adaptive systems. In Rog\u00e9rio de Lemos, Holger Giese, Hausi A. M\u00fcller, and Mary Shaw, editors, Software Engineering for Self-Adaptive Systems II, volume 7475, pages 33\u201350. Springer-Verlag, 2013."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2011.25"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025187"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025139"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.28"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEAMS.2007.4"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1316550.1316557"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2012.126"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2013.13"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264838"},{"key":"e_1_3_2_1_16_1","volume-title":"the Future of Collaborative Software Development (FCSD)","author":"Brun Yuriy","year":"2012","unstructured":"Yuriy Brun , K\u0131van\u00e7 Mu\u015flu , Reid Holmes , Michael D. Ernst , and David Notkin . Predicting development trajectories to prevent collaboration conflicts . In the Future of Collaborative Software Development (FCSD) , Seattle, WA, USA , February 2012 . Yuriy Brun, K\u0131van\u00e7 Mu\u015flu, Reid Holmes, Michael D. Ernst, and David Notkin. Predicting development trajectories to prevent collaboration conflicts. In the Future of Collaborative Software Development (FCSD), Seattle, WA, USA, February 2012."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2412134"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDMW.2009.83"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-010-0190-x"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aal4230"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.14"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.14778\/2850583.2850592"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSPEC.2016.7473149"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC4.2009.4909197"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2010.50"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2012.45"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33486-3_3"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702520"},{"key":"e_1_3_2_1_31_1","volume-title":"An empirical study of rich subgroup fairness for machine learning. CoRR, abs\/1808.08166","author":"Kearns Michael","year":"2018","unstructured":"Michael Kearns , Seth Neel , Aaron Roth , and Zhiwei Steven Wu . An empirical study of rich subgroup fairness for machine learning. CoRR, abs\/1808.08166 , 2018 . Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. An empirical study of rich subgroup fairness for machine learning. CoRR, abs\/1808.08166, 2018."},{"key":"e_1_3_2_1_32_1","volume-title":"International Conference on Machine Learning (ICML)","author":"Kearns Michael","year":"2018","unstructured":"Michael Kearns , Seth Neel , Aaron Roth , and Zhiwei Steven Wu . Preventing fairness gerrymandering: Auditing and learning for subgroup fairness . In International Conference on Machine Learning (ICML) , Stockholm, Sweden , 2018 . Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning (ICML), Stockholm, Sweden, 2018."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2012.2214212"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1595696.1595756"},{"key":"e_1_3_2_1_35_1","volume-title":"Annual Conference on Neural Information Processing Systems (NIPS)","author":"Kusner Matt J.","year":"2017","unstructured":"Matt J. Kusner , Joshua R. Loftus , Chris Russell , and Ricardo Silva . Counterfactual fairness . In Annual Conference on Neural Information Processing Systems (NIPS) , December 2017 . Matt J. Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Annual Conference on Neural Information Processing Systems (NIPS), December 2017."},{"key":"e_1_3_2_1_36_1","first-page":"2016","volume-title":"TECH Insider","author":"Letzter Rafi","year":"2016","unstructured":"Rafi Letzter . Amazon just showed us that \u2018unbiased\u2019 algorithms can be inadvertently racist . TECH Insider , April 21, 2016 . http: \/\/www.techinsider.io\/how-algorithms-can-be-racist- 2016 - 2014 . Rafi Letzter. Amazon just showed us that \u2018unbiased\u2019 algorithms can be inadvertently racist. TECH Insider, April 21, 2016. http: \/\/www.techinsider.io\/how-algorithms-can-be-racist-2016-4."},{"key":"e_1_3_2_1_37_1","article-title":"On Orbitz, Mac users steered to pricier hotels","author":"Mattioli Dana","year":"2012","unstructured":"Dana Mattioli . On Orbitz, Mac users steered to pricier hotels . The Wall Street Journal , August 23, 2012 . http:\/\/www.wsj.com\/articles\/ SB10001424052702304458604577488822667325882. Dana Mattioli. On Orbitz, Mac users steered to pricier hotels. The Wall Street Journal, August 23, 2012. http:\/\/www.wsj.com\/articles\/ SB10001424052702304458604577488822667325882.","journal-title":"The Wall Street Journal"},{"issue":"3","key":"e_1_3_2_1_38_1","first-page":"59","article-title":"Causality in databases","volume":"33","author":"Meliou Alexandra","year":"2010","unstructured":"Alexandra Meliou , Wolfgang Gatterbauer , Joseph Y. Halpern , Christoph Koch , Katherine F. Moore , and Dan Suciu . Causality in databases . IEEE Data Engineering Bulletin , 33 ( 3 ): 59 \u2013 67 , 2010 . Alexandra Meliou, Wolfgang Gatterbauer, Joseph Y. Halpern, Christoph Koch, Katherine F. Moore, and Dan Suciu. Causality in databases. IEEE Data Engineering Bulletin, 33(3):59\u201367, 2010.","journal-title":"IEEE Data Engineering Bulletin"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.14778\/1880172.1880176"},{"key":"e_1_3_2_1_40_1","volume-title":"3rd USENIX Workshop on the Theory and Practice of Provenance (TaPP)","author":"Meliou Alexandra","year":"2011","unstructured":"Alexandra Meliou , Wolfgang Gatterbauer , and Dan Suciu . Bringing provenance to its full potential using causal reasoning . In 3rd USENIX Workshop on the Theory and Practice of Provenance (TaPP) , June 2011 . Alexandra Meliou, Wolfgang Gatterbauer, and Dan Suciu. Bringing provenance to its full potential using causal reasoning. In 3rd USENIX Workshop on the Theory and Practice of Provenance (TaPP), June 2011."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.14778\/2733004.2733070"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2494580"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2771783.2771792"},{"key":"e_1_3_2_1_44_1","volume-title":"CNN Money","author":"Olson Parmy","year":"2011","unstructured":"Parmy Olson . The algorithm that beats your bank manager . CNN Money , March 15, 2011 . http:\/\/www.forbes.com\/sites\/parmyolson\/2011\/03\/15\/ the-algorithm-that-beats-your-bank-manager\/#cd84e4f77ca8. Parmy Olson. The algorithm that beats your bank manager. CNN Money, March 15, 2011. http:\/\/www.forbes.com\/sites\/parmyolson\/2011\/03\/15\/ the-algorithm-that-beats-your-bank-manager\/#cd84e4f77ca8."},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2007.37"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1214\/09-SS057"},{"key":"e_1_3_2_1_47_1","volume-title":"NPR All Things Considered","author":"Shahani Aarti","year":"2015","unstructured":"Aarti Shahani . Now algorithms are deciding whom to hire, based on voice . NPR All Things Considered , March 2015 . Aarti Shahani. Now algorithms are deciding whom to hire, based on voice. NPR All Things Considered, March 2015."},{"key":"e_1_3_2_1_48_1","first-page":"252","volume-title":"International Conference on Extending Database Technology (EDBT)","author":"Stoyanovich Julia","year":"2018","unstructured":"Julia Stoyanovich , Ke Yang , and HV Jagadish . Online set selection with fairness and diversity constraints . In International Conference on Extending Database Technology (EDBT) , pages 241\u2013 252 , Vienna, Austria , 2018 . Julia Stoyanovich, Ke Yang, and HV Jagadish. Online set selection with fairness and diversity constraints. In International Conference on Extending Database Technology (EDBT), pages 241\u2013252, Vienna, Austria, 2018."},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSPEC.2016.7473150"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2447976.2447990"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-1606"},{"key":"e_1_3_2_1_52_1","volume-title":"Bruno Castro da Silva abd Andrew G. Barto, and Emma Brunskill. On ensuring that intelligent machines are well-behaved. CoRR, abs\/1708.05448","author":"Thomas Philip S.","year":"2017","unstructured":"Philip S. Thomas , Bruno Castro da Silva abd Andrew G. Barto, and Emma Brunskill. On ensuring that intelligent machines are well-behaved. CoRR, abs\/1708.05448 , 2017 . Philip S. Thomas, Bruno Castro da Silva abd Andrew G. Barto, and Emma Brunskill. On ensuring that intelligent machines are well-behaved. CoRR, abs\/1708.05448, 2017."},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP.2017.29"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2750549"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2899388"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3035925"},{"key":"e_1_3_2_1_57_1","volume-title":"Manuel Gomez Rodriguez, and Krishna P Gummadi. Learning fair classifiers. CoRR, abs\/1507.05259","author":"Zafar Muhammad Bilal","year":"2015","unstructured":"Muhammad Bilal Zafar , Isabel Valera , Manuel Gomez Rodriguez, and Krishna P Gummadi. Learning fair classifiers. CoRR, abs\/1507.05259 , 2015 . Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Learning fair classifiers. CoRR, abs\/1507.05259, 2015."},{"key":"e_1_3_2_1_58_1","volume-title":"International Conference on Machine Learning (ICML), published in JMLR W&CP: 28(3):325\u2013333)","author":"Zemel Richard","year":"2013","unstructured":"Richard Zemel , Yu (Ledell) Wu , Kevin Swersky , Toniann Pitassi , and Cynthia Dwork . Learning fair representations . In International Conference on Machine Learning (ICML), published in JMLR W&CP: 28(3):325\u2013333) , June 2013 . Richard Zemel, Yu (Ledell) Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning (ICML), published in JMLR W&CP: 28(3):325\u2013333), June 2013."},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2011.72"}],"event":{"name":"ESEC\/FSE '18: 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Lake Buena Vista FL USA","acronym":"ESEC\/FSE '18","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3236024.3264590","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3236024.3264590","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3236024.3264590","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:57:27Z","timestamp":1750208247000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3236024.3264590"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,10,26]]},"references-count":59,"alternative-id":["10.1145\/3236024.3264590","10.1145\/3236024"],"URL":"https:\/\/doi.org\/10.1145\/3236024.3264590","relation":{},"subject":[],"published":{"date-parts":[[2018,10,26]]},"assertion":[{"value":"2018-10-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}