{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T02:52:41Z","timestamp":1776394361183,"version":"3.51.2"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,11,25]],"date-time":"2020-11-25T00:00:00Z","timestamp":1606262400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"American University of Beirut Research Board","award":["103603"],"award-info":[{"award-number":["103603"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM\/IMS Trans. Data Sci."],"published-print":{"date-parts":[[2020,11,30]]},"abstract":"<jats:p>We study fairness of scoring in online job marketplaces. We focus on group fairness and aim to algorithmically explore how a scoring function, through which individuals are ranked for jobs, treats different demographic groups. Previous work on group-level fairness has focused on the case where groups are pre-defined or where they are defined using a single protected attribute (e.g., whites vs. blacks or males vs. females). In this article, we argue for the need to examine fairness for groups of people defined with any combination of protected attributes (the-so called subgroup fairness). Existing work also assumes the availability of worker\u2019s data (i.e., data transparency) and the scoring function (i.e., process transparency). We relax that assumption in this work and run user studies to assess the effect of different data and process transparency settings on the ability to assess fairness.<\/jats:p>\n                  <jats:p>To quantify the fairness of a scoring of a group of individuals, we formulate an optimization problem to find a partitioning of those individuals on their protected attributes that exhibits the highest unfairness with respect to the scoring function. The scoring function yields one histogram of score distributions per partition and we rely on Earth Mover\u2019s Distance, a measure that is commonly used to compare histograms, to quantify unfairness. Since the number of ways to partition individuals is exponential in the number of their protected attributes, we propose a heuristic algorithm to navigate the space of all possible partitionings to identify the one with the highest unfairness. We evaluate our algorithm using a simulation of a crowdsourcing platform and show that it can effectively quantify unfairness of various scoring functions. We additionally run experiments to assess the applicability of our approach in other less-transparent data and process settings. Finally, we demonstrate the effectiveness of our approach in assessing fairness of scoring in a real dataset crawled from the online job marketplace TaskRabbit.<\/jats:p>","DOI":"10.1145\/3402883","type":"journal-article","created":{"date-parts":[[2020,11,26]],"date-time":"2020-11-26T14:24:23Z","timestamp":1606400663000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Fairness of Scoring in Online Job Marketplaces"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3491-6311","authenticated-orcid":false,"given":"Shady","family":"Elbassuoni","sequence":"first","affiliation":[{"name":"American University of Beirut, Hamra, Beirut, Lebanon"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sihem","family":"Amer-Yahia","sequence":"additional","affiliation":[{"name":"CNRS, University of Grenoble Alpes, Grenoble, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmad","family":"Ghizzawi","sequence":"additional","affiliation":[{"name":"American University of Beirut, Hamra, Beirut, Lebanon"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,11,25]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2945386"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS\u201917)","author":"Zafar Muhammad Bilal","unstructured":"Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. 2017. Fairness constraints: Mechanisms for fair classification. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS\u201917). 962--970."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 35th International Conference on Machine Learning (ICML\u201918)","author":"Kearns Michael J.","year":"2018","unstructured":"Michael J. Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In Proceedings of the 35th International Conference on Machine Learning (ICML\u201918). 2569--2577."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-010-0190-x"},{"key":"e_1_2_1_6_1","volume-title":"A survey on measuring indirect discrimination in machine learning. CoRR abs\/1511.00148","author":"Zliobaite Indre","year":"2015","unstructured":"Indre Zliobaite. 2015. A survey on measuring indirect discrimination in machine learning. CoRR abs\/1511.00148 (2015). Retrieved from http:\/\/arxiv.org\/abs\/1511.00148."},{"key":"e_1_2_1_7_1","unstructured":"Sorelle A. Friedler Carlos Scheidegger and Suresh Venkatasubramanian. 2016. On the (im)possibility of fairness. arxiv:1609.07236. Retrieved from http:\/\/arxiv.org\/abs\/1609.07236."},{"key":"e_1_2_1_8_1","first-page":"728","article-title":"The shackled runner: Time to rethink positive discrimination? Work Employ","volume":"24","author":"Noon Mike","year":"2010","unstructured":"Mike Noon. 2010. The shackled runner: Time to rethink positive discrimination? Work Employ. Soc. 24, 4 (2010), 728--739.","journal-title":"Soc."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998327"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. IEEE, 460--467","author":"Pele Ofir","year":"2009","unstructured":"Ofir Pele and Michael Werman. 2009. Fast and robust earth mover\u2019s distances. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. IEEE, 460--467."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.35.7.485"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/352871.352889"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009744630224"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1468-0297.2005.01032.x"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the International Conference on Extending Database Technology (EDBT\u201919)","author":"Ghizzawi Ahmad","year":"2019","unstructured":"Ahmad Ghizzawi, Julien Marinescu, Shady Elbassuoni, Sihem Amer-Yahia, and Gilles Bisson. 2019. FaiRank: An interactive system to explore fairness of ranking in online job marketplaces. In Proceedings of the International Conference on Extending Database Technology (EDBT\u201919)."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the International Conference on Extending Database Technology (EDBT\u201919)","author":"Elbassuoni Shady","year":"2019","unstructured":"Shady Elbassuoni, Sihem Amer-Yahia, Ahmad Ghizzawi, and Christine El Atie. 2019. Exploring fairness of ranking in online job marketplaces. In Proceedings of the International Conference on Extending Database Technology (EDBT\u201919)."},{"key":"e_1_2_1_17_1","unstructured":"SurveyMonkey. Calculating the number of respondents you need. Retrieved from https:\/\/help.surveymonkey.com\/articles\/en_US\/kb\/How-many-respondents-do-I-need."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1287\/moor.1060.0207"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1142\/S0218488502001648"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1930297500005611"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/69.971193"},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1145\/2983270","article-title":"Battling algorithmic bias: How do we ensure algorithms treat us fairly","volume":"59","author":"Kirkpatrick Keith","year":"2016","unstructured":"Keith Kirkpatrick. 2016. Battling algorithmic bias: How do we ensure algorithms treat us fairly? Commun. ACM 59, 10 (2016), 16--17.","journal-title":"Commun. ACM"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Latanya Sweeney. 2013. Discrimination in online ad delivery. arxiv:1301.6822. Retrieved from http:\/\/arxiv.org\/abs\/1301.6822.","DOI":"10.2139\/ssrn.2208240"},{"key":"e_1_2_1_24_1","unstructured":"Florian Tram\u00e8r Vaggelis Atlidakis Roxana Geambasu Daniel J. Hsu Jean-Pierre Hubaux Mathias Humbert Ari Juels and Huang Lin. 2015. Discovering unwarranted associations in data-driven applications with the fairtest testing toolkit. arxiv:1510.02377. Retrieved from http:\/\/arxiv.org\/abs\/1510.02377."},{"key":"e_1_2_1_25_1","first-page":"1","article-title":"Four reference models for transparency requirements in information systems","volume":"23","author":"Hosseini Mahmood","year":"2017","unstructured":"Mahmood Hosseini, Alimohammad Shahri, Keith Phalp, and Raian Ali. 2017. Four reference models for transparency requirements in information systems. Requir. Eng. 23, 2 (2017), 1--25.","journal-title":"Requir. Eng."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/0304-3975(85)90224-5"},{"key":"e_1_2_1_27_1","first-page":"1183","article-title":"Designing against discrimination in online markets","volume":"32","author":"Levy Karen","year":"2017","unstructured":"Karen Levy and Solon Barocas. 2017. Designing against discrimination in online markets. Berkeley Tech. LJ 32 (2017), 1183.","journal-title":"Berkeley Tech. LJ"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1002\/poi3.153"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1257\/app.20160213"},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","unstructured":"David Durward Ivo Blohm and Jan Marco Leimeister. 2016. Is there PAPA in crowd work?: A literature review on ethical dimensions in crowdsourcing. In Proceedings of the 2016 International on IEEE Conferences on Ubiquitous Intelligence 8 Computing Advanced and Trusted Computing Scalable Computing and Communications Cloud and Big Data Computing Internet of People and Smart World Congress (UIC\/ATC\/ScalCom\/CBDCom\/IoP\/SmartWorld\u201916). IEEE 823--832.","DOI":"10.1109\/UIC-ATC-ScalCom-CBDCom-IoP-SmartWorld.2016.0131"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 20th International Conference on Extending Database Technology (EDBT\u201917)","author":"Borromeo Ria Mae","year":"2017","unstructured":"Ria Mae Borromeo, Thomas Laurent, Motomichi Toyama, and Sihem Amer-Yahia. 2017. Fairness and transparency in crowdsourcing. In Proceedings of the 20th International Conference on Extending Database Technology (EDBT\u201917). 466--469. DOI:http:\/\/dx.doi.org\/10.5441\/002\/edbt.2017.46"},{"key":"e_1_2_1_32_1","article-title":"Fixing discrimination in online marketplaces. Harv","author":"Luca Michael","year":"2016","unstructured":"Michael Luca and Rayl Fisman. 2016. Fixing discrimination in online marketplaces. Harv. Bus. Rev. (December 2016).","journal-title":"Bus. Rev."},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI\u201915)","author":"Hanrahan Benjamin V.","unstructured":"Benjamin V. Hanrahan, Jutta K. Willamowski, Saiganesh Swaminathan, and David B. Martin. 2015. TurkBench: Rendering the market for turkers. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI\u201915), Bo Begole, Jinwoo Kim, Kori Inkpen, and Woontack Woo (Eds.). ACM, 1613--1616."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v2i1.13198"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132938"},{"key":"e_1_2_1_36_1","volume-title":"Vishnoi","author":"Celis L. Elisa","year":"2017","unstructured":"L. Elisa Celis, Damian Straszak, and Nisheeth K. Vishnoi. 2017. Ranking with fairness constraints. arxiv:1704.06840. Retrieved from https:\/\/arxiv.org\/abs\/1704.06840."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3085504.3085526"},{"key":"e_1_2_1_38_1","unstructured":"Ashudeep Singh and Thorsten Joachims. 2018. Fairness of exposure in rankings. arxiv:1802.07281. Retrieved from https:\/\/arxiv.org\/abs\/1802.07281."},{"key":"e_1_2_1_39_1","unstructured":"Asia J. Biega Krishna P. Gummadi and Gerhard Weikum. 2018. Equity of attention: Amortizing individual fairness in rankings. arxiv:1805.01788. Retrieved from https:\/\/arxiv.org\/abs\/1805.01788."}],"container-title":["ACM\/IMS Transactions on Data Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3402883","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3402883","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T13:56:17Z","timestamp":1776347777000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3402883"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,25]]},"references-count":39,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,11,30]]}},"alternative-id":["10.1145\/3402883"],"URL":"https:\/\/doi.org\/10.1145\/3402883","relation":{},"ISSN":["2691-1922"],"issn-type":[{"value":"2691-1922","type":"print"}],"subject":[],"published":{"date-parts":[[2020,11,25]]},"assertion":[{"value":"2019-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-05-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-11-25","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}