{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T07:19:23Z","timestamp":1779175163961,"version":"3.51.4"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2020,8]]},"abstract":"<jats:p>We frequently compute a score for each item in a data set, sometimes for its intrinsic value, but more often as a step towards classification, ranking, and so forth. The importance of computing this score fairly cannot be overstated. In this tutorial, we will develop a framework for how to think about this task, and then present techniques for responsible scoring and link these to traditional data management challenges.<\/jats:p>","DOI":"10.14778\/3415478.3415566","type":"journal-article","created":{"date-parts":[[2020,9,14]],"date-time":"2020-09-14T18:46:40Z","timestamp":1600109200000},"page":"3445-3448","source":"Crossref","is-referenced-by-count":19,"title":["Fairly evaluating and scoring items in a data set"],"prefix":"10.14778","volume":"13","author":[{"given":"Abolfazl","family":"Asudeh","sequence":"first","affiliation":[{"name":"University of Illinois at Chicago"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"H. V.","family":"Jagadish","sequence":"additional","affiliation":[{"name":"University of Michigan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,8]]},"reference":[{"key":"e_1_2_1_1_1","article-title":"The secret trust scores companies use to judge us all","author":"Mason J.","year":"2019","unstructured":"J. Mason. The secret trust scores companies use to judge us all. The Wall Street Journal, April 6, 2019.","journal-title":"The Wall Street Journal"},{"key":"e_1_2_1_2_1","volume-title":"bcrisktool.cancer.gov, (accessed","year":"2020","unstructured":"The breast cancer risk assessment tool. bcrisktool.cancer.gov, (accessed March 2020)."},{"key":"e_1_2_1_3_1","unstructured":"World cup 2018 seeding: Pots procedure & all you need to know ahead of the draw. GOAL.COM 12\/1\/2017."},{"key":"e_1_2_1_4_1","unstructured":"How u.s. news calculated the 2020 best graduate schools rankings. bit.ly\/39HjnGQ 3\/11\/2019."},{"key":"e_1_2_1_5_1","volume-title":"The New York Times","author":"Mozur P.","year":"2020","unstructured":"P. Mozur, R. Zhong, and A. Krolik. In coronavirus fight, china gives citizens a color code, with red flags. The New York Times, March 1, 2020."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2019.00013"},{"key":"e_1_2_1_7_1","first-page":"671","article-title":"Big data's disparate impact","volume":"104","author":"Barocas S.","year":"2016","unstructured":"S. Barocas and A. D. Selbst. Big data's disparate impact. Calif. L. Rev., 104:671, 2016.","journal-title":"Calif. L. Rev."},{"key":"e_1_2_1_8_1","volume-title":"ACT Research & Policy","author":"Buddin R.","year":"2014","unstructured":"R. Buddin. Gender gaps in high school gpa and act scores. ACT Research & Policy, 2014."},{"key":"e_1_2_1_9_1","volume-title":"WOSU Public Media","author":"Roth C.","year":"2018","unstructured":"C. Roth. Women job applicants punished for higher grades, study finds. WOSU Public Media, Mar 26, 2018."},{"key":"e_1_2_1_10_1","first-page":"109","article-title":"Disparate impact in big data policing","volume":"52","author":"Selbst A. D.","year":"2017","unstructured":"A. D. Selbst. Disparate impact in big data policing. Ga. L. Rev., 52:109, 2017.","journal-title":"Ga. L. Rev."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1080\/00221546.2010.11779074"},{"key":"e_1_2_1_12_1","volume-title":"Algorithm bias in credit scoring: What's inside the black box? CGAP blog","author":"Vidal M. F.","year":"2019","unstructured":"M. F. Vidal and J. Menajovsky. Algorithm bias in credit scoring: What's inside the black box? CGAP blog, 2019."},{"key":"e_1_2_1_13_1","first-page":"857","article-title":"Data-driven discrimination at work","volume":"58","author":"Kim P. T.","year":"2016","unstructured":"P. T. Kim. Data-driven discrimination at work. Wm. & Mary L. Rev., 58:857, 2016.","journal-title":"Wm. & Mary L. Rev."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-017-0506-1"},{"key":"e_1_2_1_15_1","volume-title":"Fairness and machine learning: Limitations and opportunities. fairmlbook.org","author":"Barocas S.","year":"2019","unstructured":"S. Barocas, M. Hardt, and A. Narayanan. Fairness and machine learning: Limitations and opportunities. fairmlbook.org, 2019."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098095"},{"key":"e_1_2_1_17_1","volume-title":"FAT*","author":"Menon A. K.","year":"2018","unstructured":"A. K. Menon and R. C. Williamson. The cost of fairness in binary classification. In FAT*, 2018."},{"key":"e_1_2_1_18_1","volume-title":"Inherent trade-offs in the fair determination of risk scores. CoRR, abs\/1609.05807","author":"Kleinberg J.","year":"2016","unstructured":"J. Kleinberg, S. Mullainathan, and M. Raghavan. Inherent trade-offs in the fair determination of risk scores. CoRR, abs\/1609.05807, 2016."},{"key":"e_1_2_1_19_1","volume-title":"On the (im) possibility of fairness. CoRR, abs\/1609.07236","author":"Friedler S. A.","year":"2016","unstructured":"S. A. Friedler, C. Scheidegger, and S. Venkatasubramanian. On the (im) possibility of fairness. CoRR, abs\/1609.07236, 2016."},{"key":"e_1_2_1_20_1","volume-title":"Statistical Research Memoirs","author":"Neyman J.","year":"1936","unstructured":"J. Neyman and E. S. Pearson. Contributions to the theory of testing statistical hypotheses. Statistical Research Memoirs, 1936."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2019.00056"},{"key":"e_1_2_1_22_1","volume-title":"AMW","author":"Baeza-Yates R. A.","year":"2013","unstructured":"R. A. Baeza-Yates. Big data or right data? In AMW, 2013."},{"key":"e_1_2_1_23_1","volume-title":"FAT*","author":"Narayanan A.","year":"2018","unstructured":"A. Narayanan. Translation tutorial: 21 fairness definitions and their politics. In FAT*, 2018."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287592"},{"key":"e_1_2_1_26_1","first-page":"2564","volume-title":"ICML","author":"Kearns M.","year":"2018","unstructured":"M. Kearns, S. Neel, A. Roth, and Z. S. Wu. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In ICML, pages 2564--2572, 2018."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0054"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.14778\/3291264.3291269"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287589"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/AAI28115249"},{"issue":"3","key":"e_1_2_1_31_1","first-page":"76","article-title":"Towards responsible data-driven decision making in score-based systems","volume":"42","author":"Asudeh A.","year":"2019","unstructured":"A. Asudeh, H. Jagadish, and J. Stoyanovich. Towards responsible data-driven decision making in score-based systems. Data Engineering, 42(3):76--87, 2019.","journal-title":"Data Engineering"},{"key":"e_1_2_1_32_1","volume-title":"MIT-STS-AI-snakeoil.pdf","author":"Narayanan A.","year":"2019","unstructured":"A. Narayanan. How to recognize ai snake oil www.cs.princeton.edu\/~arvindn\/talks. Technical report, MIT-STS-AI-snakeoil.pdf, 2019."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-011-0463-8"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_2_1_35_1","first-page":"3992","volume-title":"NIPS","author":"Calmon F.","year":"2017","unstructured":"F. Calmon, D. Wei, B. Vinzamuri, K. N. Ramamurthy, and K. R. Varshney. Optimized pre-processing for discrimination prevention. In NIPS, pages 3992--4001, 2017."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3319901"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3384689"},{"issue":"11","key":"e_1_2_1_38_1","first-page":"2229","article-title":"Identifying insufficient data coverage in databases with multiple relations","volume":"13","author":"Lin Y.","year":"2020","unstructured":"Y. Lin, Y. Guan, A. Asudeh, and J. H. V. Identifying insufficient data coverage in databases with multiple relations. PVLDB, 13(11):2229--2242, 2020.","journal-title":"PVLDB"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3357853"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33486-3_3"},{"key":"e_1_2_1_41_1","volume-title":"ICML","author":"Zemel R.","year":"2013","unstructured":"R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork. Learning fair representations. In ICML, 2013."},{"key":"e_1_2_1_42_1","volume-title":"Fairness constraints: Mechanisms for fair classification. CoRR, abs\/1507.05259","author":"Zafar M. B.","year":"2015","unstructured":"M. B. Zafar, I. Valera, M. G. Rodriguez, and K. P. Gummadi. Fairness constraints: Mechanisms for fair classification. CoRR, abs\/1507.05259, 2015."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3300079"},{"key":"e_1_2_1_44_1","volume-title":"Responsible scoring mechanisms through function sampling. CoRR, abs\/:1911.10073","author":"Asudeh A.","year":"2019","unstructured":"A. Asudeh and H. Jagadish. Responsible scoring mechanisms through function sampling. CoRR, abs\/:1911.10073, 2019."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3320244"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.14778\/3380750.3380762"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3415478.3415566","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,17]],"date-time":"2025-09-17T02:13:47Z","timestamp":1758075227000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3415478.3415566"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8]]},"references-count":46,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2020,8]]}},"alternative-id":["10.14778\/3415478.3415566"],"URL":"https:\/\/doi.org\/10.14778\/3415478.3415566","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2020,8]]}}}