{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T06:57:46Z","timestamp":1775199466429,"version":"3.50.1"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2018,2,22]],"date-time":"2018-02-22T00:00:00Z","timestamp":1519257600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGMOD Rec."],"published-print":{"date-parts":[[2018,2,22]]},"abstract":"<jats:p>Bias in online information has recently become a pressing issue, with search engines, social networks and recommendation services being accused of exhibiting some form of bias. In this vision paper, we make the case for a systematic approach towards measuring bias. To this end, we discuss formal measures for quantifying the various types of bias, we outline the system components necessary for realizing them, and we highlight the related research challenges and open problems.<\/jats:p>","DOI":"10.1145\/3186549.3186553","type":"journal-article","created":{"date-parts":[[2018,2,23]],"date-time":"2018-02-23T16:40:01Z","timestamp":1519404001000},"page":"16-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":39,"title":["On Measuring Bias in Online Information"],"prefix":"10.1145","volume":"46","author":[{"given":"Evaggelia","family":"Pitoura","sequence":"first","affiliation":[{"name":"University of Ioannina, Ioannina, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Panayiotis","family":"Tsaparas","sequence":"additional","affiliation":[{"name":"University of Ioannina, Ioannina, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Giorgos","family":"Flouris","sequence":"additional","affiliation":[{"name":"FORTH, Heraklion, Crete, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Irini","family":"Fundulaki","sequence":"additional","affiliation":[{"name":"FORTH, Heraklion, Crete, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Panagiotis","family":"Papadakos","sequence":"additional","affiliation":[{"name":"FORTH, Heraklion, Crete, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Serge","family":"Abiteboul","sequence":"additional","affiliation":[{"name":"INRIA&amp;ENS, Paris, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gerhard","family":"Weikum","sequence":"additional","affiliation":[{"name":"Max Planck Institute for Informatics, Saarbr\u00fccken, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,2,22]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"2","volume-title":"De-biasing user preference ratings in recommender systems","author":"Adomavicius G.","year":"2014","unstructured":"G. Adomavicius , J. Bockstedt , C. Shawn , and J. Zhang . De-biasing user preference ratings in recommender systems , volume 1253 , pages 2 -- 9 . CEUR-WS , 2014 . G. Adomavicius, J. Bockstedt, C. Shawn, and J. Zhang. De-biasing user preference ratings in recommender systems, volume 1253, pages 2--9. CEUR-WS, 2014."},{"key":"e_1_2_1_2_1","volume-title":"Big Data's Disparate Impact. SSRN eLibrary","author":"Barocas S.","year":"2014","unstructured":"S. Barocas and A. D. Selbst . Big Data's Disparate Impact. SSRN eLibrary , 2014 . S. Barocas and A. D. Selbst. Big Data's Disparate Impact. SSRN eLibrary, 2014."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-013-9321-6"},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"C. Budak S. Goel and J. M. Rao. Fair and balanced? quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly 80(S1).  C. Budak S. Goel and J. M. Rao. Fair and balanced? quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly 80(S1).","DOI":"10.1093\/poq\/nfw007"},{"key":"e_1_2_1_5_1","volume-title":"Computation and Journalism Symposium","author":"Chakraborty A.","year":"2015","unstructured":"A. Chakraborty , S. Ghosh , N. Ganguly , and K. P. Gummadi . Can trending news stories create coverage bias? on the impact of high content churn in online news media . In Computation and Journalism Symposium , 2015 . A. Chakraborty, S. Ghosh, N. Ganguly, and K. P. Gummadi. Can trending news stories create coverage bias? on the impact of high content churn in online news media. In Computation and Journalism Symposium, 2015."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098095"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1515\/popets-2015-0007"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1860702.1860709"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_2_1_10_1","volume-title":"The search engine manipulation effect (seme) and its possible impact on the outcomes of elections. PNAS, 112(20)","author":"Epstein R.","year":"2015","unstructured":"R. Epstein and R. E. Robertson . The search engine manipulation effect (seme) and its possible impact on the outcomes of elections. PNAS, 112(20) , 2015 . R. Epstein and R. E. Robertson. The search engine manipulation effect (seme) and its possible impact on the outcomes of elections. PNAS, 112(20), 2015."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_2_1_12_1","volume-title":"The case for temporal transparency: Detecting policy change events in black-box decision making systems. arXiv preprint arXiv:1610.10064","author":"Ferreira M.","year":"2016","unstructured":"M. Ferreira , M. B. Zafar , and K. P. Gummadi . The case for temporal transparency: Detecting policy change events in black-box decision making systems. arXiv preprint arXiv:1610.10064 , 2016 . M. Ferreira, M. B. Zafar, and K. P. Gummadi. The case for temporal transparency: Detecting policy change events in black-box decision making systems. arXiv preprint arXiv:1610.10064, 2016."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611974348.17"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0605525103"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2945386"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488435"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2663716.2663744"},{"key":"e_1_2_1_18_1","first-page":"3315","volume-title":"NIPS","author":"Hardt M.","year":"2016","unstructured":"M. Hardt , E. Price , and N. Srebro . Equality of opportunity in supervised learning . In NIPS , pages 3315 -- 3323 , 2016 . M. Hardt, E. Price, and N. Srebro. Equality of opportunity in supervised learning. In NIPS, pages 3315--3323, 2016."},{"key":"e_1_2_1_19_1","volume-title":"Big data: A report on algorithmic systems, opportunity, and civil rights","author":"House W.","year":"2016","unstructured":"W. House . Big data: A report on algorithmic systems, opportunity, and civil rights . Washington, DC : Executive Office of the President, White House , 2016 . W. House. Big data: A report on algorithmic systems, opportunity, and civil rights. Washington, DC: Executive Office of the President, White House, 2016."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741099"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998321"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-13734-6_25"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2004.05.005"},{"key":"e_1_2_1_24_1","volume-title":"SSNR Preprint","author":"Olteanu A.","year":"2017","unstructured":"A. Olteanu , C. Castillo , F. Diaz , and E. Kiciman . Social data: Biases, methodological pitfalls, and ethical boundaries . In SSNR Preprint , 2017 . A. Olteanu, C. Castillo, F. Diaz, and E. Kiciman. Social data: Biases, methodological pitfalls, and ethical boundaries. In SSNR Preprint, 2017."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888913000039"},{"key":"e_1_2_1_26_1","volume-title":"Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry","author":"Sandvig C.","year":"2014","unstructured":"C. Sandvig , K. Hamilton , K. Karahalios , and C. Langbort . Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry , 2014 . C. Sandvig, K. Hamilton, K. Karahalios, and C. Langbort. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry, 2014."},{"key":"e_1_2_1_27_1","volume-title":"EDBT","author":"Stoyanovich J.","year":"2016","unstructured":"J. Stoyanovich , S. Abiteboul , and G. Miklau . Data, responsibly: Fairness, neutrality and transparency in data analysis . In EDBT , 2016 . J. Stoyanovich, S. Abiteboul, and G. Miklau. Data, responsibly: Fairness, neutrality and transparency in data analysis. In EDBT, 2016."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460276.2460278"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(03)00063-3"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2492517.2492557"},{"key":"e_1_2_1_31_1","first-page":"640","article-title":"Quantifying political leaning from tweets and retweets","volume":"13","author":"Wong F. M. F.","year":"2013","unstructured":"F. M. F. Wong , C. W. Tan , S. Sen , and M. Chiang . Quantifying political leaning from tweets and retweets . ICWSM , 13 : 640 -- 649 , 2013 . F. M. F. Wong, C. W. Tan, S. Sen, and M. Chiang. Quantifying political leaning from tweets and retweets. ICWSM, 13:640--649, 2013.","journal-title":"ICWSM"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3085504.3085526"},{"key":"e_1_2_1_33_1","unstructured":"M. B. Zafar K. P. Gummadi and C. Danescu-Niculescu-Mizil. Message impartiality in social media discussions. In ICWSM.  M. B. Zafar K. P. Gummadi and C. Danescu-Niculescu-Mizil. Message impartiality in social media discussions. In ICWSM."},{"key":"e_1_2_1_34_1","volume-title":"Learning fair classifiers. arXiv preprint arXiv:1507.05259","author":"Zafar M. B.","year":"2015","unstructured":"M. B. Zafar , I. Valera , M. G. Rodriguez , and K. P. Gummadi . Learning fair classifiers. arXiv preprint arXiv:1507.05259 , 2015 . M. B. Zafar, I. Valera, M. G. Rodriguez, and K. P. Gummadi. Learning fair classifiers. arXiv preprint arXiv:1507.05259, 2015."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052660"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132938"},{"key":"e_1_2_1_37_1","volume-title":"Men also like shopping: Reducing gender bias amplification using corpus-level constraints. CoRR, abs\/1707.09457","author":"Zhao J.","year":"2017","unstructured":"J. Zhao , T. Wang , M. Yatskar , V. Ordonez , and K. Chang . Men also like shopping: Reducing gender bias amplification using corpus-level constraints. CoRR, abs\/1707.09457 , 2017 . J. Zhao, T. Wang, M. Yatskar, V. Ordonez, and K. Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. CoRR, abs\/1707.09457, 2017."}],"container-title":["ACM SIGMOD Record"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3186549.3186553","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3186549.3186553","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:11:27Z","timestamp":1750212687000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3186549.3186553"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,2,22]]},"references-count":37,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2018,2,22]]}},"alternative-id":["10.1145\/3186549.3186553"],"URL":"https:\/\/doi.org\/10.1145\/3186549.3186553","relation":{},"ISSN":["0163-5808"],"issn-type":[{"value":"0163-5808","type":"print"}],"subject":[],"published":{"date-parts":[[2018,2,22]]},"assertion":[{"value":"2018-02-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}