{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T23:49:35Z","timestamp":1783468175977,"version":"3.55.0"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,2,18]],"date-time":"2026-02-18T00:00:00Z","timestamp":1771372800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/deed.de"},{"start":{"date-parts":[[2026,2,18]],"date-time":"2026-02-18T00:00:00Z","timestamp":1771372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/deed.de"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Technische Hochschule K\u00f6ln"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Datenbank Spektrum"],"published-print":{"date-parts":[[2026,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Linguistic bias in online news and social media is widespread but difficult to measure. Yet, its identification and quantification remain difficult due to subjectivity, context dependence, and the scarcity of high-quality gold-label datasets. We aim to reduce annotation effort by leveraging pairwise comparison for bias annotation. To overcome the costliness of the approach, we evaluate more efficient implementations of pairwise comparison-based rating. We achieve this by investigating the effects of various rating techniques and the parameters of three cost-aware alternatives in a\u00a0simulation environment. Since the approach can in principle be applied to both human and large language model annotation, our work provides a\u00a0basis for creating high-quality benchmark datasets and for quantifying biases and other subjective linguistic aspects.<\/jats:p>\n                  <jats:p>The controlled simulations include latent severity distributions, distance-calibrated noise, and synthetic annotator bias to probe robustness and cost-quality trade-offs. In applying the approach to human-labeled bias benchmark datasets, we then evaluate the most promising setups and compare them to direct assessment by large language models and unmodified pairwise comparison labels as baselines. Our findings support the use of pairwise comparison as a\u00a0practical foundation for quantifying subjective linguistic aspects, enabling reproducible bias analysis. We contribute an optimization of comparison and matchmaking components, an end-to-end evaluation including simulation and real-data application, and an implementation blueprint for cost-aware large-scale annotation.<\/jats:p>","DOI":"10.1007\/s13222-026-00531-1","type":"journal-article","created":{"date-parts":[[2026,2,18]],"date-time":"2026-02-18T14:55:39Z","timestamp":1771426539000},"page":"65-73","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Pairwise Comparison for Bias Identification and Quantification"],"prefix":"10.1007","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3392-7860","authenticated-orcid":false,"given":"Fabian","family":"Haak","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Philipp","family":"Schaer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,2,18]]},"reference":[{"key":"531_CR1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505623","author":"D Saez-Trumper","year":"2013","unstructured":"Saez-Trumper\u00a0D, Castillo\u00a0C, Lalmas\u00a0M (2013) Social Media News Communities: Gatekeeping, Coverage, and Statement Bias. International Conference on Information and Knowledge Management, Proceedings. https:\/\/doi.org\/10.1145\/2505515.2505623","journal-title":"International Conference on Information and Knowledge Management, Proceedings"},{"issue":"3","key":"531_CR2","doi-asserted-by":"publisher","first-page":"548","DOI":"10.1111\/ajps.12424","volume":"63","author":"S Wolton","year":"2019","unstructured":"Wolton\u00a0S (2019) Are Biased Media Bad for Democracy? American Journal of Political Science 63(3):548\u2013562","journal-title":"American Journal of Political Science"},{"key":"531_CR3","doi-asserted-by":"publisher","unstructured":"Spinde T, Hinterreiter S, Haak F, Ruas T, Giese H, Meuschke N, Gipp B (2024) The Media Bias Taxonomy: A Systematic Literature Review on the Forms and Automated Detection of Media Bias. https:\/\/doi.org\/10.48550\/arXiv.2312.16148","DOI":"10.48550\/arXiv.2312.16148"},{"issue":"4","key":"531_CR4","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1037\/h0070288","volume":"34","author":"LL Thurstone","year":"1927","unstructured":"Thurstone\u00a0LL (1927) A\u00a0law of comparative judgment. Psychological Review 34(4):273\u2013286","journal-title":"Psychological Review"},{"issue":"3\/4","key":"531_CR5","doi-asserted-by":"publisher","first-page":"324","DOI":"10.2307\/2334029","volume":"39","author":"RA Bradley","year":"1952","unstructured":"Bradley\u00a0RA, Terry\u00a0ME (1952) Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika 39(3\/4):324\u2013345","journal-title":"Biometrika"},{"key":"531_CR6","volume-title":"Individual Choice Behavior: A Theoretical Analysis","author":"RD Luce","year":"1959","unstructured":"Luce\u00a0RD (1959) Individual Choice Behavior: A Theoretical Analysis. Wiley, New York"},{"key":"531_CR7","volume-title":"The Rating of Chessplayers, Past and Present","author":"AE Elo","year":"1978","unstructured":"Elo\u00a0AE (1978) The Rating of Chessplayers, Past and Present. Arco, New York"},{"issue":"3","key":"531_CR8","first-page":"377","volume":"48","author":"ME Glickman","year":"1999","unstructured":"Glickman\u00a0ME (1999) Parameter estimation in large dynamic paired comparison experiments. Journal of the Royal Statistical Society: Series C (Applied Statistics) 48(3):377\u2013394","journal-title":"Journal of the Royal Statistical Society: Series C (Applied Statistics)"},{"key":"531_CR9","volume-title":"NeurIPS","author":"R Herbrich","year":"2007","unstructured":"Herbrich\u00a0R, Minka\u00a0T, Graepel\u00a0T (2007) Trueskill\u2122: A\u00a0bayesian skill rating system. In: NeurIPS"},{"issue":"7","key":"531_CR10","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/0895-7177(93)90059-8","volume":"18","author":"WW Koczkodaj","year":"1993","unstructured":"Koczkodaj\u00a0WW (1993) A new definition of consistency of pairwise comparisons. Mathematical and Computer Modelling 18(7):79\u201384. https:\/\/doi.org\/10.1016\/0895-7177(93)90059-8","journal-title":"Mathematical and Computer Modelling"},{"key":"531_CR11","first-page":"133","volume-title":"KDD","author":"T Joachims","year":"2002","unstructured":"Joachims\u00a0T (2002) Optimizing search engines using clickthrough data. In: KDD, pp\u00a0133\u2013142"},{"key":"531_CR12","doi-asserted-by":"crossref","unstructured":"Boubdir M, Kim E, Ermis B, Hooker S, Fadaee M (2023) Elo Uncovered: Robustness and Best Practices in Language Model Evaluation.","DOI":"10.52202\/079017-3367"},{"key":"531_CR13","volume-title":"Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023","author":"L Zheng","year":"2023","unstructured":"Zheng\u00a0L, Chiang\u00a0W, Sheng\u00a0Y, Zhuang\u00a0S, Wu\u00a0Z, Zhuang\u00a0Y, Lin\u00a0Z, Li\u00a0Z, Li\u00a0D, Xing\u00a0EP, Zhang\u00a0H, Gonzalez\u00a0JE, Stoica\u00a0I (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. In: Oh\u00a0A, Naumann\u00a0T, Globerson\u00a0A, Saenko\u00a0K, Hardt\u00a0M, Levine\u00a0S (eds) Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023"},{"key":"531_CR14","unstructured":"Zheng, L., Sheng, Y., et\u00a0al.: Chatbot arena: An open platform for evaluating llms. In: ICLR Tiny Papers (2024)"},{"key":"531_CR15","doi-asserted-by":"publisher","first-page":"14925","DOI":"10.18653\/v1\/2024.findings-emnlp.877","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2024","author":"B Engelmann","year":"2024","unstructured":"Engelmann\u00a0B, Kreutz\u00a0CK, Haak\u00a0F, Schaer\u00a0P (2024) ARTS: Assessing Readability & Text Simplicity. In: Al-Onaizan\u00a0Y, Bansal\u00a0M, Chen\u00a0YN (eds) Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, Miami, Florida, USA, pp\u00a014925\u201314942 https:\/\/doi.org\/10.18653\/v1\/2024.findings-emnlp.877"},{"key":"531_CR16","doi-asserted-by":"publisher","first-page":"4046","DOI":"10.1109\/ITSC57777.2023.10422620","volume-title":"2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC)","author":"M Costa","year":"2023","unstructured":"Costa\u00a0M, Marques\u00a0M, Siebert\u00a0FW, Azevedo\u00a0CL, Moura\u00a0F (2023) Scoring cycling environments perceived safety using pairwise image comparisons. In: 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pp\u00a04046\u20134051 https:\/\/doi.org\/10.1109\/ITSC57777.2023.10422620"},{"key":"531_CR17","doi-asserted-by":"publisher","unstructured":"Shibata T, Miyamura Y (2025) LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Models. https:\/\/doi.org\/10.48550\/arXiv.2505.08498","DOI":"10.48550\/arXiv.2505.08498"},{"key":"531_CR18","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2019.8803573","author":"Y Ji","year":"2019","unstructured":"Ji\u00a0Y, Wang\u00a0Y, Katoy\u00a0J (2019) Visual Violence Rating with Pairwise Comparison. Proc Int Conf Image Proc. https:\/\/doi.org\/10.1109\/ICIP.2019.8803573","journal-title":"2019 IEEE International Conference on Image Processing (ICIP)"},{"key":"531_CR19","series-title":"Websci Companion \u201924","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1145\/3630744.3658415","volume-title":"Companion Publication of the 16th ACM Web Science Conference","author":"F Haak","year":"2024","unstructured":"Haak\u00a0F, Engelmann\u00a0B, Kreutz\u00a0CK, Schaer\u00a0P (2024) Investigating Bias in Political Search Query Suggestions by Relative Comparison with LLMs. In: Companion Publication of the 16th ACM Web Science Conference. Websci Companion \u201924. Association for Computing Machinery, New York, NY, USA, pp\u00a05\u20137 https:\/\/doi.org\/10.1145\/3630744.3658415"},{"issue":"1","key":"531_CR20","doi-asserted-by":"publisher","first-page":"384","DOI":"10.1214\/aos\/1079120141","volume":"32","author":"DR Hunter","year":"2004","unstructured":"Hunter\u00a0DR (2004) MM algorithms for generalized Bradley-Terry models. The Annals of Statistics 32(1):384\u2013406. https:\/\/doi.org\/10.1214\/aos\/1079120141","journal-title":"The Annals of Statistics"},{"key":"531_CR21","first-page":"129","volume-title":"ICML","author":"Z Cao","year":"2007","unstructured":"Cao\u00a0Z, Qin\u00a0T, Liu\u00a0TY, Tsai\u00a0MF, Li\u00a0H (2007) Learning to rank: From pairwise approach to listwise approach. In: ICML, pp\u00a0129\u2013136"},{"key":"531_CR22","doi-asserted-by":"publisher","first-page":"759","DOI":"10.1162\/tacl_a_00344","volume":"8","author":"E Simpson","year":"2020","unstructured":"Simpson\u00a0E, Gao\u00a0Y, Gurevych\u00a0I (2020) Interactive Text Ranking with Bayesian Optimization: A Case Study on Community QA and Summarization. Transactions of the Association for Computational Linguistics 8:759\u2013775. https:\/\/doi.org\/10.1162\/tacl_a_00344","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"531_CR23","first-page":"30039","volume-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS \u201923","author":"Y Dubois","year":"2023","unstructured":"Dubois\u00a0Y, Li\u00a0X, Taori\u00a0R, Zhang\u00a0T, Gulrajani\u00a0I, Ba\u00a0J, Guestrin\u00a0C, Liang\u00a0P, Hashimoto\u00a0TB (2023) AlpacaFarm: a\u00a0simulation framework for methods that learn from human feedback. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS \u201923. Curran Associates Inc., Red Hook, NY, USA, pp\u00a030039\u201330069"},{"issue":"330","key":"531_CR24","doi-asserted-by":"publisher","first-page":"292","DOI":"10.2307\/3608567","volume":"39","author":"IJ Good","year":"1955","unstructured":"Good\u00a0IJ (1955) On the marking of chess-players. The Mathematical Gazette 39(330):292\u2013296","journal-title":"The Mathematical Gazette"},{"key":"531_CR25","series-title":"SIGIR \u201923","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591882","volume-title":"Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"M Wessel","year":"2023","unstructured":"Wessel\u00a0M, Horych\u00a0T, Ruas\u00a0T, Aizawa\u00a0A, Gipp\u00a0B, Spinde\u00a0T (2023) Introducing mbib - the first media bias identification benchmark task and dataset collection. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR \u201923. ACM, New York, NY, USA https:\/\/doi.org\/10.1145\/3539618.3591882"},{"key":"531_CR26","doi-asserted-by":"crossref","unstructured":"Spinde, T., Plank, M., Krieger, J.-D., Ruas, T., Gipp, B., Aizawa, A.: Neural media bias detection using distant supervision with babe - bias annotations by experts. Findings of the Association for Computational Linguistics: EMNLP 2021, 1166\u20131177 (2021)","DOI":"10.18653\/v1\/2021.findings-emnlp.101"},{"key":"531_CR27","doi-asserted-by":"publisher","first-page":"1921","DOI":"10.18653\/v1\/2021.eacl-main.165","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume","author":"PL Huguet Cabot","year":"2021","unstructured":"Huguet Cabot\u00a0PL, Abadi\u00a0D, Fischer\u00a0A, Shutova\u00a0E (2021) Us vs. Them: A Dataset of Populist Attitudes, News Bias and Emotions. In: Merlo\u00a0P, Tiedemann\u00a0J, Tsarfaty\u00a0R (eds) Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics, Online, pp\u00a01921\u20131945 https:\/\/doi.org\/10.18653\/v1\/2021.eacl-main.165"}],"container-title":["Datenbank-Spektrum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13222-026-00531-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13222-026-00531-1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13222-026-00531-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T10:15:24Z","timestamp":1783073724000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13222-026-00531-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,18]]},"references-count":27,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,3]]}},"alternative-id":["531"],"URL":"https:\/\/doi.org\/10.1007\/s13222-026-00531-1","relation":{},"ISSN":["1618-2162","1610-1995"],"issn-type":[{"value":"1618-2162","type":"print"},{"value":"1610-1995","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,18]]},"assertion":[{"value":"2 October 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 January 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 February 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 February 2026","order":5,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":6,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"In this article the statement in the Funding information section was incorrectly given as \u2018Open Access funding enabled and organized by Projekt DEAL.\u2019 and should have read \u2018Open Access funding enabled and organized by Projekt DEAL. This contribution has been developed in the project PLan_CV. Within the funding Programme FH-Personal, the project PLan_CV (reference number 03FHP109) is funded by the German Federal Ministry of Research, Technology and Space (BMFTR) and Joint Science Conference (GWK).\u2019.","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}}]}}