{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,8]],"date-time":"2026-06-08T22:32:54Z","timestamp":1780957974707,"version":"3.54.1"},"reference-count":71,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,8,10]],"date-time":"2023-08-10T00:00:00Z","timestamp":1691625600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000780","name":"European Commission","doi-asserted-by":"crossref","award":["871042, 951911"],"award-info":[{"award-number":["871042, 951911"]}],"id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Italian Ministry of University"},{"name":"FPI 2017 predoctoral programme, from the Spanish Ministry of Economy and Competitiveness","award":["BES-2017-081202"],"award-info":[{"award-number":["BES-2017-081202"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2024,1,31]]},"abstract":"<jats:p>\n            Quantification, variously called\n            <jats:italic>supervised prevalence estimation<\/jats:italic>\n            or\n            <jats:italic>learning to quantify<\/jats:italic>\n            , is the supervised learning task of generating predictors of the relative frequencies (a.k.a.\n            <jats:italic>prevalence values<\/jats:italic>\n            ) of the classes of interest in unlabelled data samples. While many quantification methods have been proposed in the past for binary problems and, to a lesser extent, single-label multiclass problems, the multi-label setting (i.e., the scenario in which the classes of interest are not mutually exclusive) remains by and large unexplored. A straightforward solution to the multi-label quantification problem could simply consist of recasting the problem as a set of independent binary quantification problems. Such a solution is simple but na\u00efve, since the independence assumption upon which it rests is, in most cases, not satisfied. In these cases, knowing the relative frequency of one class could be of help in determining the prevalence of other related classes. We propose the first truly multi-label quantification methods, i.e., methods for inferring estimators of class prevalence values that strive to leverage the stochastic dependencies among the classes of interest in order to predict their relative frequencies more accurately. We show empirical evidence that natively multi-label solutions outperform the na\u00efve approaches by a large margin. The code to reproduce all our experiments is available online.\n          <\/jats:p>","DOI":"10.1145\/3606264","type":"journal-article","created":{"date-parts":[[2023,7,4]],"date-time":"2023-07-04T13:21:19Z","timestamp":1688476879000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Multi-Label Quantification"],"prefix":"10.1145","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0377-1025","authenticated-orcid":false,"given":"Alejandro","family":"Moreo","sequence":"first","affiliation":[{"name":"Istituto di Scienza e Tecnologie dell\u2019Informazione, Consiglio Nazionale delle Ricerche, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9748-2269","authenticated-orcid":false,"given":"Manuel","family":"Francisco","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Artificial Intelligence, University of Granada, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4221-6427","authenticated-orcid":false,"given":"Fabrizio","family":"Sebastiani","sequence":"additional","affiliation":[{"name":"Istituto di Scienza e Tecnologie dell\u2019Informazione, Consiglio Nazionale delle Ricerche, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,8,10]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018741"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2010.75"},{"key":"e_1_3_3_4_2","volume-title":"Machine Learning for Acquiring Knowledge in Astro-particle Physics","author":"Bunse Mirko","year":"2022","unstructured":"Mirko Bunse. 2022. Machine Learning for Acquiring Knowledge in Astro-particle Physics. Ph. D. Dissertation. University of Dortmund, Dortmund, DE."},{"key":"e_1_3_3_5_2","first-page":"43","volume-title":"Proceedings of the 2nd International Workshop on Learning to Quantify (LQ 2022)","author":"Bunse Mirko","year":"2022","unstructured":"Mirko Bunse. 2022. On multi-class extensions of adjusted classify and count. In Proceedings of the 2nd International Workshop on Learning to Quantify (LQ 2022). Grenoble, IT, 43\u201350."},{"key":"e_1_3_3_6_2","volume-title":"Proceedings of the 33rd European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML\/PKDD 2022)","author":"Bunse Mirko","year":"2022","unstructured":"Mirko Bunse, Alejandro Moreo, Fabrizio Sebastiani, and Martin Senz. 2022. Ordinal quantification through regularization. In Proceedings of the 33rd European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML\/PKDD 2022). Grenoble, FR."},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1148"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2022.3179355"},{"key":"e_1_3_3_9_2","unstructured":"Alberto Casta\u00f1o Laura Mor\u00e1n-Fern\u00e1ndez Jaime Alonso Ver\u00f3nica Bol\u00f3n-Canedo Amparo Alonso-Betanzos and Juan Jos\u00e9 del Coz. 2021. A theoretical analysis of quantification methods based on matching distributions. Retrieved from https:\/\/github.com\/bertocast\/adjust_dist_xy. Accessed September 1 2022."},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.3390\/app12031470"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914749"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.5555\/3104322.3104359"},{"key":"e_1_3_3_13_2","first-page":"681","volume-title":"Proceedings of the 15th Annual Conference on Neural Information Processing Systems (NIPS\u201901)","author":"Elisseeff Andr\u00e9","year":"2001","unstructured":"Andr\u00e9 Elisseeff and Jason Weston. 2001. A kernel method for multi-labelled classification. In Proceedings of the 15th Annual Conference on Neural Information Processing Systems (NIPS\u201901). Vancouver, CA, 681\u2013687."},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20467-8"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3269287"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99739-7_47"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/2700406"},{"key":"e_1_3_3_18_2","first-page":"79:1\u201379:33","article-title":"Quantification under prior probability shift: The ratio estimator and its extensions","volume":"20","author":"Vaz Afonso Fernandes","year":"2019","unstructured":"Afonso Fernandes Vaz, Rafael Izbicki, and Rafael Bassi Stern. 2019. Quantification under prior probability shift: The ratio estimator and its extensions. Journal of Machine Learning Research 20 (2019), 79:1\u201379:33.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_3_19_2","unstructured":"Aykut Firat. 2016. Unified framework for quantification. (2016). arXiv:1606.00868v1. Retrieved from https:\/\/arxiv.org\/abs\/1606.00868v1."},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/11564096_55"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0097-y"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015361"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/s13278-016-0327-z"},{"issue":"5","key":"e_1_3_3_24_2","first-page":"74:1\u201374:40","article-title":"A review on quantification learning","volume":"50","author":"Gonz\u00e1lez Pablo","year":"2017","unstructured":"Pablo Gonz\u00e1lez, Alberto Casta\u00f1o, Nitesh V. Chawla, and Juan Jos\u00e9 del Coz. 2017. A review on quantification learning. ACM Computing Surveys 50, 5 (2017), 74:1\u201374:40.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2012.05.028"},{"key":"e_1_3_3_26_2","volume-title":"Proceedings of the CIKM 2021 Workshop on Learning to Quantify","author":"Hassan Waqar","year":"2021","unstructured":"Waqar Hassan, Andr\u00e9 Gustavo Maletzke, and Gustavo Batista. 2021. Pitfalls in quantification assessment. In Proceedings of the CIKM 2021 Workshop on Learning to Quantify. Virtual Event."},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA49011.2020.00012"},{"key":"e_1_3_3_28_2","volume-title":"Multilabel Classification: Problem Analysis, Metrics and Techniques","author":"Herrera Francisco","year":"2016","unstructured":"Francisco Herrera, Francisco Charte, Antonio J. Rivera, and Mar\u00eda J. Del Jesus. 2016. Multilabel Classification: Problem Analysis, Metrics and Techniques. Springer, Cham, CH."},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-2099-5_20"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1540-5907.2009.00428.x"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-017-5659-z"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cmpb.2022.106638"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1214\/07-sts247"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3121050.3121083"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s13748-012-0030-x"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2012.03.004"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33014552"},{"key":"e_1_3_3_38_2","volume-title":"Proceedings of the AAAI 1999 Workshop on Text Learning","author":"McCallum Andrew K.","year":"1999","unstructured":"Andrew K. McCallum. 1999. Multi-label text classification with a mixture model trained by EM. In Proceedings of the AAAI 1999 Workshop on Text Learning. Orlando."},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2013.09.029"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220059"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2011.06.019"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482015"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72240-1_6"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0263449"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00202-021-01278-6"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623651"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2018.01.001"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2016.07.001"},{"key":"e_1_3_3_49_2","volume-title":"Scalable Multi-label Classification","author":"Read Jesse","year":"2010","unstructured":"Jesse Read. 2010. Scalable Multi-label Classification. Ph. D. Dissertation. University of Waikato, Hamilton, NZ."},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-011-5256-5"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1162\/089976602753284446"},{"key":"e_1_3_3_52_2","volume-title":"Proceedings of the CIKM 2021 Workshop on Learning to Quantify","author":"Sakai Tetsuya","year":"2021","unstructured":"Tetsuya Sakai. 2021. A closer look at evaluation measures for ordinal quantification. In Proceedings of the CIKM 2021 Workshop on Learning to Quantify. Virtual Event."},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007649029923"},{"key":"e_1_3_3_54_2","unstructured":"Tobias Schumacher Markus Strohmaier and Florian Lemmerich. 2021. A comparative evaluation of quantification methods. arXiv:2103.03223. Retrieved from https:\/\/arxiv.org\/abs\/2103.03223."},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-019-09363-y"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-23808-6_10"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.entcs.2013.02.010"},{"key":"e_1_3_3_58_2","first-page":"3","volume-title":"Proceedings of the Dataset Shift in Machine Learning","author":"Storkey Amos","year":"2009","unstructured":"Amos Storkey. 2009. When training and test sets are different: Characterizing learning transfer. In Proceedings of the Dataset Shift in Machine Learning. Joaquin Qui\u00f1onero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence (Eds.), The MIT Press, Cambridge, 3\u201328."},{"key":"e_1_3_3_59_2","unstructured":"Piotr Szymanski and Tomasz Kajdanowicz. 2019. Scikit-multilearn: A Python library for multi-Label classification. Journal of Machine Learning Research 20 (2019) 6:1\u20136:22."},{"key":"e_1_3_3_60_2","first-page":"22","volume-title":"Proceedings of the 1st International Workshop on Learning with Imbalanced Domains: Theory and Applications (LIDTA 2017)","author":"Szyma\u0144ski Piotr","year":"2017","unstructured":"Piotr Szyma\u0144ski and Tomasz Kajdanowicz. 2017. A network perspective on stratification of multi-label data. In Proceedings of the 1st International Workshop on Learning with Imbalanced Domains: Theory and Applications (LIDTA 2017). Skopje, MK, 22\u201335."},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.3390\/e18080282"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1002\/ese3.1058"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.4018\/jdwm.2007070101"},{"key":"e_1_3_3_64_2","first-page":"53","volume-title":"Proceedings of the ECML\/PKDD 2008 Workshop on Mining Multidimensional Data (MMD 2008)","author":"Tsoumakas Grigorios","year":"2008","unstructured":"Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2008. Effective and efficient multilabel classification in domains with large number of labels. In Proceedings of the ECML\/PKDD 2008 Workshop on Mining Multidimensional Data (MMD 2008). Antwerp, BE, 53\u201359."},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2010.164"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-008-5077-3"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2019.2935143"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1016\/s0893-6080(05)80023-1"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775151"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2006.12.019"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2013.39"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2021\/463"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3606264","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3606264","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:47:07Z","timestamp":1750178827000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3606264"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,10]]},"references-count":71,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1,31]]}},"alternative-id":["10.1145\/3606264"],"URL":"https:\/\/doi.org\/10.1145\/3606264","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,10]]},"assertion":[{"value":"2022-11-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-06-19","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}