{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:17:43Z","timestamp":1750306663627,"version":"3.41.0"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2015,7,22]],"date-time":"2015-07-22T00:00:00Z","timestamp":1437523200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2015,7,27]]},"abstract":"<jats:p>\n            <jats:italic>Semiautomated Text Classification<\/jats:italic>\n            (SATC) may be defined as the task of ranking a set\n            <jats:italic>D<\/jats:italic>\n            of automatically labelled textual documents in such a way that, if a human annotator validates (i.e., inspects and corrects where appropriate) the documents in a top-ranked portion of\n            <jats:italic>D<\/jats:italic>\n            with the goal of increasing the overall labelling accuracy of\n            <jats:italic>D<\/jats:italic>\n            , the expected increase is maximized. An obvious SATC strategy is to rank\n            <jats:italic>D<\/jats:italic>\n            so that the documents that the classifier has labelled with the lowest confidence are top ranked. In this work, we show that this strategy is suboptimal. We develop new utility-theoretic ranking methods based on the notion of\n            <jats:italic>validation gain<\/jats:italic>\n            , defined as the improvement in classification effectiveness that would derive by validating a given automatically labelled document. We also propose a new effectiveness measure for SATC-oriented ranking methods, based on the expected reduction in classification error brought about by partially validating a list generated by a given ranking method. We report the results of experiments showing that, with respect to the baseline method mentioned earlier, and according to the proposed measure, our utility-theoretic ranking methods can achieve substantially higher expected reductions in classification error.\n          <\/jats:p>","DOI":"10.1145\/2742548","type":"journal-article","created":{"date-parts":[[2015,7,22]],"date-time":"2015-07-22T18:49:50Z","timestamp":1437590990000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Utility-Theoretic Ranking for Semiautomated Text Classification"],"prefix":"10.1145","volume":"10","author":[{"given":"Giacomo","family":"Berardi","sequence":"first","affiliation":[{"name":"Italian National Council of Research"}]},{"given":"Andrea","family":"Esuli","sequence":"additional","affiliation":[{"name":"Italian National Council of Research"}]},{"given":"Fabrizio","family":"Sebastiani","sequence":"additional","affiliation":[{"name":"Qatar Computing Research Institute, Doha, Qatar"}]}],"member":"320","published-online":{"date-parts":[[2015,7,22]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/133160.133169"},{"volume-title":"Foundations of Rational Choice Under Risk","author":"Anand Paul","key":"e_1_2_1_2_1","unstructured":"Paul Anand . 1993. Foundations of Rational Choice Under Risk . Oxford University Press , Oxford, UK . Paul Anand. 1993. Foundations of Rational Choice Under Risk. Oxford University Press, Oxford, UK."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348411"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.2501\/IJMR-2014-032"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935872"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/3013545.3013548"},{"key":"e_1_2_1_7_1","first-page":"24","article-title":"Smoothing sparse contingency tables","volume":"49","author":"Burman Prabir","year":"1987","unstructured":"Prabir Burman . 1987 . Smoothing sparse contingency tables . The Indian Journal of Statistics 49 , 1 (1987), 24 -- 36 . Prabir Burman. 1987. Smoothing sparse contingency tables. The Indian Journal of Statistics 49, 1 (1987), 24--36.","journal-title":"The Indian Journal of Statistics"},{"key":"e_1_2_1_8_1","doi-asserted-by":"crossref","unstructured":"Olivier Chapelle Bernard Sch\u00f6lkopf and Alexander Zien (Eds.). 2006. Semi-Supervised Learning. The MIT Press Cambridge US.  Olivier Chapelle Bernard Sch\u00f6lkopf and Alexander Zien (Eds.). 2006. Semi-Supervised Learning. The MIT Press Cambridge US.","DOI":"10.7551\/mitpress\/9780262033589.001.0001"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.3115\/981863.981904"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/1642194.1642224"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/11880561_1"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-00958-7_12"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2516889"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.3115\/1220355.1220480"},{"key":"e_1_2_1_15_1","volume-title":"Church","author":"Gale William A.","year":"1994","unstructured":"William A. Gale and Kenneth W . Church . 1994 . What\u2019s wrong with adding one? In Corpus-Based Research into Language : In Honour of Jan Aarts, N. Oostdijk and P. de Haan (Eds.). Rodopi, Amsterdam, NL , 189--200. William A. Gale and Kenneth W. Church. 1994. What\u2019s wrong with adding one? In Corpus-Based Research into Language: In Honour of Jan Aarts, N. Oostdijk and P. de Haan (Eds.). Rodopi, Amsterdam, NL, 189--200."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/1053072.1053091"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2008.239"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/188490.188557"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1135777.1135870"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 4th Annual Symposium on Document Analysis and Information Retrieval (SDAIR","author":"Ittner David J.","year":"1995","unstructured":"David J. Ittner , David D. Lewis , and David D. Ahn . 1995. Text categorization of low quality images . In Proceedings of the 4th Annual Symposium on Document Analysis and Information Retrieval (SDAIR 1995 ). Las Vegas, US, 301--315. David J. Ittner, David D. Lewis, and David D. Ahn. 1995. Text categorization of low quality images. In Proceedings of the 4th Annual Symposium on Document Analysis and Information Retrieval (SDAIR 1995). Las Vegas, US, 301--315."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 16th International Conference on Machine Learning (ICML","author":"Joachims Thorsten","year":"1999","unstructured":"Thorsten Joachims . 1999 . Transductive inference for text classification using support vector machines . In Proceedings of the 16th International Conference on Machine Learning (ICML 1999). Bled, SL, 200--209. Thorsten Joachims. 1999. Transductive inference for text classification using support vector machines. In Proceedings of the 16th International Conference on Machine Learning (ICML 1999). Bled, SL, 200--209."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 20th International Joint Conference on Artifical Intelligence (IJCAI","author":"Kapoor Ashish","year":"2007","unstructured":"Ashish Kapoor , Eric Horvitz , and Sumit Basu . 2007 . Selective supervision: Guiding supervised learning with decision-theoretic active learning . In Proceedings of the 20th International Joint Conference on Artifical Intelligence (IJCAI 2007). San Francisco, US, 877--882. Ashish Kapoor, Eric Horvitz, and Sumit Basu. 2007. Selective supervision: Guiding supervised learning with decision-theoretic active learning. In Proceedings of the 20th International Joint Conference on Artifical Intelligence (IJCAI 2007). San Francisco, US, 877--882."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/243199.243276"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of 11th International Conference on Machine Learning (ICML","author":"David","year":"1994","unstructured":"David D. Lewis and Jason Catlett. 1994. Heterogeneous uncertainty sampling for supervised learning . In Proceedings of 11th International Conference on Machine Learning (ICML 1994 ). New Brunswick, US, 148--156. David D. Lewis and Jason Catlett. 1994. Heterogeneous uncertainty sampling for supervised learning. In Proceedings of 11th International Conference on Machine Learning (ICML 1994). New Brunswick, US, 148--156."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/243199.243277"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40131-2_10"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-28997-2_43"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 15th International Conference on Machine Learning (ICML","author":"Andrew","year":"1998","unstructured":"Andrew K. McCallum and Kamal Nigam. 1998. Employing EM in pool-based active learning for text classification . In Proceedings of the 15th International Conference on Machine Learning (ICML 1998 ). Madison, US, 350--358. Andrew K. McCallum and Kamal Nigam. 1998. Employing EM in pool-based active learning for text classification. In Proceedings of the 15th International Conference on Machine Learning (ICML 1998). Madison, US, 350--358."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 21st Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI","author":"Niculescu-Mizil Alexandru","year":"2005","unstructured":"Alexandru Niculescu-Mizil and Rich Caruana . 2005 . Obtaining calibrated probabilities from boosting . In Proceedings of the 21st Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI 2005). Arlington, US, 413--420. Alexandru Niculescu-Mizil and Rich Caruana. 2005. Obtaining calibrated probabilities from boosting. In Proceedings of the 21st Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI 2005). Arlington, US, 413--420."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10506-010-9093-9"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000025"},{"volume-title":"Advances in Large Margin Classifiers, Alexander Smola, Peter Bartlett","author":"Platt John C.","key":"e_1_2_1_33_1","unstructured":"John C. Platt . 2000. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods . In Advances in Large Margin Classifiers, Alexander Smola, Peter Bartlett , Bernard Sch\u00f6lkopf , and Dale Schuurmans (Eds.). The MIT Press , Cambridge, MA, 61--74. John C. Platt. 2000. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In Advances in Large Margin Classifiers, Alexander Smola, Peter Bartlett, Bernard Sch\u00f6lkopf, and Dale Schuurmans (Eds.). The MIT Press, Cambridge, MA, 61--74."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/1248547.1248608"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390453"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007649029923"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505283"},{"volume-title":"Active learning","author":"Settles Burr","key":"e_1_2_1_38_1","unstructured":"Burr Settles . 2012. Active learning . Morgan & Claypool Publishers , San Rafael, US . Burr Settles. 2012. Active learning. Morgan & Claypool Publishers, San Rafael, US."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176346071"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1162\/153244302760185243"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206705"},{"volume-title":"Theory of games and economic behavior","author":"von Neumann John","key":"e_1_2_1_42_1","unstructured":"John von Neumann and Oskar Morgenstern . 1944. Theory of games and economic behavior . Princeton University Press , Princeton, US . John von Neumann and Oskar Morgenstern. 1944. Theory of games and economic behavior. Princeton University Press, Princeton, US."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/312624.312647"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/984321.984322"},{"key":"e_1_2_1_45_1","volume-title":"Goldberg","author":"Zhu Xiaojin","year":"2009","unstructured":"Xiaojin Zhu and Andrew B . Goldberg . 2009 . Introduction to Semi-supervised Learning. Morgan and Claypool, San Rafael, US. Xiaojin Zhu and Andrew B. Goldberg. 2009. Introduction to Semi-supervised Learning. Morgan and Claypool, San Rafael, US."}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2742548","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2742548","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:00:33Z","timestamp":1750230033000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2742548"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,7,22]]},"references-count":45,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,7,27]]}},"alternative-id":["10.1145\/2742548"],"URL":"https:\/\/doi.org\/10.1145\/2742548","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"type":"print","value":"1556-4681"},{"type":"electronic","value":"1556-472X"}],"subject":[],"published":{"date-parts":[[2015,7,22]]},"assertion":[{"value":"2014-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-07-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}