{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T13:10:20Z","timestamp":1765545020417,"version":"build-2065373602"},"reference-count":35,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,9,20]],"date-time":"2025-09-20T00:00:00Z","timestamp":1758326400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union\u2014NextGenerationEU","award":["CN_00000033"],"award-info":[{"award-number":["CN_00000033"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>In the technology-assisted review (TAR) area, most research has focused on ranking effectiveness and active learning strategies within individual topics, often assuming unconstrained review effort. However, real-world applications such as legal discovery or medical systematic reviews are frequently subject to global screening budgets. In this paper, we revisit the CLEF eHealth TAR shared tasks (2017\u20132019) through the lens of budget-aware evaluation. We first reproduce and verify the official participant results, organizing them into a unified dataset for comparative analysis. Then, we introduce and assess four intuitive budget allocation strategies\u2014even, proportional, inverse proportional, and threshold-capped greedy\u2014to explore how review effort can be efficiently distributed across topics. To evaluate systems under resource constraints, we propose two cost-aware metrics: relevant found per cost unit (RFCU) and utility gain at budget (UG@B). These complement traditional recall by explicitly modeling efficiency and trade-offs between true and false positives. Our results show that different allocation strategies optimize different metrics: even and inverse proportional allocation favor recall, while proportional and capped strategies better maximize RFCU. UG@B remains relatively stable across strategies, reflecting its balanced formulation. A correlation analysis reveals that RFCU and UG@B offer distinct perspectives from recall, with varying alignment across years. Together, these findings underscore the importance of aligning evaluation metrics and allocation strategies with screening goals. We release all data and code to support reproducibility and future research on cost-sensitive TAR.<\/jats:p>","DOI":"10.3390\/make7030104","type":"journal-article","created":{"date-parts":[[2025,9,22]],"date-time":"2025-09-22T08:37:39Z","timestamp":1758530259000},"page":"104","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Screening Smarter, Not Harder: Budget Allocation Strategies for Technology-Assisted Reviews (TARs) in Empirical Medicine"],"prefix":"10.3390","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9709-6392","authenticated-orcid":false,"given":"Giorgio Maria","family":"Di Nunzio","sequence":"first","affiliation":[{"name":"Department of Information Engineering, University of Padova, 35131 Padova, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2025,9,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1034\/j.1600-051X.29.s3.15.x","article-title":"A guide to systematic reviews","volume":"29","author":"Needleman","year":"2002","journal-title":"J. Clin. Periodontol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1007\/s10796-015-9589-7","article-title":"Advanced analytics for the automation of medical systematic reviews","volume":"18","author":"Timsina","year":"2016","journal-title":"Inf. Syst. Front."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Di Nunzio, G.M., Kanoulas, E., and Majumder, P. (2024, January 28). Technology Assisted Review Systems: Current and Future Directions. Proceedings of the 3rd Workshop on Augmented Intelligence for Technology-Assisted Reviews Systems (ALTARS 2024), Glasgow, UK. CEUR Workshop Proceedings.","DOI":"10.1007\/978-3-031-56069-9_59"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Cormack, G.V., and Grossman, M.R. (2016, January 17\u201321). Engineering Quality and Reliability in Technology-Assisted Review. Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, New York, NY, USA. SIGIR \u201916.","DOI":"10.1145\/2911451.2911510"},{"key":"ref_5","unstructured":"Manning, C., and Schutze, H. (1999). Foundations of Statistical Natural Language Processing, MIT Press."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1186\/s13643-016-0259-8","article-title":"Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: Overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews","volume":"5","author":"Laursen","year":"2016","journal-title":"Syst. Rev."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Koffel, J.B. (2015). Use of Recommended Search Strategies in Systematic Reviews and the Impact of Librarian Involvement: A Cross-Sectional Survey of Recent Authors. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0125931"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.jclinepi.2017.08.010","article-title":"Living systematic review: 1. Introduction\u2014The why, what, when, and how","volume":"91","author":"Elliott","year":"2017","journal-title":"J. Clin. Epidemiol."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Yang, E., Lewis, D.D., and Frieder, O. (2021, January 24\u201327). On minimizing cost in legal document review workflows. Proceedings of the 21st ACM Symposium on Document Engineering, New York, NY, USA. DocEng \u201921.","DOI":"10.1145\/3469096.3469872"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Crestani, F., Braschler, M., Savoy, J., Rauber, A., M\u00fcller, H., Losada, D.E., Heinatz B\u00fcrki, G., Cappellato, L., and Ferro, N. (2019, January 9\u201312). Overview of the CLEF eHealth Evaluation Lab 2019. Proceedings of the Experimental IR Meets Multilinguality, Multimodality, and Interaction, Lugano, Switzerland.","DOI":"10.1007\/978-3-030-28577-7"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bellot, P., Trabelsi, C., Mothe, J., Murtagh, F., Nie, J.Y., Soulier, L., SanJuan, E., Cappellato, L., and Ferro, N. (2018, January 10\u201314). Overview of the CLEF eHealth Evaluation Lab 2018. Proceedings of the Experimental IR Meets Multilinguality, Multimodality, and Interaction, Avignon, France.","DOI":"10.1007\/978-3-319-98932-7"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Jones, G.J., Lawless, S., Gonzalo, J., Kelly, L., Goeuriot, L., Mandl, T., Cappellato, L., and Ferro, N. (2017, January 11\u201314). CLEF 2017 eHealth Evaluation Lab Overview. Proceedings of the Experimental IR Meets Multilinguality, Multimodality, and Interaction, Dublin, Ireland.","DOI":"10.1007\/978-3-319-65813-1"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Voorhees, E.M. (2018, January 22\u201326). On Building Fair and Reusable Test Collections using Bandit Techniques. Proceedings of the 27th ACM International Conference on Information and Knowledge Management, New York, NY, USA. CIKM \u201918.","DOI":"10.1145\/3269206.3271766"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Rahman, M.M., Kutlu, M., and Lease, M. (2019, January 13\u201317). Constructing Test Collections using Multi-armed Bandits and Active Learning. Proceedings of the The World Wide Web Conference, New York, NY, USA. WWW \u201919.","DOI":"10.1145\/3308558.3313675"},{"key":"ref_15","first-page":"12103","article-title":"Flexible budgets in restless bandits: A primal-dual algorithm for efficient budget allocation","volume":"Volume 37","author":"Killian","year":"2023","journal-title":"Proceedings of the Thirty\u2013Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"100559","DOI":"10.1109\/ACCESS.2024.3429359","article-title":"A Multi-Channel Advertising Budget Allocation Using Reinforcement Learning and an Improved Differential Evolution Algorithm","volume":"12","author":"Li","year":"2024","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1007\/s10115-023-01973-9","article-title":"Multilabel classification using crowdsourcing under budget constraints","volume":"66","author":"Suyal","year":"2024","journal-title":"Knowl. Inf. Syst."},{"key":"ref_18","unstructured":"Hadfi, R., Anthony, P., Sharma, A., Ito, T., and Bai, Q. (2024, January 19). Contextual Bandit with Herding Effects: Algorithms and Recommendation Applications. Proceedings of the PRICAI 2024: Trends in Artificial Intelligence, Kyoto, Japan."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Stradiotti, L., Perini, L., and Davis, J. (2024). Combining Active Learning and Learning to Reject for Anomaly Detection. ECAI 2024, IOS Press.","DOI":"10.3233\/FAIA240749"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1141","DOI":"10.1109\/TSMCA.2011.2172418","article-title":"A Budget Optimization Framework for Search Advertisements Across Markets","volume":"42","author":"Yang","year":"2012","journal-title":"IEEE Trans. Syst. Man-Cybern.-Part Syst. Humans"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Tricco, A.C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., Perrier, L., Hutton, B., Moher, D., and Straus, S.E. (2015). A scoping review of rapid review methods. BMC Med., 13.","DOI":"10.1186\/s12916-015-0465-6"},{"key":"ref_22","unstructured":"Li, S. (2024, January 23\u201325). Comparison of Tools and Methods for Technology-Assisted Review. Proceedings of the Information Management, Suva, Fiji."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Cormack, G.V., and Grossman, M.R. (2014, January 6\u201311). Evaluation of machine-learning protocols for technology-assisted review in electronic discovery. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, New York, NY, USA. SIGIR \u201914.","DOI":"10.1145\/2600428.2609601"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"273","DOI":"10.2217\/cer.13.17","article-title":"Modernizing the systematic review process to inform comparative effectiveness: Tools and methods","volume":"2","author":"Wallace","year":"2013","journal-title":"J. Comp. Eff. Res."},{"key":"ref_25","first-page":"200193","article-title":"An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews","volume":"18","author":"Kusa","year":"2023","journal-title":"Intell. Syst. Appl."},{"key":"ref_26","first-page":"200209","article-title":"Improved risk minimization algorithms for technology-assisted review","volume":"18","author":"Molinari","year":"2023","journal-title":"Intell. Syst. Appl."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Arampatzis, A., Kanoulas, E., Tsikrika, T., Vrochidis, S., Joho, H., Lioma, C., Eickhoff, C., N\u00e9v\u00e9ol, A., Cappellato, L., and Ferro, N. (2020, January 22\u201325). A Study on a Stopping Strategy for Systematic Reviews Based on a Distributed Effort Approach. Proceedings of the Experimental IR Meets Multilinguality, Multimodality, and Interaction, Thessaloniki, Greece.","DOI":"10.1007\/978-3-030-58219-7"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Lewis, D.D., Yang, E., and Frieder, O. (2021, January 1\u20135). Certifying One-Phase Technology-Assisted Reviews. Proceedings of the 30th ACM International Conference on Information & Knowledge Management, New York, NY, USA. CIKM \u201921.","DOI":"10.1145\/3459637.3482415"},{"key":"ref_29","unstructured":"Cappellato, L., Ferro, N., Goeuriot, L., and Mandl, T. (2017, January 11\u201314). CLEF eHealth 2017 Multilingual Information Extraction task Overview: ICD10 Coding of Death Certificates in English and French. Proceedings of the Working Notes of CLEF 2017\u2014Conference and Labs of the Evaluation Forum, Dublin, Ireland. CEUR Workshop Proceedings."},{"key":"ref_30","unstructured":"Cappellato, L., Ferro, N., Nie, J.Y., and Soulier, L. (2018, January 10\u201314). CLEF 2018 Technologically Assisted Reviews in Empirical Medicine Overview. Proceedings of the Working Notes of CLEF 2018\u2014Conference and Labs of the Evaluation Forum, Avignon, France. CEUR Workshop Proceedings."},{"key":"ref_31","unstructured":"Cappellato, L., Ferro, N., Losada, D.E., and M\u00fcller, H. (2019, January 9\u201312). CLEF 2019 Technology Assisted Reviews in Empirical Medicine Overview. Proceedings of the Working Notes of CLEF 2019\u2014Conference and Labs of the Evaluation Forum, Lugano, Switzerland. CEUR Workshop Proceedings."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Tran-Thanh, L., Chapman, A., de Cote, E.M., Rogers, A., and Jennings, N.R. (2010, January 11\u201315). Epsilon\u2013First Policies for Budget\u2013Limited Multi-Armed Bandits. Proceedings of the AAAI Conference on Artificial Intelligence, Atlanta, GA, USA.","DOI":"10.1609\/aaai.v24i1.7758"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1023\/A:1013689704352","article-title":"Finite-Time Analysis of the Multiarmed Bandit Problem","volume":"47","author":"Auer","year":"2002","journal-title":"Mach. Learn."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Li, L., Chu, W., Langford, J., and Schapire, R.E. (2010, January 26\u201330). A Contextual-Bandit Approach to Personalized News Article Recommendation. Proceedings of the 19th International Conference on World Wide Web, New York, NY, USA. WWW \u201910.","DOI":"10.1145\/1772690.1772758"},{"key":"ref_35","unstructured":"Di Nunzio, G. (2025, January 10\u201314). POLAR: Policy Optimization for Literature Analysis under Review Constraints. Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM 2025, Seoul, Republic of Korea."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/7\/3\/104\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,6]],"date-time":"2025-11-06T10:01:53Z","timestamp":1762423313000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/7\/3\/104"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,20]]},"references-count":35,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["make7030104"],"URL":"https:\/\/doi.org\/10.3390\/make7030104","relation":{},"ISSN":["2504-4990"],"issn-type":[{"type":"electronic","value":"2504-4990"}],"subject":[],"published":{"date-parts":[[2025,9,20]]}}}