{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T07:37:41Z","timestamp":1776411461054,"version":"3.51.2"},"publisher-location":"New York, NY, USA","reference-count":58,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,12,8]],"date-time":"2024-12-08T00:00:00Z","timestamp":1733616000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-sa\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,12,8]]},"DOI":"10.1145\/3673791.3698432","type":"proceedings-article","created":{"date-parts":[[2024,12,8]],"date-time":"2024-12-08T06:24:16Z","timestamp":1733639056000},"page":"186-196","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["A Reproducibility and Generalizability Study of Large Language Models for Query Generation"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5164-2690","authenticated-orcid":false,"given":"Moritz","family":"Staudinger","sequence":"first","affiliation":[{"name":"TU Wien, Vienna, Austria"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4420-4147","authenticated-orcid":false,"given":"Wojciech","family":"Kusa","sequence":"additional","affiliation":[{"name":"TU Wien & Allegro ML Research, Vienna, Austria"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7584-6439","authenticated-orcid":false,"given":"Florina","family":"Piroi","sequence":"additional","affiliation":[{"name":"TU Wien, Vienna, Austria"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3643-6493","authenticated-orcid":false,"given":"Aldo","family":"Lipani","sequence":"additional","affiliation":[{"name":"University College London, London, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7149-5843","authenticated-orcid":false,"given":"Allan","family":"Hanbury","sequence":"additional","affiliation":[{"name":"TU Wien, Vienna, Austria"}]}],"member":"320","published-online":{"date-parts":[[2024,12,8]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1097\/GOX.0000000000005339"},{"key":"e_1_3_2_1_2_1","unstructured":"Amal Alharbi and Mark Stevenson. 2017. Ranking Abstracts to Identify Relevant Evidence for Systematic Reviews: The University of Sheffield's Approach to CLEF eHealth 2017 Task 2. In CLEF (Working Notes)."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaa148"},{"key":"e_1_3_2_1_4_1","unstructured":"Antonios Anagnostou Athanasios Lagopoulos Grigorios Tsoumakas and Ioannis P Vlahavas. 2017. Combining Inter-Review Learning-to-Rank and Intra-Review Incremental Training for Title and Abstract Screening in Systematic Reviews. In CLEF (Working Notes)."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.eacl-long.5"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012545"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-020-03648-6"},{"key":"e_1_3_2_1_8_1","unstructured":"Jiayi Chen Su Chen Yang Song Hongyu Liu Yueyao Wang Qinmin Hu Liang He and Yan Yang. 2017. ECNU at 2017 eHealth Task 2: Technologically Assisted Reviews in Empirical Medicine. In CLEF (Working Notes)."},{"key":"e_1_3_2_1_9_1","volume-title":"Methods of clinical epidemiology","author":"Clark Justin","unstructured":"Justin Clark. 2013. Systematic reviewing: Introduction, locating studies and data abstraction. In Methods of clinical epidemiology. Springer, 187--211."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1197\/jamia.M1929"},{"key":"e_1_3_2_1_11_1","unstructured":"Giorgio Maria Di Nunzio Federica Beghini Federica Vezzani and Genevi\u00e8ve Henrot. 2017. An Interactive Two-Dimensional Approach to Query Aspects Rewriting in Systematic Reviews. In IMS Unipd At CLEF eHealth Task 2."},{"key":"e_1_3_2_1_12_1","volume-title":"Precise zero-shot dense retrieval without relevance labels. arXiv preprint arXiv:2212.10496","author":"Gao Luyu","year":"2022","unstructured":"Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without relevance labels. arXiv preprint arXiv:2212.10496 (2022)."},{"key":"e_1_3_2_1_13_1","volume-title":"ChatGPT is not all you need. A State of the Art Review of large Generative AI models. arXiv preprint arXiv:2301.04655","author":"Gozalo-Brizuela Roberto","year":"2023","unstructured":"Roberto Gozalo-Brizuela and Eduardo C Garrido-Merchan. 2023. ChatGPT is not all you need. A State of the Art Review of large Generative AI models. arXiv preprint arXiv:2301.04655 (2023)."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12967-023-04371-5"},{"key":"e_1_3_2_1_15_1","volume-title":"How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597","author":"Guo Biyang","year":"2023","unstructured":"Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597 (2023)."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2014.09.016"},{"key":"e_1_3_2_1_17_1","volume-title":"Cochrane handbook for systematic reviews of interventions","author":"Higgins Julian PT","unstructured":"Julian PT Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J Page, and Vivian A Welch. 2019. Cochrane handbook for systematic reviews of interventions. John Wiley & Sons."},{"key":"e_1_3_2_1_18_1","unstructured":"Noah Hollmann and Carsten Eickhoff. 2017. Ranking and Feedback-based Stopping for Recall-Centric Document Retrieval. In CLEF (Working Notes)."},{"key":"e_1_3_2_1_19_1","volume-title":"Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.","author":"Jiang Albert Q","year":"2023","unstructured":"Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)."},{"key":"e_1_3_2_1_20_1","volume-title":"Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.","author":"Jiang Albert Q","year":"2024","unstructured":"Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)."},{"key":"e_1_3_2_1_21_1","volume-title":"Active retrieval augmented generation. arXiv preprint arXiv:2305.06983","author":"Jiang Zhengbao","year":"2023","unstructured":"Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983 (2023)."},{"key":"e_1_3_2_1_22_1","volume-title":"Rice Stephen, Rithalia Amber, Stewart Lesley, Stock Christian, Wilson Paul, and Woolacott Nerys.","author":"Jo Akers","year":"2009","unstructured":"Akers Jo, Aguiar-Ib\u00e1\u00f1ez Raquel, Burch Jane, Chambers Duncan, Eastwood Alison, Fayter Debra, Hempel Susanne, Light Kate, Rice Stephen, Rithalia Amber, Stewart Lesley, Stock Christian, Wilson Paul, and Woolacott Nerys. 2009. Systematic Reviews: CRD's guidance for undertaking reviews in health care. CRD, University of York, York. www.york.ac.uk\/inst\/crd"},{"key":"e_1_3_2_1_23_1","volume-title":"CLEF 2017 Technologically Assisted Reviews in Empirical Medicine Overview. In CLEF'17.","author":"Kanoulas Evangelos","year":"2017","unstructured":"Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2017. CLEF 2017 Technologically Assisted Reviews in Empirical Medicine Overview. In CLEF'17."},{"key":"e_1_3_2_1_24_1","volume-title":"CLEF 2018 technologically assisted reviews in empirical medicine overview. CEURWorkshop Proceedings 2125 (7 2018","author":"Kanoulas Evangelos","year":"2018","unstructured":"Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2018. CLEF 2018 technologically assisted reviews in empirical medicine overview. CEURWorkshop Proceedings 2125 (7 2018). https:\/\/pureportal.strath.ac.uk\/en\/publications\/clef-2018-technologically-assisted-reviews-in-empirical-medicine-"},{"key":"e_1_3_2_1_25_1","volume-title":"CLEF 2018 Technology Assisted Reviews in Empirical Medicine Overview. In CLEF 2018 Evaluation Labs and Workshop: Online Working Notes (CEUR-WS).","author":"Kanoulas Evangelos","year":"2018","unstructured":"Evangelos Kanoulas, Rene Spijker, Dan Li, and Leif Azzopardi. 2018. CLEF 2018 Technology Assisted Reviews in Empirical Medicine Overview. In CLEF 2018 Evaluation Labs and Workshop: Online Working Notes (CEUR-WS)."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1651318.1651338"},{"key":"e_1_3_2_1_27_1","volume-title":"Advances in Information Retrieval, Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil N\u00f8rv\u00e5g","author":"Kusa Wojciech","unstructured":"Wojciech Kusa, Allan Hanbury, and Petr Knoth. 2022. Automation of Citation Screening for Systematic Literature Reviews Using Neural Networks: A Replicability Study. In Advances in Information Retrieval, Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil N\u00f8rv\u00e5g, and Vinay Setty (Eds.). Springer International Publishing, Cham, 584--598. https:\/\/arxiv.org\/abs\/2201.07534v1"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3583780.3614736"},{"key":"e_1_3_2_1_29_1","volume-title":"CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews. arXiv preprint arXiv:2311.12474","author":"Kusa Wojciech","year":"2023","unstructured":"Wojciech Kusa, Oscar E Mendoza, Matthias Samwald, Petr Knoth, and Allan Hanbury. 2023. CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews. arXiv preprint arXiv:2311.12474 (2023)."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3578337.3605135"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3209994"},{"key":"e_1_3_2_1_32_1","unstructured":"Adamantios Minas Athanasios Lagopoulos and Grigorios Tsoumakas. 2018. Aristotle University's Approach to the Technologically Assisted Reviews in Empirical Medicine Task of the 2018 CLEF eHealth Lab. In CLEF (Working Notes)."},{"key":"e_1_3_2_1_33_1","unstructured":"Christopher Norman Mariska Leeflang and Aur\u00e9lie N\u00e9v\u00e9ol. 2018. LIMSI@ CLEF eHealth 2018 Task 2: Technology Assisted Reviews by Stacking Active and Static Learning. In CLEF (Working Notes)."},{"key":"e_1_3_2_1_34_1","volume-title":"Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots. arXiv preprint arXiv:2302.06466","author":"Omar Reham","year":"2023","unstructured":"Reham Omar, Omij Mangukiya, Panos Kalnis, and Essam Mansour. 2023. Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots. arXiv preprint arXiv:2302.06466 (2023)."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1186\/2046-4053-4-5"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1410"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare11060887"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2006.01.007"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3269215"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-45439-5_27"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-020-09381-1"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080707"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-147-4-200708210-00179"},{"key":"e_1_3_2_1_44_1","volume-title":"Identifying nurse staffing research in Medline: development and testing of empirically derived search strategies with the PubMed interface. BMC medical research methodology 10, 1","author":"Simon Michael","year":"2010","unstructured":"Michael Simon, Elke Hausner, Susan F Klaus, and Nancy E Dunton. 2010. Identifying nurse staffing research in Medline: development and testing of empirically derived search strategies with the PubMed interface. BMC medical research methodology 10, 1 (2010), 1--8."},{"key":"e_1_3_2_1_45_1","unstructured":"Jaspreet Singh and Lini Thomas. 2017. IIIT-H at CLEF eHealth 2017 Task 2: Technologically Assisted Reviews in Empirical Medicine. In CLEF (Working Notes)."},{"key":"e_1_3_2_1_46_1","volume-title":"Evaluation of ChatGPT as a question answering system for answering complex questions. arXiv preprint arXiv:2303.07992","author":"Tan Yiming","year":"2023","unstructured":"Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Evaluation of ChatGPT as a question answering system for answering complex questions. arXiv preprint arXiv:2303.07992 (2023)."},{"key":"e_1_3_2_1_47_1","volume-title":"Hashimoto","author":"Taori Rohan","year":"2023","unstructured":"Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https:\/\/github.com\/tatsu-lab\/stanford_alpaca."},{"key":"e_1_3_2_1_48_1","unstructured":"Gemini Team Rohan Anil Sebastian Borgeaud Yonghui Wu Jean-Baptiste Alayrac Jiahui Yu Radu Soricut Johan Schalkwyk Andrew M Dai Anja Hauth et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)."},{"key":"e_1_3_2_1_49_1","volume-title":"Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1371\/JOURNAL.PONE.0003684"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","unstructured":"Guy Tsafnat Paul Glasziou Miew K. Choong Adam Dunn Filippo Galgani and Enrico Coiera. 2014. Systematic review automation technologies. 74 pages. https:\/\/doi.org\/10.1186\/2046-4053-3-74","DOI":"10.1186\/2046-4053-3-74"},{"key":"e_1_3_2_1_52_1","volume-title":"Query2doc: Query Expansion with Large Language Models. arXiv preprint arXiv:2303.07678","author":"Wang Liang","year":"2023","unstructured":"Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query Expansion with Large Language Models. arXiv preprint arXiv:2303.07678 (2023)."},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531748"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591703"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3572960.3572980"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99736-6_34"},{"key":"e_1_3_2_1_57_1","volume-title":"Retrieval-augmented multimodal language modeling. arXiv preprint arXiv:2302.13971","author":"Yasunaga Michihiro","year":"2023","unstructured":"Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Retrieval-augmented multimodal language modeling. arXiv preprint arXiv:2302.13971 (2023)."},{"key":"e_1_3_2_1_58_1","volume-title":"Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews. arXiv preprint arXiv:2305.11828","author":"Yun Hye Sun","year":"2023","unstructured":"Hye Sun Yun, Iain J Marshall, Thomas Trikalinos, and Byron C Wallace. 2023. Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews. arXiv preprint arXiv:2305.11828 (2023)."}],"event":{"name":"SIGIR-AP 2024: Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region","location":"Tokyo Japan","acronym":"SIGIR-AP 2024","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3673791.3698432","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3673791.3698432","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T16:22:41Z","timestamp":1755879761000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3673791.3698432"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,8]]},"references-count":58,"alternative-id":["10.1145\/3673791.3698432","10.1145\/3673791"],"URL":"https:\/\/doi.org\/10.1145\/3673791.3698432","relation":{},"subject":[],"published":{"date-parts":[[2024,12,8]]},"assertion":[{"value":"2024-12-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}