{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T07:19:49Z","timestamp":1740122389274,"version":"3.37.3"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2024,8,2]],"date-time":"2024-08-02T00:00:00Z","timestamp":1722556800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,2]],"date-time":"2024-08-02T00:00:00Z","timestamp":1722556800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["R018634\/1"],"award-info":[{"award-number":["R018634\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100018693","name":"HORIZON EUROPE Framework Programme","doi-asserted-by":"publisher","award":["101104278"],"award-info":[{"award-number":["101104278"]}],"id":[{"id":"10.13039\/100018693","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Min Knowl Disc"],"published-print":{"date-parts":[[2024,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We study the problem of predicting the next query to be recommended in interactive data exploratory analysis to guide users to correct content. Current query prediction approaches are based on sequence-to-sequence learning, exploiting past interaction data. However, due to the resource-hungry training process, such approaches fail to adapt to immediate user feedback. Immediate feedback is essential and considered as a signal of the user\u2019s intent. We contribute with a novel query prediction ensemble mechanism, which adapts to immediate feedback relying on multi-armed bandits framework. Our mechanism, an extension to the popular Exp3 algorithm, augments Transformer-based language models for query predictions by combining predictions from experts, thus dynamically building a candidate set during exploration. Immediate feedback is leveraged to choose the appropriate prediction in a probabilistic fashion. We provide comprehensive large-scale experimental and comparative assessment using a popular online literature discovery service, which showcases that our mechanism (i) improves the per-round regret substantially against state-of-the-art Transformer-based models and (ii) shows the superiority of causal language modelling over masked language modelling for query recommendations.<\/jats:p>","DOI":"10.1007\/s10618-024-01057-4","type":"journal-article","created":{"date-parts":[[2024,8,2]],"date-time":"2024-08-02T14:22:05Z","timestamp":1722608525000},"page":"3758-3782","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Sequential query prediction based on multi-armed bandits with ensemble of transformer experts and immediate feedback"],"prefix":"10.1007","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5338-9385","authenticated-orcid":false,"given":"Shameem A.","family":"Puthiya Parambath","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christos","family":"Anagnostopoulos","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Roderick","family":"Murray-Smith","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,8,2]]},"reference":[{"key":"1057_CR1","doi-asserted-by":"crossref","unstructured":"Ahmad WU, Chang KW, Wang H (2019) Context attentive document ranking and query suggestion. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp 385\u2013394","DOI":"10.1145\/3331184.3331246"},{"key":"1057_CR2","unstructured":"Audibert JY, Bubeck S et\u00a0al (2009) Minimax policies for adversarial and stochastic bandits. In: COLT, pp 1\u2013122"},{"issue":"1","key":"1057_CR3","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1137\/S0097539701398375","volume":"32","author":"P Auer","year":"2002","unstructured":"Auer P, Cesa-Bianchi N, Freund Y et al (2002) The nonstochastic multiarmed bandit problem. SIAM J Comput 32(1):48\u201377","journal-title":"SIAM J Comput"},{"key":"1057_CR4","unstructured":"Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: 3rd international conference on learning representations, ICLR 2015"},{"key":"1057_CR5","unstructured":"Bayati M, Hamidi N, Johari R et\u00a0al (2020) Unreasonable effectiveness of greedy algorithms in multi-armed bandit with many arms. In: Larochelle H, Ranzato M, Hadsell R et\u00a0al (eds) NeurIPS, pp 1713\u20131723"},{"key":"1057_CR6","doi-asserted-by":"publisher","first-page":"2103","DOI":"10.1214\/aos\/1069362389","volume":"25","author":"DA Berry","year":"1997","unstructured":"Berry DA, Chen RW, Zame A et al (1997) Bandit problems with infinitely many arms. Ann Stat 25:2103\u20132116","journal-title":"Ann Stat"},{"key":"1057_CR7","doi-asserted-by":"crossref","unstructured":"Bouneffouf D, Rish I, Aggarwal C (2020) Survey on applications of multi-armed and contextual bandits. In: 2020 IEEE Congress on evolutionary computation (CEC). IEEE, pp 1\u20138","DOI":"10.1109\/CEC48606.2020.9185782"},{"key":"1057_CR8","unstructured":"Carpentier A, Valko M (2015) Simple regret for infinitely many armed bandits. In: ICML, PMLR, pp 1133\u20131141"},{"key":"1057_CR9","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511546921","volume-title":"Prediction, learning, and games","author":"N Cesa-Bianchi","year":"2006","unstructured":"Cesa-Bianchi N, Lugosi G (2006) Prediction, learning, and games. Cambridge University Press, Cambridge"},{"issue":"3","key":"1057_CR10","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1145\/258128.258179","volume":"44","author":"N Cesa-Bianchi","year":"1997","unstructured":"Cesa-Bianchi N, Freund Y, Haussler D et al (1997) How to use expert advice. J ACM 44(3):427\u2013485","journal-title":"J ACM"},{"key":"1057_CR11","doi-asserted-by":"crossref","unstructured":"Dai Z, Yang Z, Yang Y et\u00a0al (2019) Transformer-XL: attentive language models beyond a fixed-length context. In: ACL, pp 2978\u20132988","DOI":"10.18653\/v1\/P19-1285"},{"key":"1057_CR12","doi-asserted-by":"crossref","unstructured":"Dehghani M, Rothe S, Alfonseca E et\u00a0al (2017) Learning to attend, copy, and generate for session-based query suggestion. In: Proceedings of the 2017 ACM on conference on information and knowledge management. ACM, pp 1747\u20131756","DOI":"10.1145\/3132847.3133010"},{"key":"1057_CR13","unstructured":"Devlin J, Chang MW, Lee K et\u00a0al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pp 4171\u20134186"},{"issue":"1","key":"1057_CR14","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1006\/jcss.1997.1504","volume":"55","author":"Y Freund","year":"1997","unstructured":"Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119\u2013139","journal-title":"J Comput Syst Sci"},{"key":"1057_CR15","doi-asserted-by":"crossref","unstructured":"Jiang JY, Wang W (2018) RIN: reformulation inference network for context-aware query suggestion. In: Proceedings of the 27th ACM international conference on information and knowledge management, pp 197\u2013206","DOI":"10.1145\/3269206.3271808"},{"key":"1057_CR16","unstructured":"Kalvit A, Zeevi A (2020) From finite to countable-armed bandits. In: Advances in neural information processing systems, pp 8259\u20138269"},{"key":"1057_CR17","unstructured":"Kanade V, McMahan HB, Bryan B (2009) Sleeping experts and bandits with stochastic action availability and adversarial rewards. In: AISTATS, PMLR, pp 272\u2013279"},{"key":"1057_CR18","unstructured":"Keskar NS, McCann B, Varshney L et\u00a0al (2019) CTRL\u2014A conditional transformer language model for controllable generation. arXiv preprint"},{"issue":"2","key":"1057_CR19","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1007\/s10994-010-5178-7","volume":"80","author":"R Kleinberg","year":"2010","unstructured":"Kleinberg R, Niculescu-Mizil A, Sharma Y (2010) Regret bounds for sleeping experts and bandits. Mach Learn 80(2):245\u2013272","journal-title":"Mach Learn"},{"issue":"4","key":"1057_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3299873","volume":"66","author":"R Kleinberg","year":"2019","unstructured":"Kleinberg R, Slivkins A, Upfal E (2019) Bandits and experts in metric spaces. J ACM 66(4):1\u201377","journal-title":"J ACM"},{"key":"1057_CR21","doi-asserted-by":"publisher","DOI":"10.1017\/9781108571401","volume-title":"Bandit algorithms","author":"T Lattimore","year":"2020","unstructured":"Lattimore T, Szepesv\u00e1ri C (2020) Bandit algorithms. Cambridge University Press, Cambridge"},{"key":"1057_CR22","doi-asserted-by":"crossref","unstructured":"Lewis M, Liu Y, Goyal N et\u00a0al (2020) BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th annual meeting of the Association for Computational Linguistics, pp 7871\u20137880","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"1057_CR23","unstructured":"Mustar A, Lamprier S, Piwowarski B (2020) Using BERT and BART for query suggestion. In: Joint conference of the information retrieval communities in Europe. CEUR-WS.org"},{"issue":"1","key":"1057_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3470562","volume":"40","author":"A Mustar","year":"2021","unstructured":"Mustar A, Lamprier S, Piwowarski B (2021) On the study of transformers for query suggestion. ACM Trans Inf Syst 40(1):1\u201327","journal-title":"ACM Trans Inf Syst"},{"key":"1057_CR25","doi-asserted-by":"crossref","unstructured":"Papineni K, Roukos S, Ward T et\u00a0al (2002) BLEU: a method for automatic evaluation of machine translation. In: ACL, pp 311\u2013318","DOI":"10.3115\/1073083.1073135"},{"key":"1057_CR26","unstructured":"Parambath SAP, Anagnostopoulos C, Murray-Smith R et\u00a0al (2021) Max-utility based arm selection strategy for sequential query recommendations. In: ACML, PMLR"},{"issue":"8","key":"1057_CR27","first-page":"9","volume":"1","author":"A Radford","year":"2019","unstructured":"Radford A, Wu J, Child R et al (2019) Language models are unsupervised multitask learners. OpenAI Blog 1(8):9","journal-title":"OpenAI Blog"},{"key":"1057_CR28","doi-asserted-by":"crossref","unstructured":"Ren G, Ni X, Malik M et\u00a0al (2018) Conversational query understanding using sequence to sequence modeling. In: Proceedings of the 2018 World Wide Web conference, pp 1715\u20131724","DOI":"10.1145\/3178876.3186083"},{"key":"1057_CR29","unstructured":"Saha A, Gaillard P, Valko M (2020) Improved sleeping bandits with stochastic action sets and adversarial rewards. In: ICML, PMLR, pp 8357\u20138366"},{"key":"1057_CR30","doi-asserted-by":"crossref","unstructured":"Sordoni A, Bengio Y, Vahabi H et\u00a0al (2015) A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In: CIKM, pp 553\u2013562","DOI":"10.1145\/2806416.2806493"},{"key":"1057_CR31","unstructured":"Vaswani A, Shazeer N, Parmar N et\u00a0al (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S et\u00a0al (eds) Advances in neural information processing systems, vol\u00a030. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/2017\/file\/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf"},{"key":"1057_CR32","unstructured":"Vinyals O, Fortunato M, Jaitly N (2015) Pointer networks. In: Cortes C, Lawrence N, Lee D et\u00a0al (eds) Advances in neural information processing systems, vol\u00a028. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper\/2015\/file\/29921001f2f04bd3baee84a12e98098f-Paper.pdf"},{"key":"1057_CR33","first-page":"1729","volume":"21","author":"Y Wang","year":"2008","unstructured":"Wang Y, Audibert JY, Munos R (2008) Algorithms for infinitely many-armed bandits. NIPS 21:1729\u20131736","journal-title":"NIPS"},{"key":"1057_CR34","doi-asserted-by":"crossref","unstructured":"Wu B, Xiong C, Sun M et\u00a0al (2018) Query suggestion with feedback memory network. In: Proceedings of the 2018 World Wide Web conference, pp 1563\u20131571","DOI":"10.1145\/3178876.3186068"},{"key":"1057_CR35","unstructured":"Zhu Y, Nowak R (2020) On regret with multiple best arms. In: NeurIPS, vol\u00a033. Curran Associates, Inc., pp 9050\u20139060"}],"container-title":["Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-024-01057-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10618-024-01057-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-024-01057-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,28]],"date-time":"2024-10-28T09:11:34Z","timestamp":1730106694000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10618-024-01057-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,2]]},"references-count":35,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,11]]}},"alternative-id":["1057"],"URL":"https:\/\/doi.org\/10.1007\/s10618-024-01057-4","relation":{},"ISSN":["1384-5810","1573-756X"],"issn-type":[{"type":"print","value":"1384-5810"},{"type":"electronic","value":"1573-756X"}],"subject":[],"published":{"date-parts":[[2024,8,2]]},"assertion":[{"value":"7 February 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 June 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 August 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no Conflict of interest to declare. All co-authors have seen and agree with the contents of the manuscript and there is no financial interest to report. We certify that the submission is original work and is not under review at any other publication.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}