{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T01:02:12Z","timestamp":1774400532207,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":56,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,10,17]],"date-time":"2022-10-17T00:00:00Z","timestamp":1665964800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,10,17]]},"DOI":"10.1145\/3511808.3557637","type":"proceedings-article","created":{"date-parts":[[2022,10,16]],"date-time":"2022-10-16T01:22:22Z","timestamp":1665883342000},"page":"4449-4453","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Measuring and Comparing the Consistency of IR Models for Query Pairs with Similar and Different Information Needs"],"prefix":"10.1145","author":[{"given":"Procheta","family":"Sen","sequence":"first","affiliation":[{"name":"University of Liverpool, Liverpool, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sourav","family":"Saha","sequence":"additional","affiliation":[{"name":"Indian Statistical Institute, Kolkata, India"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Debasis","family":"Ganguly","sequence":"additional","affiliation":[{"name":"University of Glasgow, Glasgow, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Manisha","family":"Verma","sequence":"additional","affiliation":[{"name":"Amazon, New York, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dwaipayan","family":"Roy","sequence":"additional","affiliation":[{"name":"IISER, Kolkata, India"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,10,17]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.2196\/15154"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914671"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080839"},{"key":"e_1_3_2_1_4_1","volume-title":"Proc. of the 22nd Australasian Document Computing Symposium.","author":"Benham Rodger","unstructured":"Rodger Benham and J. Shane Culpepper . 2017. Risk-Reward Trade-Offs in Rank Fusion . In Proc. of the 22nd Australasian Document Computing Symposium. Rodger Benham and J. Shane Culpepper. 2017. Risk-Reward Trade-Offs in Rank Fusion. In Proc. of the 22nd Australasian Document Computing Symposium."},{"key":"e_1_3_2_1_5_1","volume-title":"Proc. of ADC'04","author":"Billerbeck Bodo","year":"2004","unstructured":"Bodo Billerbeck and Justin Zobel . 2004 . Questioning Query Expansion: An Examination of Behaviour and Parameters . In Proc. of ADC'04 . 69--76. Bodo Billerbeck and Justin Zobel. 2004. Questioning Query Expansion: An Examination of Behaviour and Parameters. In Proc. of ADC'04. 69--76."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(98)00110-X"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772714"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767876"},{"key":"e_1_3_2_1_9_1","volume-title":"Overview of the TREC 2019 deep learning track. arxiv","author":"Craswell Nick","year":"2003","unstructured":"Nick Craswell , Bhaskar Mitra , Emine Yilmaz , Daniel Campos , and Ellen M. Voorhees . 2020 . Overview of the TREC 2019 deep learning track. arxiv : 2003 .07820 [cs.IR] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the TREC 2019 deep learning track. arxiv: 2003.07820 [cs.IR]"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613984.1613997"},{"key":"e_1_3_2_1_11_1","volume-title":"Proc. of the 2019 NAACL-HLT","volume":"1","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019 . BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proc. of the 2019 NAACL-HLT , Volume 1 (Long and Short Papers). 4171--4186. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of the 2019 NAACL-HLT, Volume 1 (Long and Short Papers). 4171--4186."},{"key":"e_1_3_2_1_12_1","volume-title":"Burges","author":"Donmez Pinar","year":"2009","unstructured":"Pinar Donmez , Krysta M. Svore , and Christopher J. C . Burges . 2009 . On the local optimality of LambdaRank. In SIGIR. 460--467. Pinar Donmez, Krysta M. Svore, and Christopher J. C. Burges. 2009. On the local optimality of LambdaRank. In SIGIR. 460--467."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331312"},{"key":"e_1_3_2_1_14_1","unstructured":"Andrea Ferrario Michele Loi and Eleonora Vigan\u00f2. 2019. In AI we trust Incrementally: a Multi-layer model of trust to analyze Human-Artificial intelligence interactions. Philosophy & Technology (2019) 1--17. Andrea Ferrario Michele Loi and Eleonora Vigan\u00f2. 2019. In AI we trust Incrementally: a Multi-layer model of trust to analyze Human-Artificial intelligence interactions. Philosophy & Technology (2019) 1--17."},{"key":"e_1_3_2_1_15_1","volume-title":"Jones","author":"Ganguly Debasis","year":"2012","unstructured":"Debasis Ganguly , Johannes Leveling , and Gareth J. F . Jones . 2012 . Cross-Lingual Topical Relevance Models. In COLING. Indian Institute of Technology Bombay , 927--942. Debasis Ganguly, Johannes Leveling, and Gareth J. F. Jones. 2012. Cross-Lingual Topical Relevance Models. In COLING. Indian Institute of Technology Bombay, 927--942."},{"key":"e_1_3_2_1_16_1","volume-title":"Proc. of 25th ACM CIKM. 55--64","author":"Guo Jiafeng","unstructured":"Jiafeng Guo , Yixing Fan , Qingyao Ai , and W. Bruce Croft . 2016. A Deep Relevance Matching Model for Ad-Hoc Retrieval . In Proc. of 25th ACM CIKM. 55--64 . Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A Deep Relevance Matching Model for Ad-Hoc Retrieval. In Proc. of 25th ACM CIKM. 55--64."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.techfore.2015.12.014"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398515"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2541268.2541270"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/358413.358430"},{"key":"e_1_3_2_1_21_1","volume-title":"TREC","author":"Jaleel Nasreen Abdul","year":"2004","unstructured":"Nasreen Abdul Jaleel , James Allan , W. Bruce Croft , Fernando Diaz , Leah S. Larkey , Xiaoyan Li , Mark D. Smucker , and Courtney Wade . 2004 . UMass at TREC 2004: Novelty and HARD . In TREC 2004. Nasreen Abdul Jaleel, James Allan, W. Bruce Croft, Fernando Diaz, Leah S. Larkey, Xiaoyan Li, Mark D. Smucker, and Courtney Wade. 2004. UMass at TREC 2004: Novelty and HARD. In TREC 2004."},{"key":"e_1_3_2_1_22_1","volume-title":"Proc. of AAAI'20","author":"Jan Steve TK","unstructured":"Steve TK Jan , Vatche I., and Vinod M . 2020. AI trust in business processes: the need for process-aware explanations . In Proc. of AAAI'20 . 13403--13404. Steve TK Jan, Vatche I., and Vinod M. 2020. AI trust in business processes: the need for process-aware explanations. In Proc. of AAAI'20. 13403--13404."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458356"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Diane Kelly and Leif Azzopardi. 2015. How many results per page?: A Study of SERP Size Search Behavior and User Experience. In SIGIR. ACM 183--192. Diane Kelly and Leif Azzopardi. 2015. How many results per page?: A Study of SERP Size Search Behavior and User Experience. In SIGIR. ACM 183--192.","DOI":"10.1145\/2766462.2767732"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401075"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911521"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767757"},{"key":"e_1_3_2_1_28_1","volume-title":"Koch","author":"Richard Landis J.","year":"1977","unstructured":"J. Richard Landis and Gary G . Koch . 1977 . The Measurement of Observer Agreement for Categorical Data. Biometrics , Vol. 33 (1977). J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics, Vol. 33 (1977)."},{"key":"e_1_3_2_1_29_1","volume-title":"Proc. of SIGIR '01","author":"Lavrenko Victor","unstructured":"Victor Lavrenko and W. Bruce Croft . 2001. Relevance Based Language Models . In Proc. of SIGIR '01 . 120--127. Victor Lavrenko and W. Bruce Croft. 2001. Relevance Based Language Models. In Proc. of SIGIR '01. 120--127."},{"key":"e_1_3_2_1_30_1","volume-title":"Proc. of 31st NeuRIPS","author":"Lundberg Scott","year":"2017","unstructured":"Scott Lundberg and Su-In Lee . 2017 . A Unified Approach to Interpreting Model Predictions . In Proc. of 31st NeuRIPS ( Long Beach, California, USA). 4768--4777. Scott Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Proc. of 31st NeuRIPS (Long Beach, California, USA). 4768--4777."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911507"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3133098"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"crossref","unstructured":"Ali Montazeralghaem Hamed Zamani and James Allan. 2020. A Reinforcement Learning Framework for Relevance Feedback. In SIGIR. ACM 59--68. Ali Montazeralghaem Hamed Zamani and James Allan. 2020. A Reinforcement Learning Framework for Relevance Feedback. In SIGIR. ACM 59--68.","DOI":"10.1145\/3397271.3401099"},{"key":"e_1_3_2_1_34_1","volume-title":"Maurice van Keulen, and Christin Seifert.","author":"Nauta Meike","year":"2022","unstructured":"Meike Nauta , Jan Trienes , Shreyasi Pathak , Elisa Nguyen , Michelle Peters , Yasmin Schmitt , J\u00f6 rg Schl\u00f6 tterer , Maurice van Keulen, and Christin Seifert. 2022 . From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI. CoRR , Vol. abs\/ 2201 .08164 (2022). Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, J\u00f6 rg Schl\u00f6 tterer, Maurice van Keulen, and Christin Seifert. 2022. From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI. CoRR, Vol. abs\/2201.08164 (2022)."},{"key":"e_1_3_2_1_35_1","volume-title":"Proc. of CoCo@NIPS.","author":"Nguyen Tri","year":"2016","unstructured":"Tri Nguyen , Mir Rosenberg , Xia Song , Jianfeng Gao , Saurabh Tiwary , Rangan Majumder , and Li Deng . 2016 . MS MARCO: A Human Generated MAchine Reading COmprehension Dataset . In Proc. of CoCo@NIPS. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In Proc. of CoCo@NIPS."},{"key":"e_1_3_2_1_36_1","volume-title":"Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085","author":"Nogueira Rodrigo","year":"2019","unstructured":"Rodrigo Nogueira and Kyunghyun Cho . 2019. Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085 ( 2019 ). Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085 (2019)."},{"key":"e_1_3_2_1_37_1","volume-title":"Lin","author":"Nogueira Rodrigo","year":"2019","unstructured":"Rodrigo Nogueira , Wei Yang , Kyunghyun Cho , and Jimmy J . Lin . 2019 . Multi-Stage Document Ranking with BERT. ArXiv , Vol. abs\/ 1910 .14424 (2019). Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy J. Lin. 2019. Multi-Stage Document Ranking with BERT. ArXiv, Vol. abs\/1910.14424 (2019)."},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1282100.1282179"},{"key":"e_1_3_2_1_39_1","volume-title":"The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. CoRR","author":"Pradeep Ronak","year":"2021","unstructured":"Ronak Pradeep , Rodrigo Nogueira , and Jimmy Lin . 2021. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. CoRR , Vol. abs\/ 2101 .05667 ( 2021 ). Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. CoRR, Vol. abs\/2101.05667 (2021)."},{"key":"e_1_3_2_1_40_1","first-page":"1","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi Zhou , Wei Li , and Peter J. Liu . 2020 . Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer . JMLR , Vol. 21 , 140 (2020), 1 -- 67 . Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, Vol. 21, 140 (2020), 1--67.","journal-title":"JMLR"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401279"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"crossref","unstructured":"S.E. Robertson S. Walker M.M. Beaulieu M. Gatford and A. Payne. 1996. Okapi at TREC-4. S.E. Robertson S. Walker M.M. Beaulieu M. Gatford and A. Payne. 1996. Okapi at TREC-4.","DOI":"10.6028\/NIST.SP.500-236.interactive-city"},{"key":"e_1_3_2_1_44_1","volume-title":"Proc. of CIKM '16","author":"Roy Dwaipayan","unstructured":"Dwaipayan Roy , Debasis Ganguly , Mandar Mitra , and Gareth J.F. Jones . 2016. Word Vector Compositionality Based Relevance Feedback Using Kernel Density Estimation . In Proc. of CIKM '16 . 1281--1290. Dwaipayan Roy, Debasis Ganguly, Mandar Mitra, and Gareth J.F. Jones. 2016. Word Vector Compositionality Based Relevance Feedback Using Kernel Density Estimation. In Proc. of CIKM '16. 1281--1290."},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11948-020-00228-y"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835494"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2180868.2180873"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1099554.1099646"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372834"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331377"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835840"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"crossref","unstructured":"Xiao Wang Craig Macdonald Nicola Tonellotto and Iadh Ounis. 2021. Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval. In ICTIR. ACM 297--306. Xiao Wang Craig Macdonald Nicola Tonellotto and Iadh Ounis. 2021. Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval. In ICTIR. ACM 297--306.","DOI":"10.1145\/3471158.3472250"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852102.1852106"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661953"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210041"},{"key":"e_1_3_2_1_56_1","volume-title":"Proc. of SIGIR'07","author":"Zhou Yun","unstructured":"Yun Zhou and W. Bruce Croft . 2007. Query Performance Prediction in Web Search Environments . In Proc. of SIGIR'07 . 543--550. Yun Zhou and W. Bruce Croft. 2007. Query Performance Prediction in Web Search Environments. In Proc. of SIGIR'07. 543--550."}],"event":{"name":"CIKM '22: The 31st ACM International Conference on Information and Knowledge Management","location":"Atlanta GA USA","acronym":"CIKM '22","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 31st ACM International Conference on Information & Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3511808.3557637","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3511808.3557637","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:51:09Z","timestamp":1750182669000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3511808.3557637"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,17]]},"references-count":56,"alternative-id":["10.1145\/3511808.3557637","10.1145\/3511808"],"URL":"https:\/\/doi.org\/10.1145\/3511808.3557637","relation":{},"subject":[],"published":{"date-parts":[[2022,10,17]]},"assertion":[{"value":"2022-10-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}