{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:18:08Z","timestamp":1784179088499,"version":"3.55.0"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2018,8,31]],"date-time":"2018-08-31T00:00:00Z","timestamp":1535673600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2018,8,31]]},"abstract":"<jats:p>The purpose of the Strategic Workshop in Information Retrieval in Lorne is to explore the long-range issues of the Information Retrieval field, to recognize challenges that are on - or even over - the horizon, to build consensus on some of the key challenges, and to disseminate the resulting information to the research community. The intent is that this description of open problems will help to inspire researchers and graduate students to address the questions, and will provide funding agencies data to focus and coordinate support for information retrieval research.<\/jats:p>","DOI":"10.1145\/3274784.3274788","type":"journal-article","created":{"date-parts":[[2018,9,4]],"date-time":"2018-09-04T12:37:30Z","timestamp":1536064650000},"page":"34-90","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":132,"title":["Research Frontiers in Information Retrieval"],"prefix":"10.1145","volume":"52","author":[{"given":"J. Shane","family":"Culpepper","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fernando","family":"Diaz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark D.","family":"Smucker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,8,31]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080832"},{"key":"e_1_2_1_2_1","first-page":"1693","volume-title":"Proceedings of the 28th International Conference on Neural Information Processing Systems -","volume":"1","author":"Hermann Karl Moritz","year":"2015","unstructured":"Karl Moritz Hermann , Tom\u00e1? Ko?isk?, Edward Grefenstette , Lasse Espeholt , Will Kay , Mustafa Suleyman , and Phil Blunsom . Teaching machines to read and comprehend . In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 , NIPS'15, pages 1693 - 1701 , Cambridge, MA, USA , 2015 . MIT Press. Karl Moritz Hermann, Tom\u00e1? Ko?isk?, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, pages 1693-1701, Cambridge, MA, USA, 2015. MIT Press."},{"key":"e_1_2_1_3_1","volume-title":"A joint many-task model: Growing a neural network for multiple NLP tasks. CoRR, abs\/1611.01587","author":"Hashimoto Kazuma","year":"2016","unstructured":"Kazuma Hashimoto , Caiming Xiong , Yoshimasa Tsuruoka , and Richard Socher . A joint many-task model: Growing a neural network for multiple NLP tasks. CoRR, abs\/1611.01587 , 2016 . Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing a neural network for multiple NLP tasks. CoRR, abs\/1611.01587, 2016."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000061"},{"key":"e_1_2_1_5_1","volume-title":"Efficient estimation of word representations in vector space. CoRR, abs\/1301.3781","author":"Mikolov Tomas","year":"2013","unstructured":"Tomas Mikolov , Kai Chen , Greg Corrado , and Jeffrey Dean . Efficient estimation of word representations in vector space. CoRR, abs\/1301.3781 , 2013 . Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. CoRR, abs\/1301.3781, 2013."},{"key":"e_1_2_1_6_1","volume-title":"On the state of the art of evaluation in neural language models. CoRR, abs\/1707.05589","author":"Melis G\u00e1bor","year":"2017","unstructured":"G\u00e1bor Melis , Chris Dyer , and Phil Blunsom . On the state of the art of evaluation in neural language models. CoRR, abs\/1707.05589 , 2017 . G\u00e1bor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. CoRR, abs\/1707.05589, 2017."},{"key":"e_1_2_1_7_1","first-page":"1291","volume-title":"Proceedings of the 26th International Conference on World Wide Web, WWW '17","author":"Mitra Bhaskar","year":"2017","unstructured":"Bhaskar Mitra , Fernando Diaz , and Nick Craswell . Learning to match using local and distributed representations of text for web search . In Proceedings of the 26th International Conference on World Wide Web, WWW '17 , pages 1291 - 1299 , Republic and Canton of Geneva, Switzerland , 2017 . International World Wide Web Conferences Steering Committee . Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, pages 1291-1299, Republic and Canton of Geneva, Switzerland, 2017. International World Wide Web Conferences Steering Committee."},{"key":"e_1_2_1_8_1","first-page":"3111","volume-title":"Advances in Neural Information Processing Systems 26","author":"Mikolov Tomas","year":"2013","unstructured":"Tomas Mikolov , Ilya Sutskever , Kai Chen , Greg S Corrado , and Jeff Dean . Distributed representations of words and phrases and their compositionality. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors , Advances in Neural Information Processing Systems 26 , pages 3111 - 3119 . Curran Associates, Inc. , 2013 . Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111-3119. Curran Associates, Inc., 2013."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098035"},{"key":"e_1_2_1_10_1","first-page":"2440","volume-title":"Proceedings of the 28th International Conference on Neural Information Processing Systems -","volume":"2","author":"Sukhbaatar Sainbayar","year":"2015","unstructured":"Sainbayar Sukhbaatar , Arthur Szlam , Jason Weston , and Rob Fergus . End-to-end memory networks . In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 , NIPS'15, pages 2440 - 2448 , Cambridge, MA, USA , 2015 . MIT Press. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, pages 2440-2448, Cambridge, MA, USA, 2015. MIT Press."},{"key":"e_1_2_1_11_1","volume-title":"Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. CoRR, abs\/1708.08296","author":"Samek Wojciech","year":"2017","unstructured":"Wojciech Samek , Thomas Wiegand , and Klaus-Robert M\u00fcller . Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. CoRR, abs\/1708.08296 , 2017 . Wojciech Samek, Thomas Wiegand, and Klaus-Robert M\u00fcller. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. CoRR, abs\/1708.08296, 2017."},{"key":"e_1_2_1_12_1","volume-title":"Memory networks. CoRR, abs\/1410.3916","author":"Weston Jason","year":"2014","unstructured":"Jason Weston , Sumit Chopra , and Antoine Bordes . Memory networks. CoRR, abs\/1410.3916 , 2014 . Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. CoRR, abs\/1410.3916, 2014."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080768"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080809"},{"key":"e_1_2_1_15_1","first-page":"2397","volume-title":"Proceedings of the 33rd International Conference on International Conference on Machine Learning -","volume":"48","author":"Xiong Caiming","year":"2016","unstructured":"Caiming Xiong , Stephen Merity , and Richard Socher . Dynamic memory networks for visual and textual question answering . In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 , ICML'16, pages 2397 - 2406 . JMLR.org, 2016 . Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, pages 2397-2406. JMLR.org, 2016."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939746"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911525"},{"key":"e_1_2_1_18_1","first-page":"138","volume-title":"ICWSM 2012 - Proceedings of the 6th International AAAI Conference on Weblogs and Social Media","author":"Hecht Brent","year":"2012","unstructured":"Brent Hecht , Jaime Teevan , Meredith Ringe Morris , and Dan Liebling . Searchbuddies : Bringing search engines into the conversation . In ICWSM 2012 - Proceedings of the 6th International AAAI Conference on Weblogs and Social Media , pages 138 - 145 , 12 2012 . Brent Hecht, Jaime Teevan, Meredith Ringe Morris, and Dan Liebling. Searchbuddies: Bringing search engines into the conversation. In ICWSM 2012 - Proceedings of the 6th International AAAI Conference on Weblogs and Social Media, pages 138-145, 12 2012."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Jiepu Jiang Ahmed Hassan Awadallah Rosie Jones Umut Ozertem Imed Zitouni Ranjitha Gurunath Kulkarni and Omar Zia Khan. Automatic online evaluation of intelligent assistants. In Proceedings of the 24th International Conference on World Wide Web WWW '15 pages 506-516 Republic and Canton of Geneva Switzerland 2015. International World Wide Web Conferences Steering Committee.   Jiepu Jiang Ahmed Hassan Awadallah Rosie Jones Umut Ozertem Imed Zitouni Ranjitha Gurunath Kulkarni and Omar Zia Khan. Automatic online evaluation of intelligent assistants. In Proceedings of the 24th International Conference on World Wide Web WWW '15 pages 506-516 Republic and Canton of Geneva Switzerland 2015. International World Wide Web Conferences Steering Committee.","DOI":"10.1145\/2736277.2741669"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1127"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3020165.3020183"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/3016387.3016435"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053175"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/E17-1042"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911542"},{"key":"e_1_2_1_26_1","first-page":"4349","volume-title":"Advances in Neural Information Processing Systems 29","author":"Bolukbasi Tolga","year":"2016","unstructured":"Tolga Bolukbasi , Kai-Wei Chang , James Y Zou , Venkatesh Saligrama , and Adam T Kalai . Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors , Advances in Neural Information Processing Systems 29 , pages 4349 - 4357 . Curran Associates, Inc. , 2016 . Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4349-4357. Curran Associates, Inc., 2016."},{"key":"e_1_2_1_27_1","first-page":"104","article-title":"Big data's disparate impact","author":"Barocas Solon","year":"2016","unstructured":"Solon Barocas and Andrew D. Selbst . Big data's disparate impact . California Law Review , 104 , 2016 . Solon Barocas and Andrew D. Selbst. Big data's disparate impact. California Law Review, 104, 2016.","journal-title":"California Law Review"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aal4230"},{"issue":"1","key":"e_1_2_1_29_1","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1515\/popets-2015-0007","article-title":"Automated experiments on ad privacy settings","volume":"2015","author":"Datta Amit","year":"2015","unstructured":"Amit Datta , Michael Carl Tschantz , and Anupam Datta . Automated experiments on ad privacy settings . PoPETs , 2015 ( 1 ): 92 - 112 , 2015 . Amit Datta, Michael Carl Tschantz, and Anupam Datta. Automated experiments on ad privacy settings. PoPETs, 2015(1):92-112, 2015.","journal-title":"PoPETs"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159654"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1093\/poq\/nfv039"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3054197"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025727"},{"key":"e_1_2_1_34_1","volume-title":"Social data: Biases, methodological pitfalls, and ethical boundaries. Available at SSRN","author":"Olteanu Alexandra","year":"2017","unstructured":"Alexandra Olteanu , Carlos Castillo , Fernando Diaz , and Emre Kiciman . Social data: Biases, methodological pitfalls, and ethical boundaries. Available at SSRN , 2017 . Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kiciman. Social data: Biases, methodological pitfalls, and ethical boundaries. Available at SSRN, 2017."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098155"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/2567709.2567766"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094076"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2808194.2809469"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767812"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018699"},{"key":"e_1_2_1_41_1","volume-title":"Incorporating user expectations and behavior into the measurement of search effectiveness. ACM Transactions on Information Systems (TOIS), 35(3):24","author":"Moffat Alistair","year":"2017","unstructured":"Alistair Moffat , Peter Bailey , Falk Scholer , and Paul Thomas . Incorporating user expectations and behavior into the measurement of search effectiveness. ACM Transactions on Information Systems (TOIS), 35(3):24 , 2017 . Alistair Moffat, Peter Bailey, Falk Scholer, and Paul Thomas. Incorporating user expectations and behavior into the measurement of search effectiveness. ACM Transactions on Information Systems (TOIS), 35(3):24, 2017."},{"key":"e_1_2_1_42_1","volume-title":"Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems (TOIS), 27(1):2","author":"Moffat Alistair","year":"2008","unstructured":"Alistair Moffat and Justin Zobel . Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems (TOIS), 27(1):2 , 2008 . Alistair Moffat and Justin Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems (TOIS), 27(1):2, 2008."},{"key":"e_1_2_1_43_1","volume-title":"Estimating the reproducibility of psychological science. Science, 349(6251)","author":"Collaboration Open Science","year":"2015","unstructured":"Open Science Collaboration . Estimating the reproducibility of psychological science. Science, 349(6251) , 2015 . Open Science Collaboration. Estimating the reproducibility of psychological science. Science, 349(6251), 2015."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911492"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-015-9273-z"},{"key":"e_1_2_1_46_1","first-page":"3632","volume-title":"Advances in Neural Information Processing Systems 30","author":"Swaminathan Adith","year":"2017","unstructured":"Adith Swaminathan , Akshay Krishnamurthy , Alekh Agarwal , Miro Dudik , John Langford , Damien Jose , and Imed Zitouni . Off-policy evaluation for slate recommendation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors , Advances in Neural Information Processing Systems 30 , pages 3632 - 3642 . Curran Associates, Inc. , 2017 . Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. Off-policy evaluation for slate recommendation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3632-3642. Curran Associates, Inc., 2017."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3086701"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911537"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911513"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983815"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080819"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080789"},{"key":"e_1_2_1_53_1","volume-title":"Jeff Dean, and Neoklis Polyzotis. The case for learned index structures.","author":"Kraska Tim","year":"2017","unstructured":"Tim Kraska , Alex Beutel , Ed H. Chi , Jeff Dean, and Neoklis Polyzotis. The case for learned index structures. 2017 . ArXiV - https:\/\/arxiv.org\/abs\/1712.01208. Tim Kraska, Alex Beutel, Ed H. Chi, Jeff Dean, and Neoklis Polyzotis. The case for learned index structures. 2017. ArXiV - https:\/\/arxiv.org\/abs\/1712.01208."},{"key":"e_1_2_1_54_1","first-page":"73","volume-title":"Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15","author":"Lucchese Claudio","year":"2015","unstructured":"Claudio Lucchese , Franco Maria Nardini , Salvatore Orlando , Raffaele Perego , Nicola Tonellotto , and Rossano Venturini . Quickscorer : A fast algorithm to rank documents with additive ensembles of regression trees . In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15 , pages 73 - 82 , New York, NY, USA , 2015 . ACM. Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. Quickscorer: A fast algorithm to rank documents with additive ensembles of regression trees. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15, pages 73-82, New York, NY, USA, 2015. ACM."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609615"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274784.3274788","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274784.3274788","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:03Z","timestamp":1750207443000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274784.3274788"}},"subtitle":["Report from the Third Strategic Workshop on Information Retrieval in Lorne (SWIRL 2018)"],"short-title":[],"issued":{"date-parts":[[2018,8,31]]},"references-count":55,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2018,8,31]]}},"alternative-id":["10.1145\/3274784.3274788"],"URL":"https:\/\/doi.org\/10.1145\/3274784.3274788","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2018,8,31]]},"assertion":[{"value":"2018-08-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}