{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T19:07:51Z","timestamp":1768590471995,"version":"3.49.0"},"reference-count":81,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2019,1,17]],"date-time":"2019-01-17T00:00:00Z","timestamp":1547683200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2019,1,17]]},"abstract":"<jats:p>Modern Information Retrieval (IR) systems, such as search engines, recommender systems, and conversational agents, are best thought of as interactive systems. And their development is best thought of as a two-stage development process: offline development followed by continued online adaptation and development based on interactions with users. In this opinion paper, we take a closer look at existing IR textbooks and teaching materials, and examine to which degree they cover the offline and online stages of the IR system development process. We notice that current teaching materials in IR focus mostly on search and on the offline development phase. Other scenarios of interacting with information are largely absent from current IR teaching materials, as is the (interactive) online development phase. We identify a list of scenarios and a list of topics that we believe are essential to any modern set of IR teaching materials that claims to fully cover IR system development. In particular, we argue for more attention, in basic IR teaching materials, to scenarios such as recommender systems, and to topics such as query and interaction mining and understanding, online evaluation, and online learning to rank.<\/jats:p>","DOI":"10.1145\/3308774.3308780","type":"journal-article","created":{"date-parts":[[2019,1,17]],"date-time":"2019-01-17T17:15:15Z","timestamp":1547745315000},"page":"19-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["What Should We Teach in Information Retrieval?"],"prefix":"10.1145","volume":"52","author":[{"given":"Ilya","family":"Markov","sequence":"first","affiliation":[{"name":"University of Amsterdam , NJ, USA"}]},{"given":"Maarten","family":"de Rijke","sequence":"additional","affiliation":[{"name":"University of Amsterdam"}]}],"member":"320","published-online":{"date-parts":[[2019,1,17]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.5555\/2931100"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970398.2970409"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3274274"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274784.3274788"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000052"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/1796408"},{"key":"e_1_2_1_7_1","doi-asserted-by":"crossref","unstructured":"K. Balog. Entity-Oriented Search. Springer 2018.   K. Balog. Entity-Oriented Search. Springer 2018.","DOI":"10.1007\/978-3-319-93935-3"},{"key":"e_1_2_1_8_1","volume-title":"Proceeding of NIPS","author":"Barocas S.","year":"2017","unstructured":"S. Barocas and M. Hardt . Fairness in machine learning . In Proceeding of NIPS , 2017 . URL https:\/\/fairmlbook.org\/. S. Barocas and M. Hardt. Fairness in machine learning. In Proceeding of NIPS, 2017. URL https:\/\/fairmlbook.org\/."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000032"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2888422.2888424"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/792550.792552"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of TheWebConf, 2018","author":"Budylin R.","year":"2018","unstructured":"R. Budylin , A. Drutsa , G. Gusev , E. Kharitonov , P. Serdyukov , and I. Yashkov . Online evaluation for effective web service development . In Proceedings of TheWebConf, 2018 . URL https:\/\/research.yandex.com\/tutorials\/online-evaluation\/www- 2018 . R. Budylin, A. Drutsa, G. Gusev, E. Kharitonov, P. Serdyukov, and I. Yashkov. Online evaluation for effective web service development. In Proceedings of TheWebConf, 2018. URL https:\/\/research.yandex.com\/tutorials\/online-evaluation\/www-2018."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of KDD. ACM, 2018","author":"Budylin R.","year":"2018","unstructured":"R. Budylin , A. Drutsa , G. Gusev , P. Serdyukov , and I. Yashkov . Online evaluation for effective web service development . In Proceedings of KDD. ACM, 2018 . URL https: \/\/research.yandex.com\/tutorials\/online-evaluation\/kdd- 2018 . R. Budylin, A. Drutsa, G. Gusev, P. Serdyukov, and I. Yashkov. Online evaluation for effective web service development. In Proceedings of KDD. ACM, 2018. URL https: \/\/research.yandex.com\/tutorials\/online-evaluation\/kdd-2018."},{"key":"e_1_2_1_14_1","volume-title":"Microsoft Research","author":"Burges C. J. C.","year":"2010","unstructured":"C. J. C. Burges . From RankNet to LambdaRank to LambdaMART: An overview. Technical report , Microsoft Research , 2010 . C. J. C. Burges. From RankNet to LambdaRank to LambdaMART: An overview. Technical report, Microsoft Research, 2010."},{"key":"e_1_2_1_15_1","volume-title":"Information Retrieval: Implementing and Evaluating Search Engines","author":"B\u00a8uttcher S.","year":"2010","unstructured":"S. B\u00a8uttcher , C. L. Clarke , and G. V. Cormack . Information Retrieval: Implementing and Evaluating Search Engines . MIT Press , 2010 . S. B\u00a8uttcher, C. L. Clarke, and G. V. Cormack. Information Retrieval: Implementing and Evaluating Search Engines. MIT Press, 2010."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000055"},{"key":"e_1_2_1_17_1","volume-title":"Scalability challenges in web search engines. Synthesis Lectures on Information Concepts, Retrieval, and Services, 7(6):1--138","author":"Cambazoglu B. B.","year":"2015","unstructured":"B. B. Cambazoglu and R. Baeza-Yates . Scalability challenges in web search engines. Synthesis Lectures on Information Concepts, Retrieval, and Services, 7(6):1--138 , 2015 . B. B. Cambazoglu and R. Baeza-Yates. Scalability challenges in web search engines. Synthesis Lectures on Information Concepts, Retrieval, and Services, 7(6):1--138, 2015."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526711"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094078"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2124295.2124351"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767881"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767882"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-02294-4"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2855113"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341545"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/3152586"},{"key":"e_1_2_1_27_1","volume-title":"Search Engines: Information Retrieval in Practice","author":"Croft W. B.","year":"2010","unstructured":"W. B. Croft , D. Metzler , and T. Strohman . Search Engines: Information Retrieval in Practice . Addison Wesley , 2010 . W. B. Croft, D. Metzler, and T. Strohman. Search Engines: Information Retrieval in Practice. Addison Wesley, 2010."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390392"},{"key":"e_1_2_1_29_1","unstructured":"P. Education. Speech and Language Processing. Jurafsky Daniel and Martin James H. 2008.  P. Education. Speech and Language Processing. Jurafsky Daniel and Martin James H. 2008."},{"key":"e_1_2_1_30_1","volume-title":"Neural approaches to conversational AI. arXiv preprint arXiv:1809.08267","author":"Gao J.","year":"2018","unstructured":"J. Gao , M. Galley , and L. Li . Neural approaches to conversational AI. arXiv preprint arXiv:1809.08267 , 2018 . J. Gao, M. Galley, and L. Li. Neural approaches to conversational AI. arXiv preprint arXiv:1809.08267, 2018."},{"key":"e_1_2_1_31_1","volume-title":"AAAI Spring Symposium Series","author":"Garcia-Gathright J.","year":"2018","unstructured":"J. Garcia-Gathright , A. Springer , and H. Cramer . Assessing and addressing algorithmic bias -- but before we get there . In AAAI Spring Symposium Series , 2018 . J. Garcia-Gathright, A. Springer, and H. Cramer. Assessing and addressing algorithmic bias -- but before we get there. In AAAI Spring Symposium Series, 2018."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3121050.3121108"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914798"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1498759.1498818"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2945386"},{"key":"e_1_2_1_36_1","volume-title":"Cambridge University Press","author":"Hearst M. A.","year":"2009","unstructured":"M. A. Hearst . Search User Interfaces . Cambridge University Press , 2009 . M. A. Hearst. Search User Interfaces. Cambridge University Press, 2009."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063618"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9197-9"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433419"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000051"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3162006"},{"key":"e_1_2_1_42_1","volume-title":"The Turn: Integration of Information Seeking and Retrieval in Context","author":"Ingwersen P.","year":"2005","unstructured":"P. Ingwersen and K. J\u00a8arvelin . The Turn: Integration of Information Seeking and Retrieval in Context . Springer , 2005 . P. Ingwersen and K. J\u00a8arvelin. The Turn: Integration of Information Seeking and Retrieval in Context. Springer, 2005."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914803"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of MobileHCI. ACM","author":"Jones M.","year":"2009","unstructured":"M. Jones . Mobile search tutorial . In Proceedings of MobileHCI. ACM , 2009 . M. Jones. Mobile search tutorial. In Proceedings of MobileHCI. ACM, 2009."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-41718-9_3"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000012"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3162009"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3109859.3109935"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1218772110"},{"key":"e_1_2_1_51_1","volume-title":"Synthesis Lectures on Information Concepts, Retrieval, and Services","author":"Lalmas M.","year":"2009","unstructured":"M. Lalmas . XML Retrieval . Synthesis Lectures on Information Concepts, Retrieval, and Services . Morgan & Claypool Publishers , 2009 . M. Lalmas. XML Retrieval. Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan & Claypool Publishers, 2009."},{"key":"e_1_2_1_52_1","volume-title":"Bandit Algorithms","author":"Lattimore T.","year":"2018","unstructured":"T. Lattimore and C. Szepesv\u00b4ari . Bandit Algorithms . 2018 . T. Lattimore and C. Szepesv\u00b4ari. Bandit Algorithms. 2018."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772758"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935878"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000016"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/1394399"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of TheWebConf","author":"Mehrotra R.","year":"2018","unstructured":"R. Mehrotra , E. Yilmaz , and A. H. Awadallah . Understanding & inferring user tasks and needs . In Proceedings of TheWebConf , 2018 . URL https:\/\/task-ir.github.io\/ Task-based-Search\/. R. Mehrotra, E. Yilmaz, and A. H. Awadallah. Understanding & inferring user tasks and needs. In Proceedings of TheWebConf, 2018. URL https:\/\/task-ir.github.io\/ Task-based-Search\/."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.5555\/3239802"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.2886526"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-017-9321-y"},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of RuSSIR, 2018","author":"Oosterhuis H.","year":"2018","unstructured":"H. Oosterhuis . Learning to rank and evaluation in the online setting . In Proceedings of RuSSIR, 2018 . URL https:\/\/sites.google.com\/view\/harrieoosterhuis\/ russir- 2018 . H. Oosterhuis. Learning to rank and evaluation in the online setting. In Proceedings of RuSSIR, 2018. URL https:\/\/sites.google.com\/view\/harrieoosterhuis\/ russir-2018."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433429"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390255"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458092"},{"key":"e_1_2_1_65_1","doi-asserted-by":"crossref","unstructured":"R. Raieli. Multimedia Information Retrieval. Chandos Publishing 2013.   R. Raieli. Multimedia Information Retrieval. Chandos Publishing 2013.","DOI":"10.1533\/9781780633886"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210185"},{"key":"e_1_2_1_67_1","doi-asserted-by":"crossref","unstructured":"T.\n      Sakai\n    . \n      Laboratory Experiments\n       in \n      Information\n      Retrieval\n       - \n      Sample Sizes Effect\n      Sizes and \n      Statistical\n      Power volume \n  40\n   of \n  The Information Retrieval Series\n  . \n  Springer 2018\n  .  T. Sakai. Laboratory Experiments in Information Retrieval - Sample Sizes Effect Sizes and Statistical Power volume 40 of The Information Retrieval Series. Springer 2018.","DOI":"10.1007\/978-981-13-1199-4"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000040"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000013"},{"key":"e_1_2_1_70_1","first-page":"814","volume-title":"Proceedings of ICML","author":"Swaminathan A.","year":"2015","unstructured":"A. Swaminathan and T. Joachims . Counterfactual risk minimization: Learning from logged bandit feedback . In Proceedings of ICML , pages 814 -- 823 . JMLR.org, 2015 . A. Swaminathan and T. Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In Proceedings of ICML, pages 814--823. JMLR.org, 2015."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484036"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.5555\/3164742"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484053"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139525305"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.5555\/1521589"},{"key":"e_1_2_1_76_1","volume-title":"Managing Gigabytes: Compressing and Indexing Documents and Images","author":"Witten I.","year":"1999","unstructured":"I. Witten , A. Moffat , and T. Bell . Managing Gigabytes: Compressing and Indexing Documents and Images . Morgan Kaufmann Publishers , 1999 . I. Witten, A. Moffat, and T. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann Publishers, 1999."},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553527"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.5555\/1457561"},{"key":"e_1_2_1_79_1","volume-title":"https:\/\/www.coursera.org\/learn\/textretrieval","author":"Zhai C.","year":"2018","unstructured":"C. Zhai . Text retrieval and search engines. https:\/\/www.coursera.org\/learn\/textretrieval , 2018 . C. Zhai. Text retrieval and search engines. https:\/\/www.coursera.org\/learn\/textretrieval, 2018."},{"key":"e_1_2_1_80_1","volume-title":"Explainable recommendation: A survey and new perspectives. arXiv preprint arXiv:1804.11192","author":"Zhang Y.","year":"2018","unstructured":"Y. Zhang and X. Chen . Explainable recommendation: A survey and new perspectives. arXiv preprint arXiv:1804.11192 , 2018 . Y. Zhang and X. Chen. Explainable recommendation: A survey and new perspectives. arXiv preprint arXiv:1804.11192, 2018."},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190580.3190584"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308774.3308780","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3308774.3308780","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:58:03Z","timestamp":1750208283000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308774.3308780"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1,17]]},"references-count":81,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,1,17]]}},"alternative-id":["10.1145\/3308774.3308780"],"URL":"https:\/\/doi.org\/10.1145\/3308774.3308780","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2019,1,17]]},"assertion":[{"value":"2019-01-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}