{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T17:38:33Z","timestamp":1740159513124,"version":"3.37.3"},"reference-count":8,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,2,10]],"date-time":"2020-02-10T00:00:00Z","timestamp":1581292800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,2,10]],"date-time":"2020-02-10T00:00:00Z","timestamp":1581292800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005626","name":"Universit\u00e4t Regensburg","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005626","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Datenbank Spektrum"],"published-print":{"date-parts":[[2020,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Systematic and repeatable measurement of information systems via test collections, the Cranfield model, has been the mainstay of Information Retrieval since the 1960s. However, this may not be appropriate for newer, more interactive systems, such as Conversational Search agents. Such systems rely on Machine Learning technologies, which are not yet sufficiently advanced to permit true human-like dialogues, and so research can be enabled by simulation via human agents.<\/jats:p><jats:p>In this work we compare dialogues obtained from two studies with the same context, assistance in the kitchen, but with different experimental setups, allowing us to learn about and evaluate conversational IR systems. We discover that users adapt their behaviour when they think they are interacting with a\u00a0system and that human-like conversations in one of the studies were unpredictable to an extent we did not expect. Our results have implications for the development of new studies in this area and, ultimately, the design of future conversational agents.<\/jats:p>","DOI":"10.1007\/s13222-020-00333-z","type":"journal-article","created":{"date-parts":[[2020,2,10]],"date-time":"2020-02-10T11:02:51Z","timestamp":1581332571000},"page":"37-41","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Comparing Wizard of Oz &amp; Observational Studies for Conversational IR Evaluation"],"prefix":"10.1007","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5791-0641","authenticated-orcid":false,"given":"David","family":"Elsweiler","sequence":"first","affiliation":[]},{"given":"Alexander","family":"Frummet","sequence":"additional","affiliation":[]},{"given":"Morgan","family":"Harvey","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,2,10]]},"reference":[{"key":"333_CR1","volume-title":"2020 Conference on Human Information Interaction and Retrieval","author":"S Barko-Sherif","year":"2020","unstructured":"Barko-Sherif S, Elsweiler D, Harvey M (2020) Conversational agents for recipe recommendation. In: 2020 Conference on Human Information Interaction and Retrieval. ACM, New York"},{"key":"333_CR2","volume-title":"The second international workshop on conversational approaches to information retrieval","author":"M Dubiel","year":"2018","unstructured":"Dubiel M, Halvey M, Azzopardi L, Daronnat S (2018) Investigating how conversational search agents affect user\u2019s behaviour, performance and search experience. In: The second international workshop on conversational approaches to information retrieval"},{"key":"333_CR3","volume-title":"Natural language for artificial intelligence","author":"A Frummet","year":"2019","unstructured":"Frummet A, Elsweiler D, Ludwig B (2019) Detecting domain-specific information needs inconversational search dialogues. In: Natural language for artificial intelligence"},{"key":"333_CR4","doi-asserted-by":"publisher","first-page":"5286","DOI":"10.1145\/2858036.2858288","volume-title":"Proceedings of the 2016 CHI conference on human factors in computing systems","author":"E Luger","year":"2016","unstructured":"Luger E, Sellen A (2016) Like having a\u00a0really bad pa: the gulf between user expectation and experience of conversational agents. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, New York, pp 5286\u20135297"},{"key":"333_CR5","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1145\/3020165.3020183","volume-title":"Proceedings of the 2017 conference on conference human information interaction and retrieval","author":"F Radlinski","year":"2017","unstructured":"Radlinski F, Craswell N (2017) A\u00a0theoretical framework for conversational search. In: Proceedings of the 2017 conference on conference human information interaction and retrieval. ACM, New York, pp 117\u2013126"},{"issue":"4","key":"333_CR6","doi-asserted-by":"publisher","first-page":"439","DOI":"10.1177\/0165551507086989","volume":"34","author":"S Robertson","year":"2008","unstructured":"Robertson S (2008) On the history of evaluation in ir. J\u00a0Inf Sci 34(4):439\u2013456","journal-title":"J Inf Sci"},{"key":"333_CR7","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1145\/3077136.3080787","volume-title":"Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval","author":"S Shiga","year":"2017","unstructured":"Shiga S, Joho H, Blanco R, Trippas JR, Sanderson M (2017) Modelling information needs in collaborative search conversations. In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp 715\u2013724"},{"key":"333_CR8","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1007\/978-3-030-22948-1_2","volume-title":"Information retrieval evaluation in a\u00a0changing world","author":"EM Voorhees","year":"2019","unstructured":"Voorhees EM (2019) The evolution of cranfield. In: Information retrieval evaluation in a\u00a0changing world. Springer, Berlin Heidelberg, pp 45\u201369"}],"container-title":["Datenbank-Spektrum"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s13222-020-00333-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s13222-020-00333-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s13222-020-00333-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,2,10]],"date-time":"2021-02-10T01:43:16Z","timestamp":1612921396000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s13222-020-00333-z"}},"subtitle":["Lessons Learned from These two Diverse Approaches"],"short-title":[],"issued":{"date-parts":[[2020,2,10]]},"references-count":8,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,3]]}},"alternative-id":["333"],"URL":"https:\/\/doi.org\/10.1007\/s13222-020-00333-z","relation":{},"ISSN":["1618-2162","1610-1995"],"issn-type":[{"type":"print","value":"1618-2162"},{"type":"electronic","value":"1610-1995"}],"subject":[],"published":{"date-parts":[[2020,2,10]]},"assertion":[{"value":"25 November 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 January 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 February 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}