{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,10]],"date-time":"2026-05-10T02:17:29Z","timestamp":1778379449930,"version":"3.51.4"},"reference-count":46,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,6,14]],"date-time":"2024-06-14T00:00:00Z","timestamp":1718323200000},"content-version":"vor","delay-in-days":165,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,6,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Answering factual questions from heterogenous sources, such as graphs and text, is a key capacity of intelligent systems. Current approaches either (i) perform question answering over text and structured sources as separate pipelines followed by a merge step or (ii) provide an early integration, giving up the strengths of particular information sources. To solve this problem, we present \u201cHumanIQ\u201d, a method that teaches language models to dynamically combine retrieved information by imitating how humans use retrieval tools. Our approach couples a generic method for gathering human demonstrations of tool use with adaptive few-shot learning for tool augmented models. We show that HumanIQ confers significant benefits, including i) reducing the error rate of our strongest baseline (GPT-4) by over 50% across 3 benchmarks, (ii) improving human preference over responses from vanilla GPT-4 (45.3% wins, 46.7% ties, 8.0% loss), and (iii) outperforming numerous task-specific baselines.<\/jats:p>","DOI":"10.1162\/tacl_a_00671","type":"journal-article","created":{"date-parts":[[2024,6,14]],"date-time":"2024-06-14T17:52:25Z","timestamp":1718387545000},"page":"786-802","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":9,"title":["Beyond Boundaries: A Human-like Approach for Question Answering over Structured and Unstructured Information Sources"],"prefix":"10.1162","volume":"12","author":[{"given":"Jens","family":"Lehmann","sequence":"first","affiliation":[{"name":"Amazon, Germany jlehmnn@amazon.com"},{"name":"ScaDS.AI \/ TU Dresden, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dhananjay","family":"Bhandiwad","sequence":"additional","affiliation":[{"name":"ScaDS.AI \/ TU Dresden, Germany dhananjay.bhandiwad@tu-dresden.de"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Preetam","family":"Gattogi","sequence":"additional","affiliation":[{"name":"ScaDS.AI \/ TU Dresden, Germany preetam.gattogi@tu-dresden.de"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sahar","family":"Vahdati","sequence":"additional","affiliation":[{"name":"ScaDS.AI \/ TU Dresden, Germany sahar.vahdati@tu-dresden.de"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2024,6,11]]},"reference":[{"key":"2024061417515237800_bib1","doi-asserted-by":"publisher","first-page":"10038","DOI":"10.18653\/v1\/2023.acl-long.558","article-title":"Direct fact retrieval from knowledge graphs without entity linking","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Baek","year":"2023"},{"key":"2024061417515237800_bib2","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1109\/WI.2018.00-54","article-title":"Hybrid question answering using heuristic methods and linked data schema","volume-title":"2018 IEEE\/WIC\/ACM International Conference on Web Intelligence (WI)","author":"Bahmid","year":"2018"},{"key":"2024061417515237800_bib3","first-page":"2206","article-title":"Improving language models by retrieving from trillions of tokens","volume-title":"International Conference on Machine Learning","author":"Borgeaud","year":"2022"},{"key":"2024061417515237800_bib4","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024061417515237800_bib5","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1145\/3477495.3531815","article-title":"Conversational question answering on heterogeneous sources","volume-title":"Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Christmann","year":"2022"},{"key":"2024061417515237800_bib6","article-title":"Compmix: A benchmark for heterogeneous question answering","author":"Christmann","year":"2023","journal-title":"arXiv preprint arXiv:2306.12235"},{"key":"2024061417515237800_bib7","doi-asserted-by":"publisher","first-page":"643","DOI":"10.1145\/3539618.3591682","article-title":"Explainable conversational question answering over heterogeneous sources via iterative graph neural networks","volume-title":"Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Christmann","year":"2023"},{"key":"2024061417515237800_bib8","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.81","volume-title":"Successive prompting for decomposing complex questions","author":"Dua","year":"2022"},{"key":"2024061417515237800_bib9","first-page":"3929","article-title":"Retrieval augmented language model pre-training","volume-title":"International Conference on Machine Learning","author":"Guu","year":"2020"},{"key":"2024061417515237800_bib10","article-title":"Thought cloning: Learning to think while acting by imitating human thinking","volume":"36","author":"Shengran","year":"2024","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024061417515237800_bib11","doi-asserted-by":"publisher","first-page":"6769","DOI":"10.18653\/v1\/2020.emnlp-main.550","article-title":"Dense passage retrieval for open-domain question answering","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Karpukhin","year":"2020"},{"key":"2024061417515237800_bib12","article-title":"Submodlib: A submodular optimization library","author":"Kaushal","year":"2022","journal-title":"arXiv preprint arXiv:2202.10680"},{"key":"2024061417515237800_bib13","article-title":"Demonstrate- search-predict: Composing retrieval and language models for knowledge-intensive nlp","author":"Khattab","year":"2022","journal-title":"arXiv preprint arXiv:2212.14024"},{"key":"2024061417515237800_bib14","article-title":"Dspy: Compiling declarative language model calls into self-improving pipelines","author":"Khattab","year":"2023","journal-title":"arXiv preprint arXiv:2310.03714"},{"key":"2024061417515237800_bib15","article-title":"Decomposed prompting: A modular approach for solving complex tasks","author":"Khot","year":"2022","journal-title":"arXiv preprint arXiv:2210.02406"},{"issue":"2\u20133","key":"2024061417515237800_bib16","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1561\/9781601986290","article-title":"Determinantal point processes for machine learning","volume":"5","author":"Kulesza","year":"2012","journal-title":"Foundations and Trends in Machine Learning"},{"key":"2024061417515237800_bib17","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1162\/tacl_a_00276","article-title":"Natural questions: A benchmark for question answering research","volume":"7","author":"Kwiatkowski","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024061417515237800_bib18","doi-asserted-by":"publisher","DOI":"10.3233\/FAIA230411","article-title":"Language models as controlled natural language semantic parsers for knowledge graph question answering","volume-title":"European Conference on Artificial Intelligence (ECAI)","author":"Lehmann","year":"2023"},{"key":"2024061417515237800_bib19","article-title":"Let\u2019s verify step by step","author":"Lightman","year":"2023","journal-title":"arXiv preprint arXiv:2305.20050"},{"key":"2024061417515237800_bib20","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00638","article-title":"Lost in the middle: How language models use long contexts","volume":"12","author":"Liu","year":"2024","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024061417515237800_bib21","article-title":"Chameleon: Plug-and-play compositional reasoning with large language models","volume":"36","author":"Pan","year":"2024","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024061417515237800_bib22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1147","article-title":"Key-value memory networks for directly reading documents","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Miller","year":"2016"},{"key":"2024061417515237800_bib23","article-title":"Webgpt: Browser-assisted question-answering with human feedback","author":"Nakano","year":"2021","journal-title":"arXiv preprint arXiv:2112.09332"},{"key":"2024061417515237800_bib24","doi-asserted-by":"publisher","first-page":"1535","DOI":"10.18653\/v1\/2022.findings-naacl.115","article-title":"Unik-qa: Unified representations of structured and unstructured knowledge for open-domain question answering","volume-title":"Findings of the Association for Computational Linguistics: NAACL 2022","author":"Oguz","year":"2022"},{"key":"2024061417515237800_bib25","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1109\/ICSC52841.2022.00045","article-title":"Qald-9-plus: A multilingual dataset for question answering over dbpedia and wikidata translated by native speakers","volume-title":"2022 IEEE 16th International Conference on Semantic Computing (ICSC)","author":"Perevalov","year":"2022"},{"key":"2024061417515237800_bib26","article-title":"Uniqorn: Unified question answering over rdf knowledge graphs and natural language text","author":"Pramanik","year":"2023","journal-title":"arXiv preprint arXiv:2108.08614"},{"key":"2024061417515237800_bib27","doi-asserted-by":"publisher","first-page":"5418","DOI":"10.18653\/v1\/2020.emnlp-main.437","article-title":"How much knowledge can you pack into the parameters of a language model?","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Roberts","year":"2020"},{"key":"2024061417515237800_bib28","doi-asserted-by":"publisher","first-page":"4193","DOI":"10.18653\/v1\/2021.emnlp-main.345","article-title":"End-to-end entity resolution and question answering using differentiable knowledge graphs","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Saffari","year":"2021"},{"key":"2024061417515237800_bib29","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1145\/2911451.2911536","article-title":"When a knowledge base is not enough: Question answering over knowledge bases with external text data","volume-title":"Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval","author":"Savenkov","year":"2016"},{"key":"2024061417515237800_bib30","doi-asserted-by":"publisher","first-page":"324","DOI":"10.1007\/s10791-018-9348-8","article-title":"Neural architecture for question answering using a knowledge graph and web corpus","volume":"22","author":"Sawant","year":"2019","journal-title":"Information Retrieval Journal"},{"key":"2024061417515237800_bib31","doi-asserted-by":"publisher","first-page":"4498","DOI":"10.18653\/v1\/2020.acl-main.412","article-title":"Improving multi-hop question answering over knowledge graphs using knowledge base embeddings","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Saxena","year":"2020"},{"key":"2024061417515237800_bib32","first-page":"1604","article-title":"Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering","volume-title":"Proceedings of the 29th International Conference on Computational Linguistics","author":"Sen","year":"2022"},{"key":"2024061417515237800_bib33","doi-asserted-by":"publisher","first-page":"2380","DOI":"10.18653\/v1\/D19-1242","article-title":"Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Sun","year":"2019"},{"key":"2024061417515237800_bib34","doi-asserted-by":"publisher","first-page":"4231","DOI":"10.18653\/v1\/D18-1455","article-title":"Open domain question answering using early fusion of knowledge bases and text","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Sun","year":"2018"},{"key":"2024061417515237800_bib35","article-title":"Head-to-tail: How knowledgeable are large language models (llm)? aka will llms replace knowledge graphs?","author":"Sun","year":"2023","journal-title":"arXiv preprint arXiv:2308.10168"},{"key":"2024061417515237800_bib36","article-title":"Lamda: Language models for dialog applications","author":"Thoppilan","year":"2022","journal-title":"arXiv preprint arXiv:2201.08239"},{"key":"2024061417515237800_bib37","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.557","article-title":"Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions","author":"Trivedi","year":"2023"},{"key":"2024061417515237800_bib38","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1007\/978-3-319-18818-8_22","article-title":"Hawk\u2013hybrid question answering using linked data","volume-title":"The Semantic Web. Latest Advances and New Domains: 12th European Semantic Web Conference, ESWC 2015, Portoroz, Slovenia, May 31\u2013June 4, 2015. Proceedings 12","author":"Usbeck","year":"2015"},{"issue":"10","key":"2024061417515237800_bib39","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1145\/2629489","article-title":"Wikidata: A free collaborative knowledgebase","volume":"57","author":"Vrande\u010di\u0107","year":"2014","journal-title":"Communications of the ACM"},{"key":"2024061417515237800_bib40","doi-asserted-by":"publisher","first-page":"6687","DOI":"10.18653\/v1\/2022.findings-emnlp.498","article-title":"Muger2: Multi-granularity evidence retrieval and reasoning for hybrid question answering","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2022","author":"Wang","year":"2022"},{"key":"2024061417515237800_bib41","article-title":"Larger language models do in-context learning differently","author":"Wei","year":"2023","journal-title":"arXiv preprint arXiv:2303.03846"},{"key":"2024061417515237800_bib42","article-title":"Rewoo: Decoupling reasoning from observations for efficient augmented language models","author":"Binfeng","year":"2023","journal-title":"arXiv preprint arXiv:2305.18323"},{"key":"2024061417515237800_bib43","first-page":"2397","article-title":"Hybrid question answering over knowledge base and free text","volume-title":"Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers","author":"Kun","year":"2016"},{"key":"2024061417515237800_bib44","doi-asserted-by":"publisher","first-page":"2326","DOI":"10.18653\/v1\/P16-1220","article-title":"Question answering on freebase via relation extraction and textual evidence","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Kun","year":"2016"},{"key":"2024061417515237800_bib45","article-title":"React: Synergizing reasoning and acting in language models","volume-title":"The Eleventh International Conference on Learning Representations","author":"Yao","year":"2023"},{"key":"2024061417515237800_bib46","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0288453","article-title":"Distinguishing chatgpt(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis","author":"Zaitsu","year":"2023","journal-title":"PLOS ONE"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00671\/2383373\/tacl_a_00671.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00671\/2383373\/tacl_a_00671.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,14]],"date-time":"2024-06-14T17:52:41Z","timestamp":1718387561000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00671\/122240\/Beyond-Boundaries-A-Human-like-Approach-for"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":46,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00671","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}