{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T00:48:50Z","timestamp":1769042930828,"version":"3.49.0"},"reference-count":28,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2001,7,1]],"date-time":"2001-07-01T00:00:00Z","timestamp":993945600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2001,7]]},"abstract":"<jats:p>The wealth of information on the web makes it an attractive resource for seeking quick answers to simple, factual questions such as \u201cwho was the first American in space?\u201d or \u201cwhat is the second tallest mountain in the world?\u201d Yet today's most advanced web search services (e.g., Google and AskJeeves) make it surprisingly tedious to locate answers to such questions. In this paper, we extend question-answering techniques, first studied in the information retrieval literature, to the web and experimentally evaluate their performance.First we introduce Mulder, which we believe to be the first general-purpose, fully-automated question-answering system available on the web. Second, we describe Mulder's architecture, which relies on multiple search-engine queries, natural-language parsing, and a novel voting procedure to yield reliable answers coupled with high recall. Finally, we compare Mulder's performance to that of Google and AskJeeves on questions drawn from the TREC-8 question answering track. We find that Mulder's recall is more than a factor of three higher than that of AskJeeves. In addition, we find that Google requires 6.6 times as much user effort to achieve the same level of recall as Mulder.<\/jats:p>","DOI":"10.1145\/502115.502117","type":"journal-article","created":{"date-parts":[[2002,7,27]],"date-time":"2002-07-27T11:29:00Z","timestamp":1027769340000},"page":"242-262","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":133,"title":["Scaling question answering to the web"],"prefix":"10.1145","volume":"19","author":[{"given":"Cody","family":"Kwok","sequence":"first","affiliation":[{"name":"University of Washington, Seattle, WA"}]},{"given":"Oren","family":"Etzioni","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA"}]},{"given":"Daniel S.","family":"Weld","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA"}]}],"member":"320","published-online":{"date-parts":[[2001,7]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"An Introduction to the Principles of Transformational Syntax","author":"AKMAJIAN A.","unstructured":"AKMAJIAN , A. AND HENY , F. 1975. An Introduction to the Principles of Transformational Syntax . MIT Press , Cambridge, Mass . AKMAJIAN,A.AND HENY, F. 1975. An Introduction to the Principles of Transformational Syntax. MIT Press, Cambridge, Mass."},{"key":"e_1_2_1_2_1","volume-title":"PC-KIMMO: A two-level processor for morphological analysis","author":"ANTWORTH E. L.","unstructured":"ANTWORTH , E. L. 1990. PC-KIMMO: A two-level processor for morphological analysis . Summer Institute of Linguistics , Dallas, Tex . ANTWORTH, E. L. 1990. PC-KIMMO: A two-level processor for morphological analysis. Summer Institute of Linguistics, Dallas, Tex."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 7th Message Understanding Conference. Morgan Kaufmann","author":"ARPA.","year":"1998","unstructured":"ARPA. 1998 . Proceedings of the 7th Message Understanding Conference. Morgan Kaufmann , San Francisco, Calif. ARPA. 1998. Proceedings of the 7th Message Understanding Conference. Morgan Kaufmann, San Francisco, Calif."},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","first-page":"194","DOI":"10.3115\/974557.974586","volume-title":"Proceedings of the Fifth Conference on Applied Natural Language Processing","author":"BIKEL D.","year":"1997","unstructured":"BIKEL , D. , MILLER , S. , SCHWARTZ , R. , AND WEISCHEDEL , R. 1997 . Nymble: A high-performance learning name finder . In Proceedings of the Fifth Conference on Applied Natural Language Processing (1997), 194 - 201 . 10.3115\/974557.974586 BIKEL, D., MILLER, S., SCHWARTZ, R., AND WEISCHEDEL, R. 1997. Nymble: A high-performance learning name finder. In Proceedings of the Fifth Conference on Applied Natural Language Processing (1997), 194-201. 10.3115\/974557.974586"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the Seventh International World Wide Web Conference (www-7","author":"BRIN S.","year":"1998","unstructured":"BRIN , S. AND PAGE , L. 1998 . The anatomy of a large-scale hypertextual web search engine . In Proceedings of the Seventh International World Wide Web Conference (www-7 , Brisborne, Australia, Apr. 14-18). BRIN,S.AND PAGE, L. 1998. The anatomy of a large-scale hypertextual web search engine. In Proceedings of the Seventh International World Wide Web Conference (www-7, Brisborne, Australia, Apr. 14-18)."},{"key":"e_1_2_1_6_1","volume-title":"NIST Special Publication 500-225: The Third Text REtrieval Conference (TREC-3)","author":"BUCKLEY C.","year":"1995","unstructured":"BUCKLEY , C. , SALTON , G. , ALLAN , J. , AND SINGHAL , A. 1995. Automatic query expansion using SMART: TREC 3 . In NIST Special Publication 500-225: The Third Text REtrieval Conference (TREC-3) ( 1995 ), Department of Commerce , National Institute of Standards and Technology, 69-80. BUCKLEY, C., SALTON, G., ALLAN,J.,AND SINGHAL, A. 1995. Automatic query expansion using SMART: TREC 3. In NIST Special Publication 500-225: The Third Text REtrieval Conference (TREC-3) (1995), Department of Commerce, National Institute of Standards and Technology, 69-80."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of 8th International World Wide Web Conference (WWW8).","author":"CHAKRABARTI S.","year":"1999","unstructured":"CHAKRABARTI , S. , BERG , M , VAN DER ., AND DOM , B. 1999 . Focused crawling: a new approach to topicspecific Web resource discovery . In Proceedings of 8th International World Wide Web Conference (WWW8). CHAKRABARTI, S., BERG,M,VAN DER., AND DOM, B. 1999. Focused crawling: a new approach to topicspecific Web resource discovery. In Proceedings of 8th International World Wide Web Conference (WWW8)."},{"key":"e_1_2_1_9_1","first-page":"4","article-title":"Statistical techniques for natural language parsing","volume":"18","author":"CHARNIAK E.","year":"1997","unstructured":"CHARNIAK , E. 1997 . Statistical techniques for natural language parsing . AI Magazine 18 , 4 (Winter). CHARNIAK, E. 1997. Statistical techniques for natural language parsing. AI Magazine 18,4 (Winter).","journal-title":"AI Magazine"},{"key":"e_1_2_1_10_1","unstructured":"CHARNIAK E. 1999. A Maximum-Entropy-Inspired Parser. Tech. Rep. CS-99-12 (Aug.) Brown University Computer Science Dept.   CHARNIAK E. 1999. A Maximum-Entropy-Inspired Parser. Tech. Rep. CS-99-12 (Aug.) Brown University Computer Science Dept."},{"key":"e_1_2_1_12_1","volume-title":"Aspects of a Theory of Syntax","author":"CHOMSKY N.","unstructured":"CHOMSKY , N. 1965. Aspects of a Theory of Syntax . MIT Press , Cambridge, Mass . CHOMSKY, N. 1965. Aspects of a Theory of Syntax. MIT Press, Cambridge, Mass."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 34th Annual Meeting of the ACL (Santa Cruz, Calif ). 10","author":"COLLINS M. J.","year":"1996","unstructured":"COLLINS , M. J. 1996 . A New Statistical Parser Based on Bigram Lexical Dependencies . In Proceedings of the 34th Annual Meeting of the ACL (Santa Cruz, Calif ). 10 .3115\/981863.981888 COLLINS, M. J. 1996. A New Statistical Parser Based on Bigram Lexical Dependencies. In Proceedings of the 34th Annual Meeting of the ACL (Santa Cruz, Calif ). 10.3115\/981863.981888"},{"key":"e_1_2_1_14_1","volume-title":"Summer","author":"ETZIONI O.","year":"1997","unstructured":"ETZIONI , O. 1997. Moving up the information food chain: softbots as information carnivores. AI Maga., special issue , Summer 1997 . ETZIONI, O. 1997. Moving up the information food chain: softbots as information carnivores. AI Maga., special issue, Summer 1997."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the Fourth International Workshop on Parsing Technologies (Prague, Sept.).","author":"GRINBERG D.","year":"1995","unstructured":"GRINBERG , D. , LAFFERTY , J. , AND SLEATOR , D. 1995 . ARobust Parsing Algorithm for Link Grammars . In Proceedings of the Fourth International Workshop on Parsing Technologies (Prague, Sept.). GRINBERG, D., LAFFERTY,J.,AND SLEATOR, D. 1995. ARobust Parsing Algorithm for Link Grammars. In Proceedings of the Fourth International Workshop on Parsing Technologies (Prague, Sept.)."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of COLING-2000 (Saarbruken Germany, Aug.). 10","author":"HARABAGIU S.","year":"2000","unstructured":"HARABAGIU , S. , MAIORANO , S. , AND PASCA , M. 2000 . Experiments with Open-Domain Textual Question Answering . In Proceedings of COLING-2000 (Saarbruken Germany, Aug.). 10 .3115\/990820.990863 HARABAGIU, S., MAIORANO,S.,AND PASCA, M. 2000. Experiments with Open-Domain Textual Question Answering. In Proceedings of COLING-2000 (Saarbruken Germany, Aug.). 10.3115\/990820.990863"},{"key":"e_1_2_1_17_1","unstructured":"KATZ B. 1997. From Sentence Processing to Information Access on the World Wide Web. In Natural Language Processing for the World Wide Web: Papers from the 1997 AAAI Spring Symposium 77-94.  KATZ B. 1997. From Sentence Processing to Information Access on the World Wide Web. In Natural Language Processing for the World Wide Web: Papers from the 1997 AAAI Spring Symposium 77-94."},{"key":"e_1_2_1_18_1","first-page":"181","volume-title":"Proceedings of the 16th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval","author":"KUPIEC J.","year":"1993","unstructured":"KUPIEC , J. 1993 . MURAX: A Robust Linguistic Approach for Question Answering Using an On-Line Encyclopedia . In Proceedings of the 16th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval ( Pittsburgh, Pa. June 27-July 1). R. Korfhage, E. M. Rasmussen, and P. Willett, Eds., ACM, New York , 181 - 190 . 10.1145\/160688.160717 KUPIEC, J. 1993. MURAX: A Robust Linguistic Approach for Question Answering Using an On-Line Encyclopedia. In Proceedings of the 16th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval (Pittsburgh, Pa. June 27-July 1). R. Korfhage, E. M. Rasmussen, and P. Willett, Eds., ACM, New York, 181-190. 10.1145\/160688.160717"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 8th Text Retrieval Conference (TREC-8). (National Institute of Standards and Technology, Gaithersburg MD), 349-356","author":"LITKOWSKI K.","year":"1999","unstructured":"LITKOWSKI , K. 1999 . Question-Answering Using Semantic Relation Triples . In Proceedings of the 8th Text Retrieval Conference (TREC-8). (National Institute of Standards and Technology, Gaithersburg MD), 349-356 . LITKOWSKI, K. 1999. Question-Answering Using Semantic Relation Triples. In Proceedings of the 8th Text Retrieval Conference (TREC-8). (National Institute of Standards and Technology, Gaithersburg MD), 349-356."},{"key":"e_1_2_1_20_1","first-page":"313","article-title":"Building a Large Annotated Corpus of English: The Penn Treebank","volume":"19","author":"MARCUS M.P.","year":"1993","unstructured":"MARCUS , M.P. , MARCINKIEWICZ , M.A. , AND SANTORINI , B. 1993 . Building a Large Annotated Corpus of English: The Penn Treebank . Computational Linguistics 19 , 313 - 330 . MARCUS,M.P.,MARCINKIEWICZ,M.A.,AND SANTORINI, B. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics 19, 313-330.","journal-title":"Computational Linguistics"},{"issue":"4","key":"e_1_2_1_21_1","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1093\/ijl\/3.4.235","article-title":"WordNet: An on-line lexical database","volume":"3","author":"MILLER G.","year":"1991","unstructured":"MILLER , G. 1991 . WordNet: An on-line lexical database . International Journal of Lexicography 3 , 4 , 235 - 312 . MILLER, G. 1991. WordNet: An on-line lexical database. International Journal of Lexicography 3, 4, 235-312.","journal-title":"International Journal of Lexicography"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 8th Text Retrieval Conference (TREC-8). (National Institute of Standards and Technology, Gaithersburg MD), 399-411","author":"RADEV D. R.","year":"1999","unstructured":"RADEV , D. R. , PRAGER , J. , AND SAMN , V. 1999 . The Use of Predictive Annotation for Question Answering in TREC8 . In Proceedings of the 8th Text Retrieval Conference (TREC-8). (National Institute of Standards and Technology, Gaithersburg MD), 399-411 . RADEV, D. R., PRAGER,J.,AND SAMN, V. 1999. The Use of Predictive Annotation for Question Answering in TREC8. In Proceedings of the 8th Text Retrieval Conference (TREC-8). (National Institute of Standards and Technology, Gaithersburg MD), 399-411."},{"key":"e_1_2_1_23_1","unstructured":"SNEIDERS E. 1999. Automated FAQ Answering: Continued Experience with Shallow Language Understanding. In Question Answering Systems. Papers from the 1999 AAAI Fall Symposium.  SNEIDERS E. 1999. Automated FAQ Answering: Continued Experience with Shallow Language Understanding. In Question Answering Systems. Papers from the 1999 AAAI Fall Symposium."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 8th Text Retrieval Conference (TREC-8). (National Institute of Standards and Technology, Gaithersburg MD), 185-196","author":"SRIHARI R.","year":"1999","unstructured":"SRIHARI , R. AND LI , W. 1999 . Information Extraction Supported Question Answering . In Proceedings of the 8th Text Retrieval Conference (TREC-8). (National Institute of Standards and Technology, Gaithersburg MD), 185-196 . SRIHARI,R.AND LI, W. 1999. Information Extraction Supported Question Answering. In Proceedings of the 8th Text Retrieval Conference (TREC-8). (National Institute of Standards and Technology, Gaithersburg MD), 185-196."},{"key":"e_1_2_1_25_1","unstructured":"TAYLOR S. E. FRANCKENPOHL H. AND PETTE J. L. 1960. Grade level norms for the component of the fundamental reading skill. EDL Information and Research Bulletin No. 3. Huntington N.Y.  TAYLOR S. E. FRANCKENPOHL H. AND PETTE J. L. 1960. Grade level norms for the component of the fundamental reading skill. EDL Information and Research Bulletin No. 3. Huntington N.Y."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of ACM SIGIR","author":"VOORHEES E.","year":"1994","unstructured":"VOORHEES , E. 1994 . Query expansion using lexical-semantic relations . In Proceedings of ACM SIGIR ( Dublin, Ireland). VOORHEES, E. 1994. Query expansion using lexical-semantic relations. In Proceedings of ACM SIGIR (Dublin, Ireland)."},{"key":"e_1_2_1_27_1","first-page":"77","volume-title":"The TREC-8 Question Answering Track Evaluation","author":"VOORHEES E.","unstructured":"VOORHEES , E. AND TICE , D. 1999. The TREC-8 Question Answering Track Evaluation , pp. 77 - 82 . Department of Commerce , National Institute of Standards and Technology. VOORHEES,E.AND TICE, D. 1999. The TREC-8 Question Answering Track Evaluation, pp. 77-82. Department of Commerce, National Institute of Standards and Technology."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the Twenty-Third Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM","author":"VOORHEES E.","year":"2000","unstructured":"VOORHEES , E. AND TICE , D. 2000 . Building a question answering test collection . In Proceedings of the Twenty-Third Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM , New York. 10.1145\/345508.345577 VOORHEES,E.AND TICE, D. 2000. Building a question answering test collection. In Proceedings of the Twenty-Third Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York. 10.1145\/345508.345577"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/0169-7552(95)00101-2"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the Eighth Int. WWW Conference.","author":"ZAMIR O.","year":"1999","unstructured":"ZAMIR , O. AND ETZIONI , O. 1999 . A Dynamic Clustering Interface to Web Search Results . In Proceedings of the Eighth Int. WWW Conference. ZAMIR,O.AND ETZIONI, O. 1999. A Dynamic Clustering Interface to Web Search Results. In Proceedings of the Eighth Int. WWW Conference."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/502115.502117","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/502115.502117","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:15:13Z","timestamp":1750281313000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/502115.502117"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2001,7]]},"references-count":28,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2001,7]]}},"alternative-id":["10.1145\/502115.502117"],"URL":"https:\/\/doi.org\/10.1145\/502115.502117","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2001,7]]},"assertion":[{"value":"2001-07-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}