{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T20:09:37Z","timestamp":1761163777315,"version":"build-2065373602"},"reference-count":31,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2019,2,1]],"date-time":"2019-02-01T00:00:00Z","timestamp":1548979200000},"content-version":"vor","delay-in-days":396,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["asistdl.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Proc. Assoc. Info. Sci. Tech."],"published-print":{"date-parts":[[2018,1]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:p>As an integral part of research activities, scholars frequently interact with bibliographic information retrieval systems to acquire scholarly information. With the dramatic increase of scientific literature, there is the pressing need of building effective and efficient bibliographic information retrieval systems that support more granular and complex bibliographic information needs. Investigating users' interaction with these systems plays an important role in understanding the systems' features and usability issues, which contributes to better system design and implementation. In this paper, we evaluate three interactive bibliographic information retrieval systems proposed in our previous studies using a user\u2010centered approach, including form\u2010, natural language\u2010, and visual graph\u2010based systems. This study recruits 20 participants to evaluate the three systems from the aspects of success rate, search time, query size, usefulness, ease of use, ease of learning, and satisfaction. Results showed that: the form\u2010based system needed the least time to formulate simple bibliographic queries and allowed the participants to make the least mistakes; the natural language\u2010based system needed the least time to formulate high\u2010complexity bibliographic queries and was rated as the most useful, easy\u2010to\u2010use, and easy\u2010to\u2010learn system; and the visual graph\u2010based system's strengths lie in the support of complex queries as shown that it got better ratings of usefulness and satisfaction by the high\u2010complexity task group.<\/jats:p>","DOI":"10.1002\/pra2.2018.14505501068","type":"journal-article","created":{"date-parts":[[2019,2,1]],"date-time":"2019-02-01T16:52:19Z","timestamp":1549039939000},"page":"628-637","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Evaluating interactive bibliographic information retrieval systems: A user\u2010centered approach"],"prefix":"10.1002","volume":"55","author":[{"given":"Yongjun","family":"Zhu","sequence":"first","affiliation":[{"name":"Department of Library and Information Science Sungkyunkwan University South Korea"}]},{"given":"Meen Chul","family":"Kim","sequence":"additional","affiliation":[{"name":"College of Computing and Informatics Drexel University United States"}]},{"given":"Erjia","family":"Yan","sequence":"additional","affiliation":[{"name":"College of Computing and Informatics Drexel University United States"}]}],"member":"311","published-online":{"date-parts":[[2019,2]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.21300"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2011.03.002"},{"issue":"3","key":"e_1_2_8_4_1","first-page":"152","article-title":"The IIR evaluation model: A framework for evaluation of interactive information retrieval systems","volume":"8","author":"Borlund P.","year":"2003","journal-title":"Information Research\u2010an International Electronic Journal"},{"key":"e_1_2_8_5_1","doi-asserted-by":"publisher","DOI":"10.1108\/JD-06-2015-0068"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1108\/EUM0000000007198"},{"key":"e_1_2_8_7_1","doi-asserted-by":"crossref","unstructured":"Borlund P. &Schneider J.(2010)Reconsideration of the simulated work task situation: A context instrument for evaluation of information retrieval interaction. Paper presented at the Third Information Interaction in Context Symposium (IIiX) 155\u2013164.","DOI":"10.1145\/1840784.1840808"},{"key":"e_1_2_8_8_1","doi-asserted-by":"publisher","DOI":"10.1108\/JD-04-2015-0051"},{"key":"e_1_2_8_9_1","unstructured":"de Ribaupierre H.(2014).Precise information retrieval in semantic scientific digital libraries (Doctoral dissertation). University of Geneva Geneva Stitzerland."},{"key":"e_1_2_8_10_1","doi-asserted-by":"crossref","unstructured":"Hall M. M. &Toms E.(2013).Building a Common Framework for IIR Evaluation. Paper presented at the Forth International Conference of the Cross\u2010Language Evaluation Forum for European Language 17\u201328.","DOI":"10.1007\/978-3-642-40802-1_3"},{"key":"e_1_2_8_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2005.06.002"},{"key":"e_1_2_8_12_1","unstructured":"ISO 9241\u201011(1998).Ergonomic Requirements of Office Work with Visual Display Terminals (VDTs): Part II: Guidance on usability. International Organization for Standardization"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2007.02.006"},{"issue":"1","key":"e_1_2_8_14_1","first-page":"1","article-title":"Methods for evaluating interactive information retrieval systems with users","volume":"3","author":"Kelly D.","year":"2009","journal-title":"Foundations and Trends in Information Retrieval"},{"key":"e_1_2_8_15_1","doi-asserted-by":"crossref","unstructured":"Kelly D. Arguello J. Edwards A. &Wu W. C.(2015).Development and evaluation of search tasks for IIR experiments using a cognitive complexity framework. Paper presented at the ACM SIGIR International Conference on the Theory of Information Retrieval (IC\u2010TIR) 101\u2010110","DOI":"10.1145\/2808194.2809465"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.22799"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20977"},{"key":"e_1_2_8_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2008.07.005"},{"key":"e_1_2_8_19_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.21359"},{"key":"e_1_2_8_20_1","doi-asserted-by":"crossref","unstructured":"Liu J. Cole M. Liu C. Bierig R. Gwizdka J. Belkin N. Zhang J. &Zhang X.(2010).Search behaviors in different task types. Paper presented at the Joint Conference on Digital Libraries (JCDL) 69\u201378.","DOI":"10.1145\/1816123.1816134"},{"issue":"2","key":"e_1_2_8_21_1","first-page":"3","article-title":"Measuring usability with the USE questionnaire","volume":"8","author":"Lund A. M.","year":"2011","journal-title":"Usability Interface"},{"key":"e_1_2_8_22_1","doi-asserted-by":"publisher","DOI":"10.1177\/0961000616656234"},{"key":"e_1_2_8_23_1","doi-asserted-by":"crossref","unstructured":"Renaud G. &Azzopardi L.(2012).SCAMP: A tool for conducting interactive information retrieval experiments. Paper presented at the Fourth Information Interaction in Context Symposium (IIiX) 286\u2013289.","DOI":"10.1145\/2362724.2362776"},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0271(70)90011-2"},{"key":"e_1_2_8_25_1","unstructured":"Smucker M. D. &Jethani C. P.(2010).Impact of retrieval precision on perceived difficulty and other user measures. Paper presented at the Fourth Symposium on Human\u2010Computer Interaction and Information Retrieval (HCIR) 20\u201323."},{"key":"e_1_2_8_26_1","doi-asserted-by":"crossref","unstructured":"Wei X. Zhang Y. &Gwizdka J.(2014).YASFIIRE: Yet another system for IIR evaluation. Paper presented at the Fifth Information Interaction in Context Symposium (IIiX).316\u2013319.","DOI":"10.1145\/2637002.2637051"},{"key":"e_1_2_8_27_1","doi-asserted-by":"crossref","unstructured":"Wildemuth B. &Freund L.(2012).Assigning search tasks designed to elicit exploratory search behaviors. Paper presented at the Sixth Symposium on Human\u2010Computer Interaction and Information Retrieval (HCIR) 1\u201310.","DOI":"10.1145\/2391224.2391228"},{"key":"e_1_2_8_28_1","doi-asserted-by":"publisher","DOI":"10.1108\/JD-03-2014-0056"},{"issue":"2","key":"e_1_2_8_29_1","first-page":"267","article-title":"Crowdsourcing interactions","volume":"16","author":"Zuccon G.","year":"2013","journal-title":"Using crowdsourcing for evaluating interactive information retrieval systems. Information Retrieval"},{"key":"e_1_2_8_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2016.09.005"},{"key":"e_1_2_8_31_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23677"},{"key":"e_1_2_8_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.datak.2017.06.006"}],"container-title":["Proceedings of the Association for Information Science and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fpra2.2018.14505501068","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/pra2.2018.14505501068","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/pra2.2018.14505501068","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/pra2.2018.14505501068","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,21]],"date-time":"2025-10-21T17:39:23Z","timestamp":1761068363000},"score":1,"resource":{"primary":{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/10.1002\/pra2.2018.14505501068"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,1]]},"references-count":31,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2018,1]]}},"alternative-id":["10.1002\/pra2.2018.14505501068"],"URL":"https:\/\/doi.org\/10.1002\/pra2.2018.14505501068","archive":["Portico"],"relation":{},"ISSN":["2373-9231","2373-9231"],"issn-type":[{"type":"print","value":"2373-9231"},{"type":"electronic","value":"2373-9231"}],"subject":[],"published":{"date-parts":[[2018,1]]},"assertion":[{"value":"2019-02-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}