{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:04:55Z","timestamp":1755839095675,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":28,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,3,14]],"date-time":"2021-03-14T00:00:00Z","timestamp":1615680000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["DP180102687"],"award-info":[{"award-number":["DP180102687"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,3,14]]},"DOI":"10.1145\/3406522.3446028","type":"proceedings-article","created":{"date-parts":[[2021,2,27]],"date-time":"2021-02-27T17:18:47Z","timestamp":1614446327000},"page":"75-84","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Quantifying Human-Perceived Answer Utility in Non-factoid Question Answering"],"prefix":"10.1145","author":[{"given":"B. Barla","family":"Cambazoglu","sequence":"first","affiliation":[{"name":"RMIT University, Melbourne, VIC, Australia"}]},{"given":"Valeriia","family":"Baranova","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, VIC, Australia"}]},{"given":"Falk","family":"Scholer","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, VIC, Australia"}]},{"given":"Mark","family":"Sanderson","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, VIC, Australia"}]},{"given":"Leila","family":"Tavakoli","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, VIC, Australia"}]},{"given":"Bruce","family":"Croft","sequence":"additional","affiliation":[{"name":"University of Massachusetts, Amherst, MA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,3,14]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1178"},{"key":"e_1_3_2_1_2_1","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. Association for Computational Linguistics","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie . 2005 . METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. Association for Computational Linguistics , Ann Arbor, Michigan, 65--72. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. Association for Computational Linguistics, Ann Arbor, Michigan, 65--72."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10286"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1177\/0165551511415584"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1002\/pra2.2015.145052010088"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593702.2593703"},{"volume-title":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. ACM","author":"Hashemi Helia","key":"e_1_3_2_1_7_1","unstructured":"Helia Hashemi , Hamed Zamani , and W. Bruce Croft . 2019. Performance Prediction for Non-Factoid Question Answering . In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. ACM , New York, NY, USA, 55--58. Helia Hashemi, Hamed Zamani, and W. Bruce Croft. 2019. Performance Prediction for Non-Factoid Question Answering. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. ACM, New York, NY, USA, 55--58."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2124295.2124371"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2910896.2910900"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.25300\/MISQ\/2019\/14160"},{"key":"e_1_3_2_1_11_1","volume-title":"ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin . 2004 . ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out . Association for Computational Linguistics , Barcelona, Spain , 74--81. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74--81."},{"key":"e_1_3_2_1_12_1","volume-title":"Proceedings of the Ninth IFIP TC13 International Conference on Human-Computer Interaction.","author":"Lin Jimmy","year":"2003","unstructured":"Jimmy Lin , Dennis Quan , Vineet Sinha , Karun Bakshi , David Huynh , Boris Katz , and David Karger . 2003 . What Makes a Good Answer? The Role of Context in Question Answering . In Proceedings of the Ninth IFIP TC13 International Conference on Human-Computer Interaction. Jimmy Lin, Dennis Quan, Vineet Sinha, Karun Bakshi, David Huynh, Boris Katz, and David Karger. 2003. What Makes a Good Answer? The Role of Context in Question Answering. In Proceedings of the Ninth IFIP TC13 International Conference on Human-Computer Interaction."},{"key":"e_1_3_2_1_13_1","volume-title":"Lorch","author":"Myers Jerome L.","year":"2010","unstructured":"Jerome L. Myers , Arnold D. Well , and Robert F . Lorch . 2010 . Research Design and Statistical Analysis. Routledge . Jerome L. Myers, Arnold D. Well, and Robert F. Lorch. 2010. Research Design and Statistical Analysis. Routledge."},{"key":"e_1_3_2_1_14_1","volume-title":"Sirois","author":"Myers Leann","year":"2006","unstructured":"Leann Myers and Maria J . Sirois . 2006 . Differences between Spearman Correlation Coefficients. In Encyclopedia of Statistical Sciences. John Wiley and Sons . Leann Myers and Maria J. Sirois. 2006. Differences between Spearman Correlation Coefficients. In Encyclopedia of Statistical Sciences. John Wiley and Sons."},{"key":"e_1_3_2_1_15_1","volume-title":"Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches, Tarek Richard Besold, Antoine Bordes, Artur S. d'Avila Garcez, and Greg Wayne (Eds.)","volume":"1773","author":"Nguyen Tri","year":"2016","unstructured":"Tri Nguyen , Mir Rosenberg , Xia Song , Jianfeng Gao , Saurabh Tiwary , Rangan Majumder , and Li Deng . 2016 . MS MARCO: A Human Generated MAchine Reading COmprehension Dataset . In Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches, Tarek Richard Besold, Antoine Bordes, Artur S. d'Avila Garcez, and Greg Wayne (Eds.) , Vol. 1773 . CEUR-WS.org. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches, Tarek Richard Besold, Antoine Bordes, Artur S. d'Avila Garcez, and Greg Wayne (Eds.), Vol. 1773. CEUR-WS.org."},{"key":"e_1_3_2_1_16_1","volume-title":"Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, USA, 311--318","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . 2002 . BLEU: A Method for Automatic Evaluation of Machine Translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, USA, 311--318 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, USA, 311--318."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecns.2014.07.005"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3295750.3298946"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-2124"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"crossref","unstructured":"Tefko Saracevic. 2016. The Notion of Relevance in Information Science: Everybody knows what relevance is. But what is it really? Morgan & Claypool Publishers.  Tefko Saracevic. 2016. The Notion of Relevance in Information Science: Everybody knows what relevance is. But what is it really? Morgan & Claypool Publishers.","DOI":"10.1007\/978-3-031-02302-6"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1002\/pra2.2015.145052010051"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835518"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1496"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/1133031.1133039"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2014.12.038"},{"key":"e_1_3_2_1_26_1","volume-title":"BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations.","author":"Tianyi","year":"2020","unstructured":"Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger , and Yoav Artzi . 2020 . BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations."},{"volume-title":"Multidimensional Relevance Modeling via Psychometrics and Crowdsourcing","author":"Zhang Yinglong","key":"e_1_3_2_1_27_1","unstructured":"Yinglong Zhang , Jin Zhang , Matthew Lease , and Jacek Gwizdka . 2014. Multidimensional Relevance Modeling via Psychometrics and Crowdsourcing . Association for Computing Machinery , New York, NY, USA , 435--444. Yinglong Zhang, Jin Zhang, Matthew Lease, and Jacek Gwizdka. 2014. Multidimensional Relevance Modeling via Psychometrics and Crowdsourcing. Association for Computing Machinery, New York, NY, USA, 435--444."},{"key":"e_1_3_2_1_28_1","volume-title":"Proceedings of the 14th International Conference on Information Quality,, Paul L. Bowen, Ahmed K. Elmagarmid, Hubert \u00d6 sterle, and Kai-Uwe Sattler (Eds.). HPI\/MIT, 264--265","author":"Zhu Zhemin","year":"2009","unstructured":"Zhemin Zhu , Delphine Bernhard , and Iryna Gurevych . 2009 . A Multi-Dimensional Model for Assessing the Quality of Answers in Social Q&A Sites . In Proceedings of the 14th International Conference on Information Quality,, Paul L. Bowen, Ahmed K. Elmagarmid, Hubert \u00d6 sterle, and Kai-Uwe Sattler (Eds.). HPI\/MIT, 264--265 . Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2009. A Multi-Dimensional Model for Assessing the Quality of Answers in Social Q&A Sites. In Proceedings of the 14th International Conference on Information Quality,, Paul L. Bowen, Ahmed K. Elmagarmid, Hubert \u00d6 sterle, and Kai-Uwe Sattler (Eds.). HPI\/MIT, 264--265."}],"event":{"name":"CHIIR '21: ACM SIGIR Conference on Human Information Interaction and Retrieval","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval","SIGCHI ACM Special Interest Group on Computer-Human Interaction"],"location":"Canberra ACT Australia","acronym":"CHIIR '21"},"container-title":["Proceedings of the 2021 Conference on Human Information Interaction and Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3406522.3446028","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3406522.3446028","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:03:19Z","timestamp":1750197799000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3406522.3446028"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,14]]},"references-count":28,"alternative-id":["10.1145\/3406522.3446028","10.1145\/3406522"],"URL":"https:\/\/doi.org\/10.1145\/3406522.3446028","relation":{},"subject":[],"published":{"date-parts":[[2021,3,14]]},"assertion":[{"value":"2021-03-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}