{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T07:14:05Z","timestamp":1774077245831,"version":"3.50.1"},"reference-count":9,"publisher":"Association for Natural Language Processing","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Journal of Natural Language Processing"],"published-print":{"date-parts":[[2026]]},"DOI":"10.5715\/jnlp.33.376","type":"journal-article","created":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T22:13:03Z","timestamp":1773526383000},"page":"376-381","source":"Crossref","is-referenced-by-count":0,"title":["How We Came to \u201cAre Checklists Really Useful for Automatic Evaluation of Generative Tasks?\u201d","Are Checklists Really Useful for Automatic Evaluation of Generative Tasks? \u306b\u81f3\u308b\u7814\u7a76\u904e\u7a0b"],"prefix":"10.5715","volume":"33","author":[{"given":"Momoka","family":"Furuhashi","sequence":"first","affiliation":[{"name":"Tohoku University"},{"name":"NII LLMC"}]}],"member":"3685","reference":[{"key":"1","unstructured":"Cook, J., Rockt\u0101schel, T., Foerster, J., Aumiller, D., and Wang, A. (2024). \u201cTICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation.\u201d <i>arXiv preprint arXiv:2410.03608<\/i>."},{"key":"2","doi-asserted-by":"crossref","unstructured":"Furuhashi, M., Nakayama, K., Kodama, T., and Sugawara, S. (2025). \u201cAre Checklists Really Useful for Automatic Evaluation of Generative Tasks?\u201d In <i>Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing<\/i>, pp. 10641\u201310664, Suzhou, China. Association for Computational Linguistics.","DOI":"10.18653\/v1\/2025.emnlp-main.538"},{"key":"3","doi-asserted-by":"crossref","unstructured":"Hayes, A. F. and Krippendorff, K. (2007). \u201cAnswering the Call for a Standard Reliability Measure for Coding Data.\u201d <i>Communication Methods and Measures<\/i>, 1 (1), pp. 77\u201389.","DOI":"10.1080\/19312450709336664"},{"key":"4","doi-asserted-by":"crossref","unstructured":"Lee, Y., Kim, J., Kim, J., Cho, H., Kang, J., Kang, P., and Kim, N. (2025). \u201cCheckEval: A Reliable LLM-as-a-Judge Framework for Evaluating Text Generation Using Checklists.\u201d In <i>Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing<\/i>, pp. 15782\u201315809, Suzhou, China. Association for Computational Linguistics.","DOI":"10.18653\/v1\/2025.emnlp-main.796"},{"key":"5","unstructured":"Lin, B. Y., Deng, Y., Chandu, K. R., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R. L., and Choi, Y. (2025). \u201cWildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild.\u201d In <i>International Conference on Learning Representations<\/i>."},{"key":"6","unstructured":"OpenAI (2024). \u201cGPT-4 Technical Report.\u201d <i>arXiv preprint arXiv:2303.08774<\/i>."},{"key":"7","doi-asserted-by":"crossref","unstructured":"Qin, Y., Song, K., Hu, Y., Yao, W., Cho, S., Wang, X., Wu, X., Liu, F., Liu, P., and Yu, D. (2024). \u201cInFoBench: Evaluating Instruction Following Ability in Large Language Models.\u201d In <i>Findings of the Association for Computational Linguistics: ACL 2024<\/i>, pp. 13025\u201313048, Bangkok, Thailand. Association for Computational Linguistics.","DOI":"10.18653\/v1\/2024.findings-acl.772"},{"key":"8","unstructured":"\u95a2\u6839\u8061\uff0c\u5b89\u85e4\u307e\u3084\uff0c\u5f8c\u85e4\u7f8e\u77e5\u5b50\uff0c\u9234\u6728\u4e45\u7f8e\uff0c\u6cb3\u539f\u5927\u8f14\uff0c\u4e95\u4e4b\u4e0a\u76f4\u4e5f\uff0c\u4e7e\u5065\u592a\u90ce (2024). ichikara-instruction LLM \u306e\u305f\u3081\u306e\u65e5\u672c\u8a9e\u30a4\u30f3\u30b9\u30c8\u30e9\u30af\u30b7\u30e7\u30f3\u30c7\u30fc\u30bf\u306e\u4f5c\u6210. \u81ea\u7136\u8a00\u8a9e\u51e6\u7406 \u7b2c 30 \u56de\u5e74\u6b21\u5927\u4f1a \u767a\u8868\u8ad6\u6587\u96c6, pp. 1508\u20131513. [S. Sekine et al. (2024). Ichikara-instruction LLM no Tameno Nihongo Innsutorakusyonnde-ta no Sakusei. Gengoshorigakkai Dai 30 Kai Happyou Ronnbunnsyu, pp. 1508\u20131513.]."},{"key":"9","unstructured":"Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. (2025). \u201cQwen2.5 Technical Report.\u201d <i>arXiv preprint arXiv:2412.15115<\/i>."}],"container-title":["Journal of Natural Language Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jstage.jst.go.jp\/article\/jnlp\/33\/1\/33_376\/_pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T03:53:44Z","timestamp":1774065224000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jstage.jst.go.jp\/article\/jnlp\/33\/1\/33_376\/_article\/-char\/ja\/"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":9,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026]]}},"URL":"https:\/\/doi.org\/10.5715\/jnlp.33.376","relation":{},"ISSN":["1340-7619","2185-8314"],"issn-type":[{"value":"1340-7619","type":"print"},{"value":"2185-8314","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026]]}}}