{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T14:59:34Z","timestamp":1776092374093,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":56,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,3,18]],"date-time":"2024-03-18T00:00:00Z","timestamp":1710720000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"the Young Scientists Fund of the National Natural Science Foundation of China","award":["62206106"],"award-info":[{"award-number":["62206106"]}]},{"name":"JST ACT-X","award":["JP-MJAX21AG"],"award-info":[{"award-number":["JP-MJAX21AG"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,3,18]]},"DOI":"10.1145\/3640543.3645174","type":"proceedings-article","created":{"date-parts":[[2024,4,5]],"date-time":"2024-04-05T18:23:12Z","timestamp":1712341392000},"page":"419-430","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["PDFChatAnnotator: A Human-LLM Collaborative Multi-Modal Data Annotation Tool for PDF-Format Catalogs"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-6594-8152","authenticated-orcid":false,"given":"Yi","family":"Tang","sequence":"first","affiliation":[{"name":"Jilin University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0390-6361","authenticated-orcid":false,"given":"Chia-Ming","family":"Chang","sequence":"additional","affiliation":[{"name":"The University of Tokyo, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5039-3680","authenticated-orcid":false,"given":"Xi","family":"Yang","sequence":"additional","affiliation":[{"name":"Jilin University, China"}]}],"member":"320","published-online":{"date-parts":[[2024,4,5]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-4160-6645-3.00160-2"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.130"},{"key":"e_1_3_2_2_3_1","unstructured":"Emilia Apostolova Sean Neilan Gary An Noriko Tomuro and Steven Lytinen. [n. d.]. Djangology: A Light-weight Web-based Tool for Distributed Collaborative Text. ([n. d.])."},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415167"},{"key":"e_1_3_2_2_5_1","unstructured":"Li Boqian. 2018. The Complete Collection of Bronzes Unearthed in China. Longmen Bookstore."},{"key":"e_1_3_2_2_6_1","volume-title":"Language models are few-shot learners. Advances in neural information processing systems 33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared\u00a0D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877\u20131901."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300234"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445165"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1155\/2020\/8818616"},{"key":"e_1_3_2_2_10_1","volume-title":"Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311","author":"Chowdhery Aakanksha","year":"2022","unstructured":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung\u00a0Won Chung, Charles Sutton, Sebastian Gehrmann, 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)."},{"key":"e_1_3_2_2_11_1","volume-title":"CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1\u20134.","author":"Joon\u00a0Young Chung John","year":"2022","unstructured":"John Joon\u00a0Young Chung, Wooseok Kim, Kang\u00a0Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: visual sketching of story generation with pretrained language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1\u20134."},{"key":"e_1_3_2_2_12_1","volume-title":"How to prompt? Opportunities and challenges of zero-and few-shot learning for human-AI interaction in creative applications of generative models. arXiv preprint arXiv:2209.01390","author":"Dang Hai","year":"2022","unstructured":"Hai Dang, Lukas Mecke, Florian Lehmann, Sven Goller, and Daniel Buschek. 2022. How to prompt? Opportunities and challenges of zero-and few-shot learning for human-AI interaction in creative applications of generative models. arXiv preprint arXiv:2209.01390 (2022)."},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_2_14_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_3_2_2_15_1","volume-title":"Exploring the feasibility of chatgpt for event extraction. arXiv preprint arXiv:2303.03836","author":"Gao Jun","year":"2023","unstructured":"Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. 2023. Exploring the feasibility of chatgpt for event extraction. arXiv preprint arXiv:2303.03836 (2023)."},{"key":"e_1_3_2_2_16_1","volume-title":"Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056","author":"Gilardi Fabrizio","year":"2023","unstructured":"Fabrizio Gilardi, Meysam Alizadeh, and Ma\u00ebl Kubli. 2023. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056 (2023)."},{"key":"e_1_3_2_2_17_1","volume-title":"Thinking about gpt-3 in-context learning for biomedical ie? think again. arXiv preprint arXiv:2203.08410","author":"Gutierrez Bernal\u00a0Jimenez","year":"2022","unstructured":"Bernal\u00a0Jimenez Gutierrez, Nikolas McNeal, Clay Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about gpt-3 in-context learning for biomedical ie? think again. arXiv preprint arXiv:2203.08410 (2022)."},{"key":"e_1_3_2_2_18_1","volume-title":"Proceedings of the 10th Conference on Open Research Areas in Information Retrieval. 57\u201360","author":"He Jiyin","year":"2013","unstructured":"Jiyin He, Jacco van Ossenbruggen, and Arjen\u00a0P de Vries. 2013. Do you need experts in the crowd? A case study in image annotation for marine biology. In Proceedings of the 10th Conference on Open Research Areas in Information Retrieval. 57\u201360."},{"key":"e_1_3_2_2_19_1","volume-title":"Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854","author":"He Xingwei","year":"2023","unstructured":"Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu\u00a0Ming Yiu, Nan Duan, Weizhu Chen, 2023. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854 (2023)."},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"crossref","unstructured":"Sheng-Jun Huang Chen-Chen Zong Kun-Peng Ning and Hai-Bo Ye. 2021. Asynchronous Active Learning with Distributed Label Querying.. In IJCAI. 2570\u20132576.","DOI":"10.24963\/ijcai.2021\/354"},{"key":"e_1_3_2_2_21_1","volume-title":"Effective human\u2013AI work design for collaborative decision-making. Kybernetes","author":"Jain Ruchika","year":"2022","unstructured":"Ruchika Jain, Naval Garg, and Shikha\u00a0N Khera. 2022. Effective human\u2013AI work design for collaborative decision-making. Kybernetes (2022)."},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3503564"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDARW.2019.00018"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501931"},{"key":"e_1_3_2_2_25_1","unstructured":"Alex Krizhevsky Geoffrey Hinton 2009. Learning multiple layers of features from tiny images. (2009)."},{"key":"e_1_3_2_2_26_1","volume-title":"W1","author":"Kwon Dongseop","year":"2018","unstructured":"Dongseop Kwon, Sun Kim, Chih-Hsuan Wei, Robert Leaman, and Zhiyong Lu. 2018. ezTag: tagging biomedical concepts via interactive learning. Nucleic acids research 46, W1 (2018), W523\u2013W529."},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501999"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502030"},{"key":"e_1_3_2_2_29_1","volume-title":"Man-computer symbiosis. IRE transactions on human factors in electronics1","author":"Licklider CR","year":"1960","unstructured":"Joseph\u00a0CR Licklider. 1960. Man-computer symbiosis. IRE transactions on human factors in electronics1 (1960), 4\u201311."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_3_2_2_31_1","volume-title":"Summary of chatgpt\/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.01852","author":"Liu Yiheng","year":"2023","unstructured":"Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, 2023. Summary of chatgpt\/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.01852 (2023)."},{"key":"e_1_3_2_2_32_1","volume-title":"Large language model is not a good few-shot information extractor, but a good reranker for hard samples!arXiv preprint arXiv:2303.08559","author":"Ma Yubo","year":"2023","unstructured":"Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023. Large language model is not a good few-shot information extractor, but a good reranker for hard samples!arXiv preprint arXiv:2303.08559 (2023)."},{"key":"e_1_3_2_2_33_1","volume-title":"Excellent Collection of Yunnan Provincial Museum Porcelain I","author":"Museum Yunnan\u00a0Provincial","unstructured":"Yunnan\u00a0Provincial Museum. 2013. Excellent Collection of Yunnan Provincial Museum Porcelain I. Yunnan People\u2019s Publishing House."},{"key":"e_1_3_2_2_34_1","volume-title":"PAWLS: PDF annotation with labels and structure. arXiv preprint arXiv:2101.10281","author":"Neumann Mark","year":"2021","unstructured":"Mark Neumann, Zejiang Shen, and Sam Skjonsberg. 2021. PAWLS: PDF annotation with labels and structure. arXiv preprint arXiv:2101.10281 (2021)."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357236.3395454"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2018.2881928"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"e_1_3_2_2_38_1","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730\u201327744.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_2_39_1","volume-title":"Automated Annotation with Generative AI Requires Validation. arXiv preprint arXiv:2306.00176","author":"Pangakis Nicholas","year":"2023","unstructured":"Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. Automated Annotation with Generative AI Requires Validation. arXiv preprint arXiv:2306.00176 (2023)."},{"key":"e_1_3_2_2_40_1","volume-title":"Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering\u2013Example of ChatGPT. arXiv preprint arXiv:2303.05352","author":"Polak P","year":"2023","unstructured":"Maciej\u00a0P Polak and Dane Morgan. 2023. Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering\u2013Example of ChatGPT. arXiv preprint arXiv:2303.05352 (2023)."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bts435"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047247"},{"key":"e_1_3_2_2_44_1","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC","author":"Shindo Hiroyuki","year":"2018","unstructured":"Hiroyuki Shindo, Yohei Munesada, and Yuji Matsumoto. 2018. PDFAnno: a web-based linguistic annotation tool for pdf documents. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)."},{"key":"e_1_3_2_2_45_1","volume-title":"Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics. 102\u2013107","author":"Stenetorp Pontus","year":"2012","unstructured":"Pontus Stenetorp, Sampo Pyysalo, Goran Topi\u0107, Tomoko Ohta, Sophia Ananiadou, and Jun\u2019ichi Tsujii. 2012. BRAT: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics. 102\u2013107."},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.sigpro.2015.09.027"},{"key":"e_1_3_2_2_47_1","volume-title":"Does synthetic data generation of llms help clinical text mining?arXiv preprint arXiv:2303.04360","author":"Tang Ruixiang","year":"2023","unstructured":"Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. 2023. Does synthetic data generation of llms help clinical text mining?arXiv preprint arXiv:2303.04360 (2023)."},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3604273"},{"key":"e_1_3_2_2_49_1","first-page":"63","article-title":"FoLiA: A practical XML format for linguistic annotation\u2013a descriptive and comparative study","volume":"3","author":"van Gompel Maarten","year":"2013","unstructured":"Maarten van Gompel and Martin Reynaert. 2013. FoLiA: A practical XML format for linguistic annotation\u2013a descriptive and comparative study. Computational Linguistics in the Netherlands Journal 3 (2013), 63\u201381.","journal-title":"Computational Linguistics in the Netherlands Journal"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580895"},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"crossref","unstructured":"Dakuo Wang Elizabeth Churchill Pattie Maes Xiangmin Fan Ben Shneiderman Yuanchun Shi and Qianying Wang. 2020. From human-human collaboration to Human-AI collaboration: Designing AI systems that can work together with people. In Extended abstracts of the 2020 CHI conference on human factors in computing systems. 1\u20136.","DOI":"10.1145\/3334480.3381069"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359313"},{"key":"e_1_3_2_2_53_1","volume-title":"Code4struct: Code generation for few-shot structured prediction from natural language. arXiv preprint arXiv:2210.12810","author":"Wang Xingyao","year":"2022","unstructured":"Xingyao Wang, Sha Li, and Heng Ji. 2022. Code4struct: Code generation for few-shot structured prediction from natural language. arXiv preprint arXiv:2210.12810 (2022)."},{"key":"e_1_3_2_2_54_1","volume-title":"Zero-shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302.10205","author":"Wei Xiang","year":"2023","unstructured":"Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, 2023. Zero-shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302.10205 (2023)."},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220012"},{"key":"e_1_3_2_2_56_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 4, CSCW3","author":"Zhang Rui","year":"2021","unstructured":"Rui Zhang, Nathan\u00a0J McNeese, Guo Freeman, and Geoff Musick. 2021. \" An ideal human\" expectations of AI teammates in human-AI teaming. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1\u201325."}],"event":{"name":"IUI '24: 29th International Conference on Intelligent User Interfaces","location":"Greenville SC USA","acronym":"IUI '24","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 29th International Conference on Intelligent User Interfaces"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640543.3645174","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3640543.3645174","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T00:57:20Z","timestamp":1764550640000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640543.3645174"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,18]]},"references-count":56,"alternative-id":["10.1145\/3640543.3645174","10.1145\/3640543"],"URL":"https:\/\/doi.org\/10.1145\/3640543.3645174","relation":{},"subject":[],"published":{"date-parts":[[2024,3,18]]},"assertion":[{"value":"2024-04-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}