{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T17:13:10Z","timestamp":1774631590960,"version":"3.50.1"},"reference-count":176,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T00:00:00Z","timestamp":1754092800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2020YFA0909100"],"award-info":[{"award-number":["2020YFA0909100"]}]},{"name":"National Key Research and Development Program of China","award":["2023TQ07A264"],"award-info":[{"award-number":["2023TQ07A264"]}]},{"name":"Basic and Applied Basic Research Foundation of Guangdong Province","award":["2020YFA0909100"],"award-info":[{"award-number":["2020YFA0909100"]}]},{"name":"Basic and Applied Basic Research Foundation of Guangdong Province","award":["2023TQ07A264"],"award-info":[{"award-number":["2023TQ07A264"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Data annotation serves as a critical foundation for artificial intelligence (AI) and machine learning (ML). Recently, AI agents powered by large language models (LLMs) have emerged as effective solutions to longstanding challenges in data annotation, such as scalability, consistency, cost, and limitations in domain expertise. These agents facilitate intelligent automation and adaptive decision-making, thereby enhancing the efficiency and reliability of annotation workflows across various fields. Despite the growing interest in this area, a systematic understanding of the role and capabilities of AI agents in annotation is still underexplored. This paper seeks to fill that gap by providing a comprehensive review of how LLM-driven agents support advanced reasoning strategies, adaptive learning, and collaborative annotation efforts. We analyze agent architectures, integration patterns within workflows, and evaluation methods, along with real-world applications in sectors such as healthcare, finance, technology, and media. Furthermore, we evaluate current tools and platforms that support agent-based annotation, addressing key challenges such as quality assurance, bias mitigation, transparency, and scalability. Lastly, we outline future research directions, highlighting the importance of federated learning, cross-modal reasoning, and responsible system design to advance the development of next-generation annotation ecosystems.<\/jats:p>","DOI":"10.3390\/fi17080353","type":"journal-article","created":{"date-parts":[[2025,8,4]],"date-time":"2025-08-04T15:30:06Z","timestamp":1754321406000},"page":"353","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Transforming Data Annotation with AI Agents: A Review of Architectures, Reasoning, Applications, and Impact"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1753-065X","authenticated-orcid":false,"given":"Md Monjurul","family":"Karim","sequence":"first","affiliation":[{"name":"Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4193-9340","authenticated-orcid":false,"given":"Sangeen","family":"Khan","sequence":"additional","affiliation":[{"name":"Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-7564-2415","authenticated-orcid":false,"given":"Dong Hoang","family":"Van","sequence":"additional","affiliation":[{"name":"Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China"}]},{"given":"Xinyue","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Management, South-Central Minzu University, Wuhan 430074, China"}]},{"given":"Chunhui","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Technology, Zhejiang University, Hangzhou 310027, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5814-8460","authenticated-orcid":false,"given":"Qiang","family":"Qu","sequence":"additional","affiliation":[{"name":"Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China"}]}],"member":"1968","published-online":{"date-parts":[[2025,8,2]]},"reference":[{"key":"ref_1","unstructured":"Williams, K.L. (2005, January 27\u201329). The Role of Data in Artificial Intelligence: Informing, Training, and Enhancing AI Systems. Proceedings of the International Conference on Information Technology-New Generations, Las Vegas, NV, USA."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Ghaisas, S., and Singhal, A. (2025). Dealing with Data for RE: Mitigating Challenges while using NLP and Generative AI. Handbook on Natural Language Processing for Requirements Engineering, Springer.","DOI":"10.1007\/978-3-031-73143-3_17"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Houenou, B. (2025, July 31). AI Labor Markets: Tradability, Wage Inequality and Talent Development. Available online: https:\/\/ssrn.com\/abstract=5163063.","DOI":"10.2139\/ssrn.5163063"},{"key":"ref_4","unstructured":"Haq, M.U.U., Rigoni, D., and Sperduti, A. (2025). LLMs as Data Annotators: How Close Are We to Human Performance. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Mirzakhmedova, N., Gohsen, M., Chang, C.H., and Stein, B. (2024, January 5\u20137). Are Large Language Models Reliable Argument Quality Annotators?. Proceedings of the Conference on Advances in Robust Argumentation Machines, Bielefeld, Germany.","DOI":"10.1007\/978-3-031-63536-6_8"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Yin, Z., Tyson, G., Haq, E.U., Lee, L.H., and Hui, P. (2024, January 13\u201317). Apt-pipe: A prompt-tuning tool for social data annotation using chatgpt. Proceedings of the ACM Web Conference 2024, Singapore.","DOI":"10.1145\/3589334.3645642"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"6347","DOI":"10.1109\/LRA.2023.3304116","article-title":"Weakly supervised learning for point cloud semantic segmentation with dual teacher","volume":"8","author":"Yao","year":"2023","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Sun, G., Zhan, X., and Such, J. (2024, January 8\u201310). Building better ai agents: A provocation on the utilisation of persona in llm-based conversational agents. Proceedings of the 6th ACM Conference on Conversational User Interfaces, Luxembourg.","DOI":"10.1145\/3640794.3665887"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Zhang, L., Zhang, Q., Wang, H., Xiao, E., Jiang, Z., Chen, H., and Xu, R. (2024, January 14\u201318). Trihelper: Zero-shot object navigation with dynamic assistance. Proceedings of the 2024 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Abu Dhabi, UAE.","DOI":"10.1109\/IROS58592.2024.10802670"},{"key":"ref_10","unstructured":"Gottipati, S.K., Nguyen, L.H., Mars, C., and Taylor, M.E. (June, January 29). Hiking up that hill with cogment-verse: Train & operate multi-agent systems learning from humans. Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, London, UK."},{"key":"ref_11","unstructured":"Tsiakas, K., and Murray-Rust, D. (July, January 29). Using human-in-the-loop and explainable AI to envisage new future work practices. Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments, Corfu, Greece."},{"key":"ref_12","unstructured":"Yuan, S., Chen, Z., Xi, Z., Ye, J., Du, Z., and Chen, J. (2025). Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Renze, M., and Guven, E. (2024). Self-reflection in llm agents: Effects on problem-solving performance. arXiv.","DOI":"10.1109\/FLLM63129.2024.10852426"},{"key":"ref_14","unstructured":"Gr\u00f6tschla, F., M\u00fcller, L., T\u00f6nshoff, J., Galkin, M., and Perozzi, B. (2025). AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMs. arXiv."},{"key":"ref_15","unstructured":"Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., and Liu, J. (2024, January 7\u20139). Autogen: Enabling next-gen LLM applications via multi-agent conversations. Proceedings of the First Conference on Language Modeling, Philadelphia, PA, USA."},{"key":"ref_16","unstructured":"Park, J.S., O\u2019Brien, J., Cai, C.J., Morris, M.R., Liang, P., and Bernstein, M.S. (November, January 29). Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th Annual Acm Symposium on User Interface Software and Technology, Francisco, CA, USA."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1007\/s11023-024-09664-2","article-title":"Reflective artificial intelligence","volume":"34","author":"Lewis","year":"2024","journal-title":"Minds Mach."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Chu, S.Y., Kim, J.W., and Yi, M.Y. (May, January 26). Think together and work better: Combining humans\u2019 and LLMs\u2019 think-aloud outcomes for effective text evaluation. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.","DOI":"10.1145\/3706598.3713181"},{"key":"ref_19","unstructured":"Nathani, D., Madaan, L., Roberts, N., Bashlykov, N., Menon, A., Moens, V., Budhiraja, A., Magka, D., Vorotilov, V., and Chaurasia, G. (2025). Mlgym: A new framework and benchmark for advancing ai research agents. arXiv."},{"key":"ref_20","unstructured":"Yu, A., Lebedev, E., Everett, L., Chen, X., and Chen, T. (2025). Autonomous Deep Agent. arXiv."},{"key":"ref_21","unstructured":"Demrozi, F., Turetta, C., Machot, F.A., Pravadelli, G., and Kindt, P.H. (2023). A comprehensive review of automated data annotation techniques in human activity recognition. arXiv."},{"key":"ref_22","unstructured":"Zhou, Y., Guo, C., Wang, X., Chang, Y., and Wu, Y. (2024). A survey on data augmentation in large model era. arXiv."},{"key":"ref_23","unstructured":"Wang, K., Zhu, J., Ren, M., Liu, Z., Li, S., Zhang, Z., Zhang, C., Wu, X., Zhan, Q., and Liu, Q. (2024). A survey on data synthesis and augmentation for large language models. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Tan, Z., Beigi, A., Wang, S., Guo, R., Bhattacharjee, A., Jiang, B., Karami, M., Li, J., Cheng, L., and Liu, H. (2024). Large language models for data annotation: A survey. arXiv.","DOI":"10.18653\/v1\/2024.emnlp-main.54"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"121101","DOI":"10.1007\/s11432-024-4222-0","article-title":"The rise and potential of large language model based agents: A survey","volume":"68","author":"Xi","year":"2025","journal-title":"Sci. China Inf. Sci."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3722214","article-title":"Data readiness for AI: A 360-degree survey","volume":"57","author":"Hiniduma","year":"2025","journal-title":"ACM Comput. Surv."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3711118","article-title":"Data-centric artificial intelligence: A survey","volume":"57","author":"Zha","year":"2025","journal-title":"ACM Comput. Surv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"669","DOI":"10.1038\/s42256-022-00516-1","article-title":"Advances, challenges and opportunities in creating data for trustworthy AI","volume":"4","author":"Liang","year":"2022","journal-title":"Nat. Mach. Intell."},{"key":"ref_29","unstructured":"Cao, Y., Hong, S., Li, X., Ying, J., Ma, Y., Liang, H., Liu, Y., Yao, Z., Wang, X., and Huang, D. (2025). Toward generalizable evaluation in the llm era: A survey beyond benchmarks. arXiv."},{"key":"ref_30","unstructured":"Xu, F., Hao, Q., Zong, Z., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., Ouyang, T., and Meng, F. (2025). Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models. arXiv."},{"key":"ref_31","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_32","unstructured":"Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv."},{"key":"ref_33","unstructured":"Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N.A., Khashabi, D., and Hajishirzi, H. (2022). Self-instruct: Aligning language models with self-generated instructions. arXiv.","DOI":"10.18653\/v1\/2023.acl-long.754"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., and Zhenqiang Gong, N. (2023). Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv.","DOI":"10.1145\/3689217.3690621"},{"key":"ref_36","unstructured":"Est\u00e9vez-Almenzar, M., Baeza-Yates, R., and Castillo, C. (2025). A Comparison of Human and Machine Learning Errors in Face Recognition. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Wei, J., Zhu, Z., Luo, T., Amid, E., Kumar, A., and Liu, Y. (2023, January 6\u201310). To aggregate or not? learning with separate noisy labels. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Long Beach, CA, USA.","DOI":"10.1145\/3580305.3599522"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"71876","DOI":"10.1109\/ACCESS.2024.3402809","article-title":"Chatgpt label: Comparing the quality of human-generated and llm-generated annotations in low-resource language nlp tasks","volume":"12","author":"Nasution","year":"2024","journal-title":"IEEE Access"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Mamat, N., Othman, M.F., Abdoulghafor, R., Belhaouari, S.B., Mamat, N., and Mohd Hussein, S.F. (2022). Advanced technology in agriculture industry by implementing image annotation technique and deep learning approach: A review. Agriculture, 12.","DOI":"10.3390\/agriculture12071033"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"2173","DOI":"10.1016\/j.prostr.2024.09.331","article-title":"A Deep Active Learning Framework for Crack Detection in Digital Images of Paintings","volume":"64","author":"Nadisic","year":"2024","journal-title":"Procedia Struct. Integr."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Newman, J., and Cox, C. (2021). Corpus annotation. A Practical Handbook of Corpus Linguistics, Springer.","DOI":"10.1007\/978-3-030-46216-1_2"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"104156","DOI":"10.1109\/ACCESS.2022.3210119","article-title":"Chinese named entity recognition of epidemiological investigation of information on COVID-19 based on BERT","volume":"10","author":"Yang","year":"2022","journal-title":"IEEE Access"},{"key":"ref_43","first-page":"2533613","article-title":"Enhancing Learning in Fine-Tuned Transfer Learning for Rotating Machinery via Negative Transfer Mitigation","volume":"73","author":"Kumar","year":"2024","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_44","unstructured":"Tu, S., Sun, J., Zhang, Q., Lan, X., and Zhao, D. (2024). Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model. arXiv."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"18912","DOI":"10.1109\/ACCESS.2025.3532853","article-title":"Agentic AI: Autonomous Intelligence for Complex Goals\u2013A Comprehensive Survey","volume":"13","author":"Acharya","year":"2025","journal-title":"IEEE Access"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1747","DOI":"10.1111\/bjet.13429","article-title":"Towards prescriptive analytics of self-regulated learning strategies: A reinforcement learning approach","volume":"55","author":"Osakwe","year":"2024","journal-title":"Br. J. Educ. Technol."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"2257","DOI":"10.1016\/j.procs.2025.01.286","article-title":"SAMBA: A reference framework for Human-in-the-Loop in adaptive Smart Manufacturing","volume":"253","author":"Bianchini","year":"2025","journal-title":"Procedia Comput. Sci."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Karim, M.M., Van, D.H., Khan, S., Qu, Q., and Kholodov, Y. (2025). AI Agents Meet Blockchain: A Survey on Secure and Scalable Collaboration for Multi-Agents. Future Internet, 17.","DOI":"10.3390\/fi17020057"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Dong, X., Zhang, X., Bu, W., Zhang, D., and Cao, F. (2024, January 13\u201315). A Survey of LLM-based Agents: Theories, Technologies, Applications and Suggestions. Proceedings of the 2024 3rd International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC), Wuhan, China.","DOI":"10.1109\/AIoTC63215.2024.10748304"},{"key":"ref_50","unstructured":"Huang, Y. (2024). Levels of AI agents: From rules to large language models. arXiv."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Boyina, K., Reddy, G.M., Akshita, G., and Nair, P.C. (2024, January 24\u201328). Zero-Shot and Few-Shot Learning for Telugu News Classification: A Large Language Model Approach. Proceedings of the 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), Kamand, India.","DOI":"10.1109\/ICCCNT61001.2024.10724558"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Faggioli, G., Dietz, L., Clarke, C.L., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., and Stein, B. (2023, January 23). Perspectives on large language models for relevance judgment. Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval, Taipei, Taiwan.","DOI":"10.1145\/3578337.3605136"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1007\/s42001-024-00345-9","article-title":"Open-source LLMs for text annotation: A practical guide for model setting and fine-tuning","volume":"8","author":"Alizadeh","year":"2025","journal-title":"J. Comput. Soc. Sci."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Wang, X., Kim, H., Rahman, S., Mitra, K., and Miao, Z. (2024, January 11\u201316). Human-llm collaborative annotation through effective verification of llm labels. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.","DOI":"10.1145\/3613904.3641960"},{"key":"ref_55","unstructured":"Zhang, Z., Zhang, A., Li, M., and Smola, A. (2022). Automatic chain of thought prompting in large language models. arXiv."},{"key":"ref_56","unstructured":"Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023, January 1\u20135). React: Synergizing reasoning and acting in language models. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1007\/s12599-023-00841-8","article-title":"Self-learning agents for recommerce markets","volume":"66","author":"Groeneveld","year":"2024","journal-title":"Bus. Inf. Syst. Eng."},{"key":"ref_58","unstructured":"Ransiek, J., Reis, P., and Sax, E. (2024). Adversarial and Reactive Traffic Agents for Realistic Driving Simulation. arXiv."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1111\/cgf.14823","article-title":"Human\u2013Computer Collaboration for Visual Analytics: An Agent-based Framework","volume":"Volume 42","author":"Monadjemi","year":"2023","journal-title":"Proceedings of the Computer Graphics Forum"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Joshi, R., Pandey, K., Kumari, S., and Badola, R. (2025). Artificial Intelligence: A Gateway to the Twenty-First Century. The Intersection of 6G, AI\/Machine Learning, and Embedded Systems, CRC Press.","DOI":"10.1201\/9781003540212-8"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Macedo, L. (2025). Artificial Intelligence Paradigms and Agent-Based Technologies. Human-Centered AI: An Illustrated Scientific Quest, Springer.","DOI":"10.1007\/978-3-031-61375-3_3"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Roby, M. (2023). Learning and Reasoning Using Artificial Intelligence. Machine Intelligence, Auerbach Publications.","DOI":"10.1201\/9781003424550-13"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Sapkota, R., Roumeliotis, K.I., and Karkee, M. (2025). AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenge. arXiv.","DOI":"10.70777\/si.v2i3.15161"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"630","DOI":"10.1162\/tacl_a_00664","article-title":"Federated Learning for Exploiting Annotators\u2019 Disagreements in Natural Language Processing","volume":"12","author":"Collados","year":"2024","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_65","unstructured":"Azeemi, A.H., Qazi, I.A., and Raza, A.A. (2024). Language Model-Driven Data Pruning Enables Efficient Active Learning. arXiv."},{"key":"ref_66","unstructured":"Bayer, M., and Reuter, C. (2024). Activellm: Large language model-based active learning for textual few-shot scenarios. arXiv."},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Zhu, Q., Mao, Q., Zhang, J., Huang, X., and Zheng, W. (2025). Towards a robust group-level emotion recognition via uncertainty-aware learning. IEEE Trans. Affect. Comput.","DOI":"10.1109\/TAFFC.2025.3547994"},{"key":"ref_68","unstructured":"Mishra, S., Shinde, M., Yadav, A., Ayyub, B., and Rao, A. (2024). An AI-Driven Data Mesh Architecture Enhancing Decision-Making in Infrastructure Construction and Public Procurement. arXiv."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"29375","DOI":"10.1109\/ACCESS.2025.3536095","article-title":"A multifaceted vision of the Human-AI collaboration: A comprehensive review","volume":"13","author":"Chen","year":"2025","journal-title":"IEEE Access"},{"key":"ref_70","doi-asserted-by":"crossref","unstructured":"Zhang, R., Li, Y., Ma, Y., Zhou, M., and Zou, L. (2023). Llmaaa: Making large language models as active annotators. arXiv.","DOI":"10.18653\/v1\/2023.findings-emnlp.872"},{"key":"ref_71","doi-asserted-by":"crossref","unstructured":"Xia, Y., Mukherjee, S., Xie, Z., Wu, J., Li, X., Aponte, R., Lyu, H., Barrow, J., Chen, H., and Dernoncourt, F. (2025). From Selection to Generation: A Survey of LLM-based Active Learning. arXiv.","DOI":"10.18653\/v1\/2025.acl-long.708"},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"Li, X., Whan, A., McNeil, M., Starns, D., Irons, J., Andrew, S.C., and Suchecki, R. (2025). A Conceptual Framework for Human-AI Collaborative Genome Annotation. arXiv.","DOI":"10.1093\/bib\/bbaf377"},{"key":"ref_73","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1038\/s44401-024-00011-2","article-title":"Current and future state of evaluation of large language models for medical summarization tasks","volume":"2","author":"Croxford","year":"2025","journal-title":"NPJ Health Syst."},{"key":"ref_74","doi-asserted-by":"crossref","unstructured":"Yang, L., Sun, X., Li, H., Xu, R., and Wei, X. (2025). Difficulty aware programming knowledge tracing via large language models. Sci. Rep., 15.","DOI":"10.1038\/s41598-025-96540-3"},{"key":"ref_75","unstructured":"Sainz, O., Garc\u00eda-Ferrero, I., Agerri, R., de Lacalle, O.L., Rigau, G., and Agirre, E. (2023). Gollie: Annotation guidelines improve zero-shot information-extraction. arXiv."},{"key":"ref_76","unstructured":"Bibal, A., Gerlek, N., Muric, G., Boschee, E., Fincke, S.C., Ross, M., and Minton, S.N. (2025, January 19). Automating Annotation Guideline Improvements using LLMs: A Case Study. Proceedings of the Context and Meaning: Navigating Disagreements in NLP Annotation, Abu Dhabi, UAE."},{"key":"ref_77","doi-asserted-by":"crossref","unstructured":"Rodler, P., Shchekotykhin, K., Fleiss, P., and Friedrich, G. (2012). RIO: Minimizing user interaction in ontology debugging. arXiv.","DOI":"10.1007\/978-3-642-39666-3_12"},{"key":"ref_78","doi-asserted-by":"crossref","unstructured":"Zheng, J., Shi, C., Cai, X., Li, Q., Zhang, D., Li, C., Yu, D., and Ma, Q. (2025). Lifelong Learning of Large Language Model based Agents: A Roadmap. arXiv.","DOI":"10.1145\/3716629"},{"key":"ref_79","unstructured":"Zhang, G., Liang, W., Hsu, O., and Olukotun, K. (2025). Adaptive Self-improvement LLM Agentic System for ML Library Development. arXiv."},{"key":"ref_80","unstructured":"Ashktorab, Z., Pan, Q., Geyer, W., Desmond, M., Danilevsky, M., Johnson, J.M., Dugan, C., and Bachman, M. (2024). Emerging Reliance Behaviors in Human-AI Text Generation: Hallucinations, Data Quality Assessment, and Cognitive Forcing Functions. arXiv."},{"key":"ref_81","unstructured":"Wang, X., Hu, J., and Ali, S. (2025). MAATS: A Multi-Agent Automated Translation System Based on MQM Evaluation. arXiv."},{"key":"ref_82","doi-asserted-by":"crossref","unstructured":"Ara, Z., Salemi, H., Hong, S.R., Senarath, Y., Peterson, S., Hughes, A.L., and Purohit, H. (2024, January 18\u201321). Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems. Proceedings of the 29th International Conference on Intelligent User Interfaces, Greenville, SC, USA.","DOI":"10.1145\/3640543.3645214"},{"key":"ref_83","doi-asserted-by":"crossref","unstructured":"Cronin, I. (2024). Autonomous AI agents: Decision-making, data, and algorithms. Understanding Generative AI Business Applications: A Guide to Technical Principles and Real-World Applications, Springer.","DOI":"10.1007\/979-8-8688-0282-9"},{"key":"ref_84","doi-asserted-by":"crossref","first-page":"148553","DOI":"10.1109\/ACCESS.2024.3478805","article-title":"Sharing to learn and learning to share; fitting together meta, multi-task, and transfer learning: A meta review","volume":"12","author":"Upadhyay","year":"2024","journal-title":"IEEE Access"},{"key":"ref_85","unstructured":"Liu, C., Kang, Y., Zhao, F., Kuang, K., Jiang, Z., Sun, C., and Wu, F. (2024). Evolving knowledge distillation with large language models and active learning. arXiv."},{"key":"ref_86","doi-asserted-by":"crossref","unstructured":"Ding, B., Qin, C., Zhao, R., Luo, T., Li, X., Chen, G., Xia, W., Hu, J., Tuan, L.A., and Joty, S. (2024, January 11\u201316). Data augmentation using llms: Data perspectives, learning paradigms and challenges. Proceedings of the Findings of the Association for Computational Linguistics ACL 2024, Bangkok, Thailand.","DOI":"10.18653\/v1\/2024.findings-acl.97"},{"key":"ref_87","unstructured":"T\u00f6rnberg, P. (2023). Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv."},{"key":"ref_88","doi-asserted-by":"crossref","unstructured":"Choi, J., Yun, J., Jin, K., and Kim, Y. (2024). Multi-news+: Cost-efficient dataset cleansing via llm-based data annotation. arXiv.","DOI":"10.18653\/v1\/2024.emnlp-main.2"},{"key":"ref_89","unstructured":"Nahum, O., Calderon, N., Keller, O., Szpektor, I., and Reichart, R. (2024). Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance. arXiv."},{"key":"ref_90","unstructured":"Gat, Y., Calderon, N., Feder, A., Chapanin, A., Sharma, A., and Reichart, R. (2023). Faithful explanations of black-box nlp models using llm-generated counterfactuals. arXiv."},{"key":"ref_91","doi-asserted-by":"crossref","first-page":"75735","DOI":"10.1109\/ACCESS.2024.3401547","article-title":"Applications, challenges, and future directions of human-in-the-loop learning","volume":"12","author":"Kumar","year":"2024","journal-title":"IEEE Access"},{"key":"ref_92","doi-asserted-by":"crossref","first-page":"2327890","DOI":"10.1080\/08839514.2024.2327890","article-title":"Collaborative Intelligence: A scoping review of current applications","volume":"38","author":"Schleiger","year":"2024","journal-title":"Appl. Artif. Intell."},{"key":"ref_93","unstructured":"Liu, Z., Zhang, Y., Li, P., Liu, Y., and Yang, D. (2024, January 7\u20139). A dynamic LLM-powered agent network for task-oriented agent collaboration. Proceedings of the First Conference on Language Modeling, Philadelphia, PA, USA."},{"key":"ref_94","unstructured":"Yang, J., Ding, R., Brown, E., Qi, X., and Xie, S. (October, January 29). V-irl: Grounding virtual intelligence in real life. Proceedings of the European Conference on Computer Vision, Milan, Italy."},{"key":"ref_95","first-page":"46534","article-title":"Self-refine: Iterative refinement with self-feedback","volume":"36","author":"Madaan","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_96","first-page":"8634","article-title":"Reflexion: Language agents with verbal reinforcement learning","volume":"36","author":"Shinn","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_97","unstructured":"Li, D., Li, Y., Mekala, D., Li, S., Wang, X., Hogan, W., and Shang, J. (2025). DAIL: Data Augmentation for In-Context Learning via Self-Paraphrase. arXiv."},{"key":"ref_98","unstructured":"Bubeck, S., Chadrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., and Lundberg, S. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv."},{"key":"ref_99","unstructured":"Chen, Y., and Si, M. (2024, January 20\u201325). Reflections & Resonance: Two-Agent Partnership for Advancing LLM-based Story Annotation. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italia."},{"key":"ref_100","doi-asserted-by":"crossref","unstructured":"Cohen, R., Hamri, M., Geva, M., and Globerson, A. (2023). Lm vs lm: Detecting factual errors via cross examination. arXiv.","DOI":"10.18653\/v1\/2023.emnlp-main.778"},{"key":"ref_101","first-page":"443","article-title":"Framework to enable and test conversational assistant for APIs and RPAs","volume":"45","author":"Bandlamudi","year":"2024","journal-title":"AI Mag."},{"key":"ref_102","unstructured":"Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S.K.S., Lin, Z., and Zhou, L. (2025). Metagpt: Meta programming for multi-agent collaborative framework. arXiv."},{"key":"ref_103","doi-asserted-by":"crossref","unstructured":"Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., and Cong, X. (2023). Chatdev: Communicative agents for software development. arXiv.","DOI":"10.18653\/v1\/2024.acl-long.810"},{"key":"ref_104","unstructured":"Lin, M., Chen, Z., Liu, Y., Zhao, X., Wu, Z., Wang, J., Zhang, X., Wang, S., and Chen, H. (2024). Decoding Time Series with LLMs: A Multi-Agent Framework for Cross-Domain Annotation. arXiv."},{"key":"ref_105","doi-asserted-by":"crossref","unstructured":"Alam, F., Biswas, M.R., Shah, U., Zaghouani, W., and Mikros, G. (2024, January 2\u20135). Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-agent LLMs. Proceedings of the International Conference on Web Information Systems Engineering, Doha, Qatar.","DOI":"10.1007\/978-981-96-0576-7_28"},{"key":"ref_106","first-page":"466","article-title":"Research on Intelligent Agent Technology and Applications Based on Large Models","volume":"Volume 4","author":"Liu","year":"2024","journal-title":"Proceedings of the 2024 IEEE 4th International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA)"},{"key":"ref_107","doi-asserted-by":"crossref","unstructured":"Li, M., Shi, T., Ziems, C., Kan, M.Y., Chen, N.F., Liu, Z., and Yang, D. (2023). Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation. arXiv.","DOI":"10.18653\/v1\/2023.emnlp-main.92"},{"key":"ref_108","doi-asserted-by":"crossref","unstructured":"Colucci Cante, L., D\u2019Angelo, S., Di Martino, B., and Graziano, M. (2024, January 3\u20135). Text Annotation Tools: A Comprehensive Review and Comparative Analysis. Proceedings of the International Conference on Complex, Intelligent, and Software Intensive Systems, Taichung, Taiwan.","DOI":"10.1007\/978-3-031-70011-8_33"},{"key":"ref_109","unstructured":"Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., and Yang, K. (2023). Agentbench: Evaluating llms as agents. arXiv."},{"key":"ref_110","doi-asserted-by":"crossref","unstructured":"Verma, G., Kaur, R., Srishankar, N., Zeng, Z., Balch, T., and Veloso, M. (2024). Adaptagent: Adapting multimodal web agents with few-shot learning from human demonstrations. arXiv.","DOI":"10.18653\/v1\/2025.acl-long.1008"},{"key":"ref_111","doi-asserted-by":"crossref","first-page":"8659","DOI":"10.1109\/LRA.2024.3444670","article-title":"Generalized Synchronized Active Learning for Multi-Agent-Based Data Selection on Mobile Robotic Systems","volume":"9","author":"Schmidt","year":"2024","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_112","doi-asserted-by":"crossref","unstructured":"Wan, M., Safavi, T., Jauhar, S.K., Kim, Y., Counts, S., Neville, J., Suri, S., Shah, C., White, R.W., and Yang, L. (2024, January 25\u201329). Tnt-llm: Text mining at scale with large language models. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Barcelona, Spain.","DOI":"10.1145\/3637528.3671647"},{"key":"ref_113","first-page":"2936","article-title":"Investigating trust in human-AI collaboration for a speech-based data analytics task","volume":"41","author":"Tutul","year":"2025","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_114","doi-asserted-by":"crossref","unstructured":"Bolock, A.e., Abouras, M., Sabty, C., Abdennadher, S., and Herbert, C. (2024, January 26\u201328). CARE: A Framework for Collecting and Annotating Emotions of Code-Switched Words. Proceedings of the International Conference on Practical Applications of Agents and Multi-Agent Systems, Salamanca, Spain.","DOI":"10.1007\/978-3-031-73058-0_9"},{"key":"ref_115","doi-asserted-by":"crossref","first-page":"3231","DOI":"10.1109\/TKDE.2016.2601611","article-title":"Efficient online summarization of large-scale dynamic networks","volume":"28","author":"Qu","year":"2016","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_116","unstructured":"Hadian, A., Nobari, S., Minaei-Bidgoli, B., and Qu, Q. (July, January 26). Roll: Fast in-memory generation of gigantic scale-free networks. Proceedings of the 2016 International Conference on Management of Data, Francisco, CA, USA."},{"key":"ref_117","unstructured":"Chang, C.M., He, Y., Du, X., Yang, X., and Xie, H. (July, January 29). Dynamic labeling: A control system for labeling styles in image annotation tasks. Proceedings of the International Conference on Human-Computer Interaction, Washington, DC, USA."},{"key":"ref_118","unstructured":"Efrat, A., and Levy, O. (2020). The turking test: Can language models understand instructions?. arXiv."},{"key":"ref_119","unstructured":"Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. (2021, January 18\u201324). Calibrate before use: Improving few-shot performance of language models. Proceedings of the International Conference on Machine Learning. PMLR, Online."},{"key":"ref_120","doi-asserted-by":"crossref","first-page":"774","DOI":"10.1111\/nph.18387","article-title":"RootPainter: Deep learning segmentation of biological images with corrective annotation","volume":"236","author":"Smith","year":"2022","journal-title":"New Phytol."},{"key":"ref_121","doi-asserted-by":"crossref","unstructured":"Kim, H., Hessel, J., Jiang, L., West, P., Lu, X., Yu, Y., Zhou, P., Bras, R.L., Alikhani, M., and Kim, G. (2022). Soda: Million-scale dialogue distillation with social commonsense contextualization. arXiv.","DOI":"10.18653\/v1\/2023.emnlp-main.799"},{"key":"ref_122","unstructured":"Ho, N., Schmid, L., and Yun, S.Y. (2022). Large language models are reasoning teachers. arXiv."},{"key":"ref_123","unstructured":"Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J., and Sukhbaatar, S. (2024). Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv."},{"key":"ref_124","doi-asserted-by":"crossref","unstructured":"Kang, H.J., Harel-Canada, F., Gulzar, M.A., Peng, V., and Kim, M. (2024). Human-in-the-Loop Synthetic Text Data Inspection with Provenance Tracking. arXiv.","DOI":"10.18653\/v1\/2024.findings-naacl.197"},{"key":"ref_125","doi-asserted-by":"crossref","unstructured":"Wu, J., Deng, J., Pang, S., Chen, Y., Xu, J., Li, X., and Xu, W. (2024, January 14\u201318). Legilimens: Practical and unified content moderation for large language model services. Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, Salt Lake City, UT, USA.","DOI":"10.1145\/3658644.3690322"},{"key":"ref_126","doi-asserted-by":"crossref","unstructured":"Palla, K., Garc\u00eda, J.L.R., Hauff, C., Fabbri, F., Lindstr\u00f6m, H., Taber, D.R., Damianou, A., and Lalmas, M. (2025). Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models. arXiv.","DOI":"10.1145\/3715275.3732054"},{"key":"ref_127","unstructured":"Wang, Y., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., and Liu, Q. (2023). Aligning large language models with human: A survey. arXiv."},{"key":"ref_128","unstructured":"Wu, S., Fung, M., Qian, C., Kim, J., Hakkani-Tur, D., and Ji, H. (2024). Aligning LLMs with Individual Preferences via Interaction. arXiv."},{"key":"ref_129","doi-asserted-by":"crossref","first-page":"108103","DOI":"10.1016\/j.asoc.2021.108103","article-title":"Information fusion oriented heterogeneous social network for friend recommendation via community detection","volume":"114","author":"Huang","year":"2022","journal-title":"Appl. Soft Comput."},{"key":"ref_130","doi-asserted-by":"crossref","unstructured":"Boji\u0107, L., Zagovora, O., Zelenkauskaite, A., Vukovi\u0107, V., \u010cabarkapa, M., Veseljevi\u0107 Jerkovi\u0107, S., and Jovan\u010devi\u0107, A. (2025). Comparing large Language models and human annotators in latent content analysis of sentiment, political leaning, emotional intensity and sarcasm. Sci. Rep., 15.","DOI":"10.1038\/s41598-025-96508-3"},{"key":"ref_131","doi-asserted-by":"crossref","unstructured":"Harrer, S., Rane, R.V., and Speight, R.E. (2024). Generative AI agents are transforming biology research: High resolution functional genome annotation for multiscale understanding of life. EBioMedicine, 109.","DOI":"10.1016\/j.ebiom.2024.105446"},{"key":"ref_132","doi-asserted-by":"crossref","unstructured":"Toubal, I.E., Avinash, A., Alldrin, N.G., Dlabal, J., Zhou, W., Luo, E., Stretcu, O., Xiong, H., Lu, C.T., and Zhou, H. (2024, January 16\u201322). Modeling collaborator: Enabling subjective vision classification with minimal human effort via llm tool-use. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.01662"},{"key":"ref_133","doi-asserted-by":"crossref","first-page":"4366","DOI":"10.1109\/JSTARS.2025.3528192","article-title":"Towards Integrating ChatGPT into Satellite Image Annotation Workflows. A Comparison of Label Quality and Costs of Human and Automated Annotators","volume":"18","author":"Beck","year":"2025","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_134","first-page":"269","article-title":"Efficient top-k spatial locality search for co-located spatial web objects","volume":"Volume 1","author":"Qu","year":"2014","journal-title":"Proceedings of the 2014 IEEE 15th International Conference on Mobile Data Management"},{"key":"ref_135","unstructured":"Cao, X., Chen, L., Cong, G., Jensen, C.S., Qu, Q., Skovsgaard, A., Wu, D., and Yiu, M.L. (2012, January 15\u201318). Spatial keyword querying. Proceedings of the Conceptual Modeling: 31st International Conference ER 2012, Florence, Italy. Proceedings 31."},{"key":"ref_136","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3664522","article-title":"Unpacking Human-AI interactions: From interaction primitives to a design space","volume":"14","author":"Tsiakas","year":"2024","journal-title":"ACM Trans. Interact. Intell. Syst."},{"key":"ref_137","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1002\/med4.70000","article-title":"Agentic Large Language Models for Healthcare: Current Progress and Future Opportunities","volume":"3","author":"Yuan","year":"2025","journal-title":"Med. Adv."},{"key":"ref_138","first-page":"58","article-title":"Space-Time Aware Behavioral Topic Modeling for Microblog Posts","volume":"38","author":"Qu","year":"2015","journal-title":"IEEE Data Eng. Bull."},{"key":"ref_139","doi-asserted-by":"crossref","unstructured":"Kim, H., Mitra, K., Chen, R.L., Rahman, S., and Zhang, D. (2024). Meganno+: A human-llm collaborative annotation system. arXiv.","DOI":"10.18653\/v1\/2024.eacl-demo.18"},{"key":"ref_140","doi-asserted-by":"crossref","unstructured":"El Khoury, K., Godelaine, T., Delvaux, S., Lugan, S., and Macq, B. (2024, January 27\u201330). Streamlined hybrid annotation framework using scalable codestream for bandwidth-restricted uav object detection. Proceedings of the 2024 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, UAE.","DOI":"10.1109\/ICIP51287.2024.10647448"},{"key":"ref_141","unstructured":"Chen, Z.Z., Ma, J., Zhang, X., Hao, N., Yan, A., Nourbakhsh, A., Yang, X., McAuley, J., Petzold, L., and Wang, W.Y. (2024). A survey on large language models for critical societal domains: Finance, healthcare, and law. arXiv."},{"key":"ref_142","doi-asserted-by":"crossref","unstructured":"Lazo, G.R., Ayyappan, D., Sharma, P.K., and Tiwari, V.K. (2025). Contextual Science and Genome Analysis for Air-Gapped AI Research. bioRxiv.","DOI":"10.1101\/2025.03.21.644606"},{"key":"ref_143","doi-asserted-by":"crossref","unstructured":"Olawore, K., McTear, M., and Bi, Y. (2024, January 4\u20135). Development and Evaluation of a University Chatbot Using Deep Learning: A RAG-Based Approach. Proceedings of the International Symposium on Chatbots and Human-Centered AI, Thessaloniki, Greece.","DOI":"10.1007\/978-3-031-88045-2_7"},{"key":"ref_144","doi-asserted-by":"crossref","unstructured":"Li, J. (2024, January 14\u201319). A comparative study on annotation quality of crowdsourcing and LLM via label aggregation. Proceedings of the ICASSP 2024\u20132024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Korea.","DOI":"10.1109\/ICASSP48485.2024.10447803"},{"key":"ref_145","doi-asserted-by":"crossref","unstructured":"Zhou, Y., Cheng, X., Zhang, Q., Wang, L., Ding, W., Xue, X., Luo, C., and Pu, J. (2024). ALGPT: Multi-Agent Cooperative Framework for Open-Vocabulary Multi-Modal Auto-Annotating in Autonomous Driving. IEEE Trans. Intell. Veh., 1\u201315.","DOI":"10.1109\/TIV.2024.3461651"},{"key":"ref_146","doi-asserted-by":"crossref","unstructured":"Mots\u2019 oehli, M. (2024, January 23\u201325). Assistive Image Annotation Systems with Deep Learning and Natural Language Capabilities: A Review. Proceedings of the 2024 International Conference on Emerging Trends in Networks and Computer Communications (ETNCC), Windhoek, Namibia.","DOI":"10.1109\/ETNCC63262.2024.10767526"},{"key":"ref_147","doi-asserted-by":"crossref","unstructured":"Mazhar, A., Shaik, Z.H., Srivastava, A., Ruhnke, P., Vaddavalli, L., Katragadda, S.K., Yadav, S., and Akhtar, M.S. (May, January 28). Figurative-cum-Commonsense Knowledge Infusion for Multimodal Mental Health Meme Classification. Proceedings of the ACM on Web Conference 2025, Sydney, Australia.","DOI":"10.1145\/3696410.3714778"},{"key":"ref_148","doi-asserted-by":"crossref","unstructured":"Sandhu, R., Channi, H.K., Ghai, D., Cheema, G.S., and Kaur, M. (2024). An introduction to generative AI tools for education 2030. Integr. Gener. Educ. Achieve Sustain. Dev. Goals, 1\u201328.","DOI":"10.4018\/979-8-3693-2440-0.ch001"},{"key":"ref_149","doi-asserted-by":"crossref","unstructured":"Ming, X., Li, S., Li, M., He, L., and Wang, Q. (2024, January 16\u201318). AutoLabel: Automated Textual Data Annotation Method Based on Active Learning and Large Language Model. Proceedings of the International Conference on Knowledge Science, Engineering and Management, Birmingham, UK.","DOI":"10.1007\/978-981-97-5501-1_30"},{"key":"ref_150","unstructured":"Krishnan, N. (2025). Advancing Multi-Agent Systems Through Model Context Protocol: Architecture, Implementation, and Applications. arXiv."},{"key":"ref_151","doi-asserted-by":"crossref","unstructured":"Aejas, B., Belhi, A., and Bouras, A. (2023). Toward an nlp approach for transforming paper contracts into smart contracts. Intelligent Sustainable Systems: Selected Papers of WorldS4 2022, Volume 2, Springer.","DOI":"10.1007\/978-981-19-7663-6_70"},{"key":"ref_152","doi-asserted-by":"crossref","first-page":"110136","DOI":"10.1016\/j.engappai.2025.110136","article-title":"Unlocking language barriers: Assessing pre-trained large language models across multilingual tasks and unveiling the black box with Explainable Artificial Intelligence","volume":"149","author":"Kastrati","year":"2025","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_153","doi-asserted-by":"crossref","first-page":"100098","DOI":"10.1016\/j.nlp.2024.100098","article-title":"HarmonyNet: Navigating hate speech detection","volume":"8","author":"Raza","year":"2024","journal-title":"Nat. Lang. Process. J."},{"key":"ref_154","doi-asserted-by":"crossref","first-page":"200112","DOI":"10.1016\/j.sasc.2024.200112","article-title":"Telugu language hate speech detection using deep learning transformer models: Corpus generation and evaluation","volume":"6","author":"Khanduja","year":"2024","journal-title":"Syst. Soft Comput."},{"key":"ref_155","doi-asserted-by":"crossref","first-page":"100021","DOI":"10.1016\/j.ejrai.2025.100021","article-title":"Large Language Models in radiology: A technical and clinical perspective","volume":"2","author":"Kao","year":"2025","journal-title":"Eur. J. Radiol. Artif. Intell."},{"key":"ref_156","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/j.ajem.2025.03.060","article-title":"Evaluating a large language model\u2019s accuracy in chest X-ray interpretation for acute thoracic conditions","volume":"93","author":"Ostrovsky","year":"2025","journal-title":"Am. J. Emerg. Med."},{"key":"ref_157","doi-asserted-by":"crossref","unstructured":"Almalky, A.M.A., Zhou, R., Angizi, S., and Rakin, A.S. (July, January 30). How Vulnerable are Large Language Models (LLMs) against Adversarial Bit-Flip Attacks?. Proceedings of the Great Lakes Symposium on VLSI 2025, New Orleans, LA, USA.","DOI":"10.1145\/3716368.3735278"},{"key":"ref_158","doi-asserted-by":"crossref","unstructured":"Zhang, L., Zou, Q., Singhal, A., Sun, X., and Liu, P. (2024, January 21). Evaluating large language models for real-world vulnerability repair in c\/c++ code. Proceedings of the 10th ACM International Workshop on Security and Privacy Analytics, Porto, Portugal.","DOI":"10.1145\/3643651.3659892"},{"key":"ref_159","doi-asserted-by":"crossref","first-page":"126712","DOI":"10.1016\/j.eswa.2025.126712","article-title":"HaluCheck: Explainable and verifiable automation for detecting hallucinations in LLM responses","volume":"272","author":"Heo","year":"2025","journal-title":"Expert Syst. Appl."},{"key":"ref_160","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.iotcps.2025.01.001","article-title":"Generative ai in cybersecurity: A comprehensive review of llm applications and vulnerabilities","volume":"5","author":"Ferrag","year":"2025","journal-title":"Internet Things-Cyber-Phys. Syst."},{"key":"ref_161","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1145\/3641858","article-title":"Resolving the Human-Subjects Status of ML\u2019s Crowdworkers","volume":"67","author":"Kaushik","year":"2024","journal-title":"Commun. ACM"},{"key":"ref_162","doi-asserted-by":"crossref","unstructured":"Reif, Y., and Schwartz, R. (2024). Beyond performance: Quantifying and mitigating label bias in llms. arXiv.","DOI":"10.18653\/v1\/2024.naacl-long.378"},{"key":"ref_163","doi-asserted-by":"crossref","unstructured":"Feretzakis, G., Papaspyridis, K., Gkoulalas-Divanis, A., and Verykios, V.S. (2024). Privacy-preserving techniques in generative ai and large language models: A narrative review. Information, 15.","DOI":"10.3390\/info15110697"},{"key":"ref_164","doi-asserted-by":"crossref","first-page":"706","DOI":"10.1049\/blc2.12091","article-title":"Privacy preserving large language models: Chatgpt case study based vision and framework","volume":"4","author":"Ullah","year":"2024","journal-title":"IET Blockchain"},{"key":"ref_165","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1038\/s41746-025-01786-w","article-title":"Framework for bias evaluation in large language models in healthcare settings","volume":"8","author":"Templin","year":"2025","journal-title":"NPJ Digit. Med."},{"key":"ref_166","doi-asserted-by":"crossref","first-page":"4498","DOI":"10.1109\/JBHI.2025.3528526","article-title":"Taming unleashed large language models with blockchain for massive personalized reliable healthcare","volume":"29","author":"Sun","year":"2025","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_167","doi-asserted-by":"crossref","unstructured":"Moreno-S\u00e1nchez, P.A., Del Ser, J., van Gils, M., and Hernesniemi, J. (2025). A Design Framework for operationalizing Trustworthy Artificial Intelligence in Healthcare: Requirements, Tradeoffs and Challenges for its Clinical Adoption. arXiv.","DOI":"10.2139\/ssrn.5249603"},{"key":"ref_168","doi-asserted-by":"crossref","first-page":"101587","DOI":"10.1016\/j.imu.2024.101587","article-title":"A survey of explainable artificial intelligence in healthcare: Concepts, applications, and challenges","volume":"51","author":"Mienye","year":"2024","journal-title":"Inform. Med. Unlocked"},{"key":"ref_169","unstructured":"T\u00f6rnberg, P. (2024). Best practices for text annotation with large language models. arXiv."},{"key":"ref_170","doi-asserted-by":"crossref","first-page":"10351","DOI":"10.1007\/s00521-025-11099-4","article-title":"GPU-accelerated homomorphic encryption computing: Empowering federated learning in IoV","volume":"37","author":"Khan","year":"2025","journal-title":"Neural Comput. Appl."},{"key":"ref_171","unstructured":"Xie, T., Harel, D., Ran, D., Li, Z., Li, M., Yang, Z., Wang, L., Chen, X., Zhang, Y., and Zhang, W. (2025). Data and System Perspectives of Sustainable Artificial Intelligence. arXiv."},{"key":"ref_172","unstructured":"Dai, X., Li, J., Liu, X., Yu, A., and Lui, J. (2024). Cost-effective online multi-llm selection with versatile reward models. arXiv."},{"key":"ref_173","first-page":"1725","article-title":"D-llm: A token adaptive computing resource allocation strategy for large language models","volume":"37","author":"Jiang","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_174","doi-asserted-by":"crossref","unstructured":"Li, J., Han, B., Li, S., Wang, X., and Li, J. (2024, January 7\u20139). Collm: A collaborative llm inference framework for resource-constrained devices. Proceedings of the 2024 IEEE\/CIC International Conference on Communications in China (ICCC), Hangzhou, China.","DOI":"10.1109\/ICCC62479.2024.10681712"},{"key":"ref_175","doi-asserted-by":"crossref","unstructured":"Lang, J., Guo, Z., and Huang, S. (2024, January 27\u201329). A comprehensive study on quantization techniques for large language models. Proceedings of the 2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), Xiamen, China.","DOI":"10.1109\/ICAIRC64177.2024.10899941"},{"key":"ref_176","unstructured":"An, Y., Zhao, X., Yu, T., Tang, M., and Wang, J. (2024, January 26\u201327). Fluctuation-based adaptive structured pruning for large language models. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, Canada."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/17\/8\/353\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:21:57Z","timestamp":1760034117000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/17\/8\/353"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,2]]},"references-count":176,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2025,8]]}},"alternative-id":["fi17080353"],"URL":"https:\/\/doi.org\/10.3390\/fi17080353","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,2]]}}}