{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T06:30:29Z","timestamp":1775543429113,"version":"3.50.1"},"reference-count":73,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2025,3,24]],"date-time":"2025-03-24T00:00:00Z","timestamp":1742774400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Large language models (LLMs) hold the potential to significantly enhance data annotation for free-text healthcare records. However, ensuring their accuracy and reliability is critical, especially in clinical research applications requiring the extraction of patient characteristics. This study introduces a novel evaluation framework based on Multi-Criteria Decision Analysis (MCDA) and the Order of Preference by Similarity to Ideal Solution (TOPSIS) technique, designed to benchmark LLMs on their annotation quality. The framework defines ten evaluation metrics across key criteria such as age, gender, BMI, disease presence, and blood markers (e.g., white blood count and platelets). Using this methodology, we assessed leading open source and commercial LLMs, achieving accuracy scores of 0.59, 1, 0.84, 0.56, and 0.92, respectively, for the specified criteria. Our work not only provides a rigorous framework for evaluating LLM capabilities in healthcare data annotation but also highlights their current performance limitations and strengths. By offering a comprehensive benchmarking approach, we aim to support responsible adoption and decision-making in healthcare applications.<\/jats:p>","DOI":"10.3390\/fi17040138","type":"journal-article","created":{"date-parts":[[2025,3,24]],"date-time":"2025-03-24T04:48:18Z","timestamp":1742791698000},"page":"138","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Benchmarking Large Language Models from Open and Closed Source Models to Apply Data Annotation for Free-Text Criteria in Healthcare"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0287-4487","authenticated-orcid":false,"given":"Ali","family":"Nemati","sequence":"first","affiliation":[{"name":"Health Informatics Department, Zilber College of Public Health, University of Wisconsin, Milwaukee, WI 53211, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9629-9662","authenticated-orcid":false,"given":"Mohammad","family":"Assadi Shalmani","sequence":"additional","affiliation":[{"name":"Health Informatics Department, Zilber College of Public Health, University of Wisconsin, Milwaukee, WI 53211, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8217-2305","authenticated-orcid":false,"given":"Qiang","family":"Lu","sequence":"additional","affiliation":[{"name":"Beijing Key Laboratory of Petroleum Data Mining, China University of Petroleum, Beijing 102249, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3900-643X","authenticated-orcid":false,"given":"Jake","family":"Luo","sequence":"additional","affiliation":[{"name":"Health Informatics & Administration Department, Zilber College of Public Health, University of Wisconsin, Milwaukee, WI 53211, USA"}]}],"member":"1968","published-online":{"date-parts":[[2025,3,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Kachris, C. (2024). A Survey on Hardware Accelerators for Large Language Models. arXiv.","DOI":"10.3390\/app15020586"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., and Liu, Q. (2023). A Survey on Large Language Models for Recommendation. arXiv.","DOI":"10.1007\/s11280-024-01291-2"},{"key":"ref_3","unstructured":"Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., and Dong, Z. (2023). A Survey of Large Language Models. arXiv."},{"key":"ref_4","unstructured":"Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open Foundation and Fine-tuned Chat Models. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., and Sontag, D. (2022). Large Language Models Are Few-shot Clinical Information Extractors. arXiv.","DOI":"10.18653\/v1\/2022.emnlp-main.130"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Roussinov, D., Conkie, A., Patterson, A., and Sainsbury, C. (2022). Predicting Clinical Events Based on Raw Text: From Bag-of-Words to Attention-based Transformers. Front. Digit. Health, 3.","DOI":"10.3389\/fdgth.2021.810260"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Ollitrault, P.J., Loipersberger, M., Parrish, R.M., Erhard, A., Maier, C., Sommer, C., Ulmanis, J., Monz, T., Gogolin, C., and Tautermann, C.S. (2023). Estimation of Electrostatic Interaction Energies on a Trapped-ion Quantum Computer. arXiv.","DOI":"10.1021\/acscentsci.4c00058"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1016\/j.orhc.2013.03.001","article-title":"Multi-criteria decision analysis (MCDA) in health care: A bibliometric analysis","volume":"2","author":"Diaby","year":"2013","journal-title":"Oper. Res. Health Care"},{"key":"ref_9","unstructured":"McIntosh, T.R., Susnjak, T., Liu, T., Watters, P., and Halgamuge, M.N. (2024). Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.ejor.2016.11.052","article-title":"A modified TOPSIS with a different ranking index","volume":"260","author":"Kuo","year":"2017","journal-title":"Eur. J. Oper. Res."},{"key":"ref_11","unstructured":"Tang, R., Han, X., Jiang, X., and Hu, X. (2023). Does Synthetic Data Generation of LLMs Help Clinical Text Mining?. arXiv."},{"key":"ref_12","first-page":"100943","article-title":"Can Large Language Models Reason About Medical Questions?","volume":"5","author":"Hother","year":"2023","journal-title":"Patterns"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Park, Y.J., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., and Naugler, C. (2024). Assessing the Research Landscape and Clinical Utility of Large Language Models: A Scoping Review. BMC Med. Inform. Decis. Mak., 24.","DOI":"10.1186\/s12911-024-02459-6"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Jahan, I., Laskar, M.T.R., Peng, C., and Huang, J.X. (2024). A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Tasks. Comput. Biol. Med., 171.","DOI":"10.1016\/j.compbiomed.2024.108189"},{"key":"ref_15","unstructured":"Shivade, C., Hebert, C., Regan, K., Fosler-Lussier, E., and Lai, A.M. (2016, January 12\u201316). Automatic Data Source Identification for Clinical Trial Eligibility Criteria Resolution. Proceedings of the AMIA Annual Symposium Proceedings. American Medical Informatics Association, Chicago, IL, USA."},{"key":"ref_16","unstructured":"U.S. National Library of Medicine (2024, June 21). ClinicalTrials.gov, Available online: https:\/\/www.clinicaltrials.gov\/."},{"key":"ref_17","unstructured":"(2024, March 31). Clinical Trials Transformation Initiative. Available online: https:\/\/ctti-clinicaltrials.org\/."},{"key":"ref_18","unstructured":"Clinical Trials Transformation Initiative (2023). Patient Engagement Collaborative Framework, Clinical Trials Transformation Initiative."},{"key":"ref_19","unstructured":"Manning, C.D., Raghavan, P., and Sch\u00fctze, H. (2024, June 21). Stemming and Lemmatization. Available online: https:\/\/nlp.stanford.edu\/IR-book\/html\/htmledition\/stemming-and-lemmatization-1.html."},{"key":"ref_20","unstructured":"Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language Models are Few-Shot Learners. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1016\/1060-3743(94)90018-3","article-title":"Guidelines for designing writing prompts: Clarifications, caveats, and cautions","volume":"3","author":"Kroll","year":"1994","journal-title":"J. Second Lang. Writ."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1038\/s41562-024-01847-2","article-title":"How to write effective prompts for large language models","volume":"8","author":"Lin","year":"2024","journal-title":"Nat. Hum. Behav."},{"key":"ref_23","unstructured":"Behan, J., and Smith, J. (2023). A Survey of Data Science Education. arXiv."},{"key":"ref_24","first-page":"1324","article-title":"Large language models for healthcare data augmentation: An example on patient-trial matching","volume":"Volume 2023","author":"Yuan","year":"2023","journal-title":"Proceedings of the AMIA Annual Symposium Proceedings"},{"key":"ref_25","unstructured":"Ye, S., Kim, D., Kim, S., Hwang, H., Kim, S., Jo, Y., Thorne, J., Kim, J., and Seo, M. (2023). FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets. arXiv."},{"key":"ref_26","unstructured":"Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., and Kumar, A. (2022). Holistic evaluation of language models. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"72303","DOI":"10.1109\/ACCESS.2024.3403858","article-title":"Methodology for Code Synthesis Evaluation of LLMs Presented by a Case Study of ChatGPT and Copilot","volume":"12","author":"Siket","year":"2024","journal-title":"IEEE Access"},{"key":"ref_28","unstructured":"Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Reimers, N., and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv.","DOI":"10.18653\/v1\/D19-1410"},{"key":"ref_30","unstructured":"(2024, June 23). Confident AI Documentation. Available online: https:\/\/docs.confident-ai.com\/."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zhong, M., Liu, Y., Yin, D., Mao, Y., Jiao, Y., Liu, P., Zhu, C., Ji, H., and Han, J. (2022). Towards a unified multi-dimensional evaluator for text generation. arXiv.","DOI":"10.18653\/v1\/2022.emnlp-main.131"},{"key":"ref_32","unstructured":"Ma, Z. (2024, June 23). UniEval: Unified Evaluation of Text Generation Models. Available online: https:\/\/github.com\/maszhongming\/UniEval."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"102261","DOI":"10.1016\/j.omega.2020.102261","article-title":"How to support the application of multiple criteria decision analysis? Let us start with a comprehensive taxonomy","volume":"96","author":"Cinelli","year":"2020","journal-title":"Omega"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1080\/13921525.1999.10531475","article-title":"Property valuation by multiple criteria methods","volume":"5","author":"Zavadskas","year":"1999","journal-title":"Statyba"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., and Cambria, E. (2023). A Survey of Large Language Models for Healthcare: From Data, Technology, and Applications to Accountability and Ethics. arXiv.","DOI":"10.2139\/ssrn.4809363"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Goutte, C., and Gaussier, E. (2005, January 21\u201323). A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. Proceedings of the European Conference on Information Retrieval, March Santiago de Compostela, Spain.","DOI":"10.1007\/978-3-540-31865-1_25"},{"key":"ref_37","unstructured":"Sheets, G. (2024, June 25). Untitled Spreadsheet. Available online: https:\/\/docs.google.com\/spreadsheets\/d\/1RKOVpselB98Nnh_EOC4A2BYn8_201tmPODpNWu4w7xI\/edit?gid=0."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Henderson, M., Budzianowski, P., Casanueva, I., Coope, S., Gerz, D., Kumar, G., Mrk\u0161i\u0107, N., Spithourakis, G., Su, P.H., and Vuli\u0107, I. (2019). A repository of conversational datasets. arXiv.","DOI":"10.18653\/v1\/W19-4101"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Galli, C., Donos, N., and Calciolari, E. (2024). Performance of 4 Pre-Trained Sentence Transformer Models in the Semantic Query of a Systematic Review Dataset on Peri-Implantitis. Information, 15.","DOI":"10.3390\/info15020068"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023). G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv.","DOI":"10.18653\/v1\/2023.emnlp-main.153"},{"key":"ref_41","unstructured":"Huang, Y., Feng, X., Feng, X., and Qin, B. (2021). The factual inconsistency problem in abstractive text summarization: A survey. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Li, Y., Li, L., Hu, D., Li, Y., Zhou, Y., Litvak, M., and Vanetik, N. (2023). Just ClozE! A Novel Framework for Evaluating the Factual Consistency Faster in Abstractive Summarization. arXiv.","DOI":"10.21203\/rs.3.rs-3695932\/v1"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Chen, Y., Liu, P., and Qiu, X. (2021, January 16\u201320). Are factuality checkers reliable? Adversarial meta-evaluation of factuality in summarization. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual.","DOI":"10.18653\/v1\/2021.findings-emnlp.179"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Papinesi, K. (2002, January 6\u201312). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Actual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Tzeng, G.H., and Huang, J.J. (2011). Multiple Attribute Decision Making: Methods and Applications, CRC Press.","DOI":"10.1201\/b11032"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"13051","DOI":"10.1016\/j.eswa.2012.05.056","article-title":"A state-of the-art survey of TOPSIS applications","volume":"39","author":"Behzadian","year":"2012","journal-title":"Expert Syst. Appl."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Chen, S.J., and Hwang, C.L. (1992). Fuzzy multiple attribute decision making methods and applications. Fuzzy Multiple Attribute Decision Making, Springer.","DOI":"10.1007\/978-3-642-46768-4"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Hwang, C.L., and Yoon, K. (1981). Multiple Attribute Decision Making: Methods and Applications, Springer.","DOI":"10.1007\/978-3-642-48318-9"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"12314","DOI":"10.1016\/j.eswa.2009.04.045","article-title":"A hybrid multi-criteria decision-making model for firms competence evaluation","volume":"36","author":"Amiri","year":"2009","journal-title":"Expert Syst. Appl."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: A literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J. Biomed. Inform."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1136\/amiajnl-2013-001935","article-title":"A review of approaches to identifying patient phenotype cohorts using electronic health records","volume":"21","author":"Shivade","year":"2014","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large Language Models Encode Clinical Knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Alsentzer, E., Murphy, J.R., Boag, W., Weng, W.H., Jin, D., Naumann, T., and McDermott, M. (2019). Publicly available clinical BERT embeddings. arXiv.","DOI":"10.18653\/v1\/W19-1909"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1055\/s-0038-1638592","article-title":"Extracting information from textual documents in the electronic health record: A review of recent research","volume":"17","author":"Meystre","year":"2008","journal-title":"Yearb. Med. Inform."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Shahriar, S., and Dara, R. (2025). Priv-IQ: A Benchmark and Comparative Evaluation of Large Multimodal Models on Privacy Competencies. AI, 6.","DOI":"10.3390\/ai6020029"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.jbi.2017.07.012","article-title":"Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic review","volume":"73","author":"Kreimeyer","year":"2017","journal-title":"J. Biomed. Inform."},{"key":"ref_58","unstructured":"Johnson, A.E., Pollard, T.J., and Mark, R.G. (2017, January 18\u201319). Reproducibility in critical care: A mortality prediction case study. Proceedings of the Machine Learning for Healthcare Conference, PMLR, Boston, MA, USA."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1093\/jamia\/ocz200","article-title":"Deep learning in clinical natural language processing: A methodical review","volume":"27","author":"Wu","year":"2020","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1016\/j.jbi.2009.08.007","article-title":"What can natural language processing do for clinical decision support?","volume":"42","author":"Chapman","year":"2009","journal-title":"J. Biomed. Inform."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1093\/jamia\/ocac216","article-title":"Machine learning approaches for electronic health records phenotyping: A methodical review","volume":"30","author":"Yang","year":"2023","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.ijmedinf.2018.03.013","article-title":"Concurrence of big data analytics and healthcare: A systematic review","volume":"114","author":"Mehta","year":"2018","journal-title":"Int. J. Med. Inform."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1038\/s41746-024-01083-y","article-title":"Evaluating large language models as agents in the clinic","volume":"7","author":"Mehandru","year":"2024","journal-title":"NPJ Digit. Med."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1097\/CM9.0000000000003456","article-title":"Application of large language models in disease diagnosis and treatment","volume":"138","author":"Yang","year":"2025","journal-title":"Chin. Med. J."},{"key":"ref_65","unstructured":"Google (2024, March 31). Our Responsible Approach to Building Guardrails for Generative AI. Available online: https:\/\/blog.google\/technology\/ai\/our-responsible-approach-to-building-guardrails-for-generative-ai\/."},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Yacouby, R., and Axman, D. (2020, January 20). Probabilistic extension of precision, recall, and f1 score for more thorough evaluation of classification models. Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, Online.","DOI":"10.18653\/v1\/2020.eval4nlp-1.9"},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"548","DOI":"10.1016\/j.ics.2005.03.290","article-title":"Definition of accuracy and precision\u2014Evaluating CAS-systems","volume":"Volume 1281","author":"Hofer","year":"2005","journal-title":"Proceedings of the International Congress Series"},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.mcm.2010.07.022","article-title":"Cosine similarity measures for intuitionistic fuzzy sets and their applications","volume":"53","author":"Ye","year":"2011","journal-title":"Math. Comput. Model."},{"key":"ref_69","doi-asserted-by":"crossref","unstructured":"Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., and Matias, Y. (2022). TRUE: Re-evaluating factual consistency evaluation. arXiv.","DOI":"10.18653\/v1\/2022.naacl-main.287"},{"key":"ref_70","doi-asserted-by":"crossref","unstructured":"Anderson, C., Vandenberg, B., Hauser, C., Johansson, A., and Galloway, N. (2024). Semantic Coherence Dynamics in Large Language Models Through Layered Syntax-Aware Memory Retention Mechanism, Authroea.","DOI":"10.22541\/au.173016320.04754825\/v1"},{"key":"ref_71","unstructured":"Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., and Liu, H. (2024). A Survey on LLM-as-a-Judge. arXiv."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"100103","DOI":"10.1016\/j.chbah.2024.100103","article-title":"The fluency-based semantic network of LLMs differs from humans","volume":"3","author":"Wang","year":"2025","journal-title":"Comput. Hum. Behav. Artif. Hum."},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Mackie, I., Chatterjee, S., and Dalton, J. (2023, January 23\u201327). Generative relevance feedback with large language models. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, Taipei, Taiwan.","DOI":"10.1145\/3539618.3591992"}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/17\/4\/138\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:58:56Z","timestamp":1760029136000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/17\/4\/138"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,24]]},"references-count":73,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,4]]}},"alternative-id":["fi17040138"],"URL":"https:\/\/doi.org\/10.3390\/fi17040138","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,24]]}}}