{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T06:05:02Z","timestamp":1789538702027,"version":"build-2803163510"},"reference-count":26,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T00:00:00Z","timestamp":1789516800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Res. Metr. Anal."],"abstract":"<jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Large language models (LLMs) have been widely applied to text classification; however, their effectiveness in domain-specific scholarly metadata classification remains insufficiently explored. This study investigates the use of open-source LLMs to classify modeling and simulation research articles across three predefined metadata dimensions: Industry, Application Area, and Modeling Method.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Using a curated dataset of research articles from the AnyLogic research repository, we evaluated models from the Qwen2.5, Llama3.1, Mistral, Ministral, and Gemma3 families. We compared Direct, Few-shot, and Chain-of-Thought prompting using either the article title alone or the title and abstract together. Performance was evaluated using Macro-F1 across the three metadata dimensions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The best overall configuration, Ministral-3-14B with Direct prompting and the title and abstract as input, achieved a Macro-F1 of 53.8%. Performance varied substantially across the metadata dimensions. Modeling Method was the easiest to classify, reaching a maximum Macro-F1 of 69.1% with Ministral-3\u201314B. The best Macro-F1 scores for Industry and Application Area were 57.6% and 39.4%, respectively. Including abstracts consistently improved classification performance across the evaluated models, whereas differences among the three prompting strategies were relatively small. Categories that were semantically overlapping, broadly defined, highly imbalanced, or not explicitly mentioned in the article text were particularly difficult to classify.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>The findings indicate that open-source LLMs can support domain-specific scholarly metadata classification without task-specific fine-tuning. However, their moderate and dimension-dependent performance limits their suitability for fully automated fine-grained metadata enrichment. Richer contextual information, more clearly defined taxonomies, and human validation are therefore needed to improve the reliability of LLM-assisted scholarly metadata classification.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/frma.2026.1882130","type":"journal-article","created":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T05:33:01Z","timestamp":1789536781000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating large language models for domain-specific scholarly metadata classification"],"prefix":"10.3389","volume":"11","author":[{"given":"Muhammad","family":"Haris","sequence":"first","affiliation":[{"name":"TIB\u2014Leibniz Information Center for Science and Technology, Leibniz University Hannover","place":["Hannover, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maryam","family":"Badar","sequence":"additional","affiliation":[{"name":"L3S Research Center, Leibniz Information Center for Science and Technology, Leibniz University Hannover","place":["Hannover, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,16]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.findings-emnlp.655","article-title":"\u201cMOLE: metadata extraction and validation in scientific papers using LLMs,\u201d","author":"Alyafeai","year":"2025","journal-title":"Findings of the Association for Computational Linguistics: EMNLP 2025, 12236-12264, Suzhou, China"},{"key":"B2","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1017\/S1351324923000438","article-title":"Improving short text classification with augmented data using gpt-3","volume":"30","author":"Balkus","year":"2024","journal-title":"Nat. Lang. Eng"},{"key":"B3","doi-asserted-by":"publisher","first-page":"2215","DOI":"10.1002\/asi.23329","article-title":"Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references","volume":"66","author":"Bornmann","year":"2015","journal-title":"J. Assoc. Inf. Sci. Technol"},{"key":"B4","article-title":"\u201cFrom system dynamics and discrete event to practical agent based modeling: reasons, techniques, tools,\u201d","volume-title":"Proceedings of the 22nd international conference of the system dynamics society","author":"Borshchev","year":"2004"},{"key":"B5","first-page":"1877","article-title":"\u201cLanguage models are few-shot learners,\u201d","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems, Vol. 33"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1418","DOI":"10.1038\/s41467-024-45563-x","article-title":"Structured information extraction from scientific text with large language models","volume":"15","author":"Dagdelen","year":"2024","journal-title":"Nat. Commun"},{"key":"B7","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1016\/j.aiopen.2022.11.003","article-title":"PTR: prompt tuning with rules for text classification","volume":"3","author":"Han","year":"2022","journal-title":"AI Open"},{"key":"B8","doi-asserted-by":"publisher","first-page":"374","DOI":"10.1145\/3767695.3769519","article-title":"\u201cText clustering as classification with LLMs,\u201d","author":"Huang","year":"2025","journal-title":"Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP 2025"},{"key":"B9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3677389.3702588","article-title":"\u201cLeveraging LLMs for scientific abstract summarization: unearthing the essence of research in a single sentence,\u201d","author":"Keya","year":"2024","journal-title":"Proceedings of the 24th ACM\/IEEE joint conference on digital libraries"},{"key":"B10","doi-asserted-by":"publisher","first-page":"100","DOI":"10.18653\/v1\/2022.deelio-1.10","article-title":"\u201cWhat makes good in-context examples for GPT-3?,\u201d","author":"Liu","year":"2022","journal-title":"Proceedings of deep learning inside out (DeeLIO 2022): the 3rd workshop on knowledge extraction and integration for deep learning architectures"},{"key":"B11","doi-asserted-by":"publisher","first-page":"61","DOI":"10.18653\/v1\/2022.acl-short.8","article-title":"\u201cP-tuning: prompt tuning can be comparable to fine-tuning across scales and tasks,\u201d","author":"Liu","year":"2022","journal-title":"Proceedings of the 60th annual meeting of the association for computational linguistics, Vol. 2 (short papers)"},{"key":"B12","doi-asserted-by":"publisher","first-page":"114","DOI":"10.18653\/v1\/2025.sdp-1.10","article-title":"\u201cData gatherer: LLM-powered dataset reference extraction from scientific literature,\u201d","author":"Marini","year":"2025","journal-title":"Proceedings of the fifth workshop on scholarly document processing (SDP 2025)"},{"key":"B13","doi-asserted-by":"publisher","first-page":"173","DOI":"10.18653\/v1\/2023.genbench-1.14","article-title":"\u201cIn-context learning for text classification with many labels,\u201d","author":"Milios","year":"2023","journal-title":"Proceedings of the 1st GenBench workshop on (benchmarking) generalisation in NLP"},{"key":"B14","doi-asserted-by":"publisher","first-page":"4962","DOI":"10.3390\/electronics14244962","article-title":"Automatic metadata extraction leveraging large language models in digital humanities","volume":"14","author":"Morej\u00f3n","year":"2025","journal-title":"Electronics"},{"key":"B15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/AICT59525.2023.10313167","article-title":"\u201cEffectiveness of generative artificial intelligence for scientific content analysis,\u201d","author":"Platt","year":"2023","journal-title":"2023 IEEE 17th international conference on application of information and communication technologies (AICT)"},{"key":"B16","doi-asserted-by":"publisher","first-page":"26839","DOI":"10.1109\/ACCESS.2024.3365742","article-title":"A review on large language models: architectures, applications, taxonomies, open issues and challenges","volume":"12","author":"Raiaan","year":"2024","journal-title":"IEEE Access"},{"key":"B17","doi-asserted-by":"publisher","first-page":"145","DOI":"10.18653\/v1\/2024.sdp-1.14","article-title":"\u201cMetadata enhancement using large language models,\u201d","author":"Song","year":"2024","journal-title":"Proceedings of the fourth workshop on scholarly document processing (SDP 2024)"},{"key":"B18","first-page":"1050","article-title":"\u201cStructured knowledge base enhances effective use of large language models for metadata curation,\u201d","volume-title":"AMIA annual symposium proceedings 2024","author":"Sundaram","year":"2025"},{"key":"B19","first-page":"163","article-title":"\u201cCan large language models serve as effective classifiers for hierarchical multi-label classification of scientific documents at industrial scale?\u201d","author":"Tabatabaei","year":"2025","journal-title":"Proceedings of the 31st international conference on computational linguistics: industry track"},{"key":"B20","doi-asserted-by":"publisher","first-page":"524","DOI":"10.18653\/v1\/2022.findings-emnlp.37","article-title":"\u201cTowards unified prompt tuning for few-shot text classification,\u201d","author":"Wang","year":"2022","journal-title":"Findings of the association for computational linguistics: EMNLP 2022"},{"key":"B21","doi-asserted-by":"publisher","first-page":"4257","DOI":"10.18653\/v1\/2025.findings-naacl.239","article-title":"\u201cGPT-NER: named entity recognition via large language models,\u201d","author":"Wang","year":"2025","journal-title":"Findings of the association for computational linguistics: NAACL 2025"},{"key":"B22","doi-asserted-by":"publisher","first-page":"13484","DOI":"10.18653\/v1\/2023.acl-long.754","article-title":"\u201cSelf-instruct: aligning language models with self-generated instructions,\u201d","author":"Wang","year":"2023","journal-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Vol. 1 (long papers)"},{"key":"B23","doi-asserted-by":"publisher","first-page":"280","DOI":"10.1007\/978-981-96-0865-2_23","article-title":"\u201cCapabilities and challenges of LLMs in metadata extraction from scholarly papers,\u201d","author":"Watanabe","year":"2024","journal-title":"Sustainability and empowerment in the context of digital libraries: 26th international conference on Asia-Pacific digital libraries, ICADL 2024, proceedings, part I"},{"key":"B24","doi-asserted-by":"publisher","first-page":"24824","DOI":"10.52202\/068431-1800","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B25","doi-asserted-by":"publisher","first-page":"371","DOI":"10.65286\/icic.v20i2.89433","article-title":"\u201cLLM-driven interactive document classification through keyword feedback,\u201d","author":"Yu","year":"2024","journal-title":"Proceedings of the 20th international conference on intelligent computing: poster volume"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1524","DOI":"10.1145\/3701716.3715528","article-title":"\u201cPushing the limit of LLM capacity for text classification,\u201d","author":"Zhang","year":"2025","journal-title":"Companion proceedings of the ACM on web conference 2025, WWW '25"}],"container-title":["Frontiers in Research Metrics and Analytics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frma.2026.1882130\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T05:33:03Z","timestamp":1789536783000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frma.2026.1882130\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,16]]},"references-count":26,"alternative-id":["10.3389\/frma.2026.1882130"],"URL":"https:\/\/doi.org\/10.3389\/frma.2026.1882130","relation":{},"ISSN":["2504-0537"],"issn-type":[{"value":"2504-0537","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,16]]},"article-number":"1882130"}}