{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"bioRxiv"}],"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T12:11:50Z","timestamp":1785413510307,"version":"3.56.0"},"posted":{"date-parts":[[2026,4,27]]},"group-title":"Neuroscience","reference-count":24,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T00:00:00Z","timestamp":1777248000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"id":[{"id":"https:\/\/ror.org\/03v76x132","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,4,27]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>High-quality metadata is essential for scientific discovery, yet sparse annotations in rapidly growing repositories leave many biologically relevant details uncaptured. We evaluated whether large language models (LLMs) can accurately infer ion channel and receptor subtype metadata from source code in a neuroscience repository.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We extracted 5,133 model files from ModelDB. A subset of 1,100 was manually annotated; 253 were held out for testing, and the remainder split into training (80%) and validation (20%) sets. LLM-based approaches (GPT-5.2 and GPT-mini) were evaluated under zero-shot and heuristic-augmented prompting. Performance was assessed at type and subtype levels using accuracy, precision, recall, and F1 score. A feature-engineered XGBoost model using text- and simulation-derived features served as a baseline.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>LLMs outperformed the XGBoost baseline. At the type level, GPT-mini with heuristic augmentation achieved the highest performance (accuracy 96.0%, F1 0.962). At the subtype level, both GPT-5.2+heuristics and GPT-mini+heuristics achieved identical accuracy (88.1%), with GPT-5.2+heuristics achieving the highest F1(0.878). Model outputs were consistent across runs and errors confined to related mechanistic families.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Discussion and Conclusion<\/jats:title>\n                  <jats:p>LLMs demonstrate strong potential for metadata annotation directly from source code, outperforming feature-engineering approaches with minimal tuning. However, performance varied across subtypes, and errors often reflected ambiguity or bias toward more common labels. These findings suggest LLMs may serve as practical tools for scalable metadata generation in biomedical repositories, although careful evaluation and domain-specific validation remain important. While demonstrated in computational neuroscience, this approach may generalize to repository-agnostic metadata annotation in other scientific code repositories.<\/jats:p>\n                <\/jats:sec>","DOI":"10.64898\/2026.04.23.720371","type":"posted-content","created":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T23:05:11Z","timestamp":1777331111000},"source":"Crossref","is-referenced-by-count":0,"title":["All Models are Wrong, Some are Annotated: Automating Metadata in Biomedical Repositories"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5807-2635","authenticated-orcid":false,"given":"Inessa","family":"Cohen","sequence":"first","affiliation":[{"name":"Program in Computational Biology and Bioinformatics, Yale University, New Haven, CT 06510, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-9884-4400","authenticated-orcid":false,"given":"Hongyi","family":"Yu","sequence":"additional","affiliation":[{"name":"Biostatistics, Yale School of Public Health, Yale University, New Haven, CT 06510, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6394-3127","authenticated-orcid":false,"given":"Robert A.","family":"McDougal","sequence":"additional","affiliation":[{"name":"Program in Computational Biology and Bioinformatics, Yale University, New Haven, CT 06510, United States"},{"name":"Biostatistics, Yale School of Public Health, Yale University, New Haven, CT 06510, United States"},{"name":"Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT 06510, United States"},{"name":"Wu Tsai Institute, Yale University, New Haven, CT 06510, United States"},{"name":"Interdepartmental Neuroscience Program, Yale University, New Haven, CT 06510, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2026043000351420000_2026.04.23.720371v1.1","unstructured":"Riley J. Understanding metadata: what is metadata, and what is it for. Baltimore, MD: : National Information Standards Organization 2017."},{"key":"2026043000351420000_2026.04.23.720371v1.2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2507.22391"},{"key":"2026043000351420000_2026.04.23.720371v1.3","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.69013"},{"key":"2026043000351420000_2026.04.23.720371v1.4","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.18"},{"key":"2026043000351420000_2026.04.23.720371v1.5","doi-asserted-by":"publisher","DOI":"10.1016\/j.csbj.2023.10.006"},{"key":"2026043000351420000_2026.04.23.720371v1.6","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2021.100322"},{"key":"2026043000351420000_2026.04.23.720371v1.7","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-026-01219-7"},{"key":"2026043000351420000_2026.04.23.720371v1.8","unstructured":"Martin K. NIH Releases New Public Access Policy and Supplemental Guidance; Releases Plan for Metadata and Personal Identifiers (PIDs). Off. Sci. Policy. 2024. https:\/\/osp.od.nih.gov\/nih-releases-new-public-access-policy-and-supplemental-guidance-releases-plan-for-metadata-and-personal-identifiers-pids\/ (accessed 31 Aug 2025)."},{"key":"2026043000351420000_2026.04.23.720371v1.9","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2017.10.013"},{"key":"2026043000351420000_2026.04.23.720371v1.10","doi-asserted-by":"publisher","DOI":"10.1101\/2023.09.11.557287"},{"key":"2026043000351420000_2026.04.23.720371v1.11","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-025-01096-6"},{"key":"2026043000351420000_2026.04.23.720371v1.12","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocae097"},{"key":"2026043000351420000_2026.04.23.720371v1.13","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2024.104724"},{"key":"2026043000351420000_2026.04.23.720371v1.14","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-22940-0"},{"key":"2026043000351420000_2026.04.23.720371v1.15","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.03602"},{"key":"2026043000351420000_2026.04.23.720371v1.16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2201.11903"},{"key":"2026043000351420000_2026.04.23.720371v1.17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2601.06423"},{"key":"2026043000351420000_2026.04.23.720371v1.18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2203.11171"},{"key":"2026043000351420000_2026.04.23.720371v1.19","doi-asserted-by":"publisher","DOI":"10.1007\/s10827-016-0623-7"},{"key":"2026043000351420000_2026.04.23.720371v1.20","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.1996.97084512"},{"key":"2026043000351420000_2026.04.23.720371v1.21","doi-asserted-by":"publisher","DOI":"10.1113\/jphysiol.1952.sp004764"},{"key":"2026043000351420000_2026.04.23.720371v1.22","doi-asserted-by":"publisher","DOI":"10.1162\/089976600300015475"},{"key":"2026043000351420000_2026.04.23.720371v1.23","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooag043"},{"key":"2026043000351420000_2026.04.23.720371v1.24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.415"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.64898\/2026.04.23.720371","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T07:35:27Z","timestamp":1777534527000},"score":1,"resource":{"primary":{"URL":"http:\/\/biorxiv.org\/lookup\/doi\/10.64898\/2026.04.23.720371"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,27]]},"references-count":24,"URL":"https:\/\/doi.org\/10.64898\/2026.04.23.720371","relation":{},"subject":[],"published":{"date-parts":[[2026,4,27]]},"subtype":"preprint"}}